spymemcached + vbuckets + non default bucket

 Hey Dan, happy to help.  SASL auth is what you'll want to be using here, though it should be possible to get it working with the dedicated port as well.  On the wiki, there is some example code:

 mc = new MemcachedClient(baseURIs, "bucket_name", "bucket_name", "bucket_password", true);

This is where you define which bucket you want to access, and it uses the above parameters as the SASL authentication.

Let me know if you run into any issues with this.  You should be using the latest packages as they've been updated a bit over the last couple days.

Perry

Oh wow, I need to learn how to read, that works beautifully, thanks!

It only works if I do not set a password for the bucket in the membase client. If I set a SASL password the bucket doesn't show up in /pools/default/buckets and my client can't connect:

Couldn't create a connection, bailing out:
ConfigureException Configuration for bucket foo was not found.

I don't need SASL so I am fine with leaving the password blank.

Thanks,

Dan

Dan,

I will try to find out why the  bucket  with SASL is not working. Let us know if you see any other issues.

How many nodes are you using? Do you plan to use membase or memcached buckets? Have you tried the latest Membase release - 1.6.5 ?

Thanks

bhawana

Thanks for the feedback Dan, I'll take a look into that...glad it's working well enough for you.

Perry

We have 6 nodes with membase buckets. We upgraded to 1.6.5 yesterday but haven't had a chance to test the scenario with SASL again.

 Thanks for the feedback Dan, I trust the upgrade went smoothly?

You may want to make sure you're running the latest spymemcached client (I believe there's going to be another update shortly if it hasn't happened yet) as we identified a bug here in an early release.

Let me know if there's anything else I can do to help.

Perry

I typed up a reply but it was marked as spam for some reason so I'll try a shorter one this time.

One issue with the spymemcached client we are seeing is that when we add a server to the cluster we have to restart the client or else it gives us timeouts on get operations.

We are using: memcached-2.5-42-g7c63cef.jar

 Thanks Dan, I'm raising this issue directly now.  That's not quite the latest, though the latest is not yet available so I'll get it up as soon as possible.

I've also added you to our spam whitelist so that shouldn't be an issue going forward...we've had some issues in the past and it's now a balance of potentially blocking legitimate messages to avoid having the forums overrun.  Did you know that they've got humans filling out Captchas for the spammers now?  Crazy...

Keep your eye out for an update to spy soon...I'm pestering my best buddy over here ;-)

Understandable on the spam filtering, I run a forum that gets constantly hammered by spam registrations.

So here is what we've found with testing 1.6.5 on our dev cluster:

Create a 2 node cluster, connect with spymemcached client, doing a mix of set/gets. While the client is running we add a 3rd server and attempt a rebalance. The rebalance fails at 50% and the client throws a time out exception:

net.spy.memcached.OperationTimeoutException: Timeout waiting for value

Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: 10.201.32.53/10.201.32.53:11210

Note that 10.201.32.53 is the 2nd node, not the 3rd.

Any further attempts to rebalance fails. However, if we disconnect the client and attempt to rebalance again it succeeds.

 Thanks for your continued investigation and patience Dan.  Let's separate the issues of the client from the server for a moment.  Hopefully the latest client will resolve these issues when it is released, but I'm more concerned at the moment about the rebalance failing.

Can you run /opt/membase/bin/ep_engine/management/collect_info <output_file> on each of the nodes and send that over to perry -at- membase -dot- com?

Thanks

Perry

 This issue was caused by a known bug that causes a new node to be delayed in shutting down its services before being able to start them back up again and particpate in the rebalance.  Retrying the rebalance until it succeeds should be a viable workaround and I'll see about getting this bug fixed as soon as possible.

Thanks

Perry

Thanks Perry.

We've modified our Java code so that when it detects the timeout it shuts down the spymemcache client and starts a new one. That let's us add/remove nodes without issue.

Dan

 Dan, we've updated the spymemcached beta client on our website and it would be good to get some mileage on it from you and see if it has resolved these issues.

let me know if you have any other questions.

Perry

When I originally created the cluster I had allocated too much memory on each node. (I forgot the section in the wiki about only allocating up to 80%). We loaded in about 180GB of data and found that each node was swapping like crazy. Each memcached process was using between 46-48GB per node. Each node has 64GB of ram with 12GB being taken up by cache (I'm not sure why memcached wasn't grabbing more memory from the kernel).

I took down the cluster and wiped all 6 boxes and started over, and specified 50GB per node this time. I built it with only 5 nodes and loaded 12GB of data. I then attempted to add the 6th node. It failed at 50% rebalancing. i tried rebalancing it again with no luck. I uninstalled membase and removed all the left over files, reinstalled it and attempted to add it to the cluster a 2nd time and it worked fine. I don't have any logs left over from that. Today we are trying to duplicate this issue on a dev cluster but it is possible that I didn't remove all of the left over data files the first time.

As for the client, I'm using it in servlets running on tomcat6. When we added the 6th node the servlets showed timeouts on get operations. Bouncing tomcat fixed the issue. We are going to be test that as well once we get a dev cluster running. Most likely we'll just have the client reconnect when it sees a timeout. Unless this is the known issue you were talking about.

We are using memcached-2.5-42-g7c63cef.jar

Thanks for the quick replies Perry.

Dan

Perry -

(Dan and I work together) What is happening is that the spymemcached client throws a net.spy.memcached.OperationTimeoutException.  If you catch that and try again, it will then throw a java.lang.RuntimeException. 

If you catch that and try again ... it will then block on the get/set and continue to try and connect to the node that no longer exists in the cluster, forever.  The only way we are able to get around this is to call the shutdown() method of the client when we catch one of those exceptions and instantiate a new client object. 

If you don't catch either of those and also don't shutdown() the client (or let it go out of scope) ... the cluster will never allow you to rebalance until you kill the client process. It will always fail as long as that client is connected. In our case, I can reproduce this in our test client because the spymemcached client is inside a Runnable object that doesn't get collected by the main thread when those exceptions cause it to exit.

I accidentally blew away the log from my test client, so I'll need to set up everything and repeat the series of events to get exactly what happens - I'll do that ts weekend.

Thanks,

Brian Roach 

Thanks Perry, I've grabbed the new client and will start playing with it today.

Dan