Trying to use TAP to dump keys dies but clients disconnects before finishing
Membase version 1.7.1
I need to dump all the key/values in one of buckets (460 million keys) so I can go back and compress the values to free up some memory for other buckets.
I decided to try it out first on one of our smaller buckets that only has 10 million keys. It gets through about 1.8 million keys and dies. No exceptions are thrown, the client just disconnects.
First I tried spymemcached-2.7.3, then I tried the Couchbase 1.0.1 Java client. Both have the same result.
For fun I then tried it on the bcuket with 460 million keys. It made it through about 120 million before the client disconnected. I remember trying this about 3 months ago and had similar results.
This is what my code looks like with the Couchbase library:
TapClient client = new TapClient ( membaseURIs, "bucketofdoom", "");
Operation op = client.tapDump ( "danlovestap" );
while ( client.hasMoreMessages () )
{
ResponseMessage message = client.getNextMessage ();
if ( message != null )
{
System.out.println ( message.getKey () );
}
}Is this an issue with Membase 1.7.1, the java client or am I doing something stupid again? I've been holding off updating to the 1.8 release since there are a few bugs that affect us which are listed to be fixed with 1.8.1.
Dan
Hi ingenthr,
Can you identify what the problem was with the client and perhaps point to the associate ticket/bug that resolves it? I'm experiencing the same issue with SpyMemcached 2.7.3 against Membase 1.7.2. Because we have a fairly large and mature cluster, upgrading to 1.8.1 isn't quite feasible so I'm wondering if there are modifications I can make within SpyMemcached library to resolve them.
Thanks,
Prashanth
The issue was that the client was not correctly initiating a TAP connection against all nodes.
The 1.0.2 client should work against Membase 1.7.2, though we haven't tested that configuration for support. I see no reason it shouldn't work though. Because of some other changes, that'd be your best bet. The TAP interface has changed with the refactoring between spymemcached 2.7.3 and 2.8, so it'd be a bit of work to backport it.
It's in the set of commits around Feb 19 here if you wanted to try to backport the change:
https://github.com/couchbase/couchbase-java-client/compare/1.0.1...1.0.2
Matt,
I finally got a chance to start migrating from spymemcached 2.7.3 to the couchbase 1.0.3 client last week but I haven't had a chance to try out the test case for the TAP interface yet since I'm currently reaping about 800 million keys out of the bucket I'm interested in.
I'll post a reply when I get a chance to test it out.
Dan
Some updates: Couchbase 1.8.0, Couchbase Client 1.0.3
I don't know if this post still belongs in this forum but it fits this thread.
I started the tap dump and I immediately had to shutdown our App that uses the bucket due to timeout issues. We do deletes, adds, gets, and cas operations all using a custom transcoder that uses kryo to serialize java objects in and out of couchbase.
Our App uses a connection pool of clients and retries operations until they succeed. We were seeing lots of timeouts from one of our nodes and the clients kept failing it out. When that happens we previously destroyed the client and checked out a new one but that meant we were shutting down/starting clients too much. I changed the logic to check the client back in and check out a new one but then we hit our max number of clients and incoming requests failed to get a client from the pool.
This behvaior is identical to when we rebalance the cluster (which uses the TAP interface). We have to shut down the App then too which really kills the whole HA idea.
Our bucket has 1.04 billion keys in it. When my TAP dump finished it had pulled out 1.45 billion keys, so obviously there are some duplicates in my dump file, I'll have to dig through it.
Is there any way to increase the timeout for when the client fails out a node?
Is there any way to rate limit the TAP dump so it doesn't hammer the servers and affect performance so much? We were pulling out around 800k keys a second with 12 tap senders.
Hi all,
I'm also experiencing what dan mentions above. We have a cluster of four machines, all running Couchbase 1.8.1 with about 14GB of RAM each. I started a TAP dump against the cluster from a single machine which ran smoothly for about 5 minutes, processing about 2millions keys, but then ground to a halt afterwards. Looking through the Couchbase Management Console, we were seeing a drain of about 140keys/second down from 15k/second and massive amounts of swap usage on each of the boxes. However, we weren't seeing any errors in the logs or otherwise in the console itself which led us to believe the cluster is healthy.
Our application-tier services runing against the cluster were also failing with numerous timeout errors and other generic "Membase failure" exceptions using Couchbase Client 1.0.3. Each of our services uses a pool of never more than five connections against couchbase and all perform typical read, delete and CAS opertions. We stopped all the application-tier services in an attempt to alleviate the TAP dump, but it didn't help; the TAP dump remained at about 200keys/second and never returned to it's normal speed. We eventually had to kill the TAP dump process altogether to bring the application back up.
Our bucket isn't all that large, topping off at about 55millions keys. Can someone shed some light on what could be going on here?
Thanks,
Prashanth
My guess is that TAP is streaming out data that exists on disk, and that disk IO is in the way of your application IO. Application IO (reads) are prioritized, but even when prioritized they can take longer to service if there are other requests in front of it.
My recommendation would be to throttle the TAP receiving side. This should allow for more room for your application to do its work.
As to why you first have a high rate of throughput and then a lower rate, the data in memory is sent first and then the data on disk. (well, there's some parallelism, but memory generally happens first).
Our replication within a cluster also uses TAP, so many of the recommendations related to rebalance apply here.
You can rate limit TAP by simply not reading things too quickly. The TAP client automatically sends acknowledgements, and those let the server send more.
On timeout tuning, do you mean server autofailover, or client timeouts? See the existing Java documentation at couchbase.com/docs for timeout tuning info.
Hmm, if that's the case, should I be seeing the disk fetches/sec metric going up when I see the TAP stream slow down, presumably because it's touching disk to fetch the data? I ask because I wasn't actually seeing this number move from zero at all. I'm not sure if it's a bug in the metric/dashaboard or if there's something more devious going on.
As for throttling, do you mean to pause between calls to get the next message from the stream? Do you have a recommendation on what the length of the pause should be and when to perform it, if not randomly? The client itself doesn't seem to offer any advice in this area.
Thanks,
Prashanth
Yes, I mean to pause on calls to getting the next message. You can either do it in batches, or all at once.
On the length of the pause, that really depends. This isn't affecting traffic on the TAP side, it's really affecting your "front end" traffic on the cluster itself. The only reason there is an effect is that you don't have enough IO capacity to service both effectively.
The server already prioritizes it's "front end" traffic, but there is always some cost for this additional IO. I'd probably recommend starting in a pretty conservative way, calculating for the items I have how long do I want to receive them within and then go a little below that, and then lower it.
As far as the stats are concerned, which stat specifically? bgfetch? It's not going to register as a cache miss, but it should show in the TAP area of the stats.
There was also a TAP issue which we addressed in the 1.0.2 Java client. I think this is the cause of what you were seeing above.