Left over replicas after expiration

I loaded more keys and let them expire. Now I have 540k keys left over in the replica count.

:~> /opt/membase/bin/mbstats localhost:11212 all | grep items
curr_items: 0
curr_items_tot: 539068
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 693003611
ep_total_del_items: 304600794
ep_total_new_items: 304927464
ep_uncommitted_items: 0
vb_active_curr_items: 0
vb_pending_curr_items: 0
vb_replica_curr_items: 539068

Thanks Dan, we'll try to reproduce this on our side as well to confirm if there is any bug.

If you let the systems sit for long enough, do the replica counts continue to decrease? Since each node runs individually, the "hourly" process will run at different times on each node and so that counts might be off a bit in between.

The stats command that you are running aggregates the stats from all nodes, if you replace 11212 with 11210 and then put the name of the bucket on the end, you can get the specific stats for each node:
/opt/membase/bin/mbstats localhost:11210 all | grep items

It would be very helpful if you could isolate this to one node, and then watch the stats on that one to see if the expired items get cleaned up. You can also grep for "expir" to watch the expiry purger process run once an hour and how many items it has expired.

As for the CAS badval, I'll put in a request to have it added back to the UI, you should still be able to grab it from the text-based stats right?

Perry

Perry,

So I have some interesting results. I was able to reproduce the same issue with 1.6.5.4 on that same dev cluster. For now I'm going to assume there was something weird going on with those servers.

I did let the cluster sit overnight and it still had replicas the next day. If I get a chance I'll go back to try and isolate which node was being a problem.

Sadly today our 12 node 1.6.5 production cluster completely imploded. Last week we lost a DIMM in one of the nodes and had to fail it out. However, after we rebalanced we received a thousands of badcas val errors. We coded around that even though it meant we had invalid data in the cluster. Throughout the week we had memcached die randomly on other nodes and failed them out. Today we lost two DIMMS in another server and the memcached process died on every node. Membase came back up but memcached would not start on the servers with errors like:

Port server memcached on node 'ns_1@ip' exited with status 134. Restarting. Messages: Item was expired at load: key
Item was expired at load: key
memcached: stored-value.hh:974: add_type_t HashTable::add(const Item&, bool, bool): Assertion `v->isDirty() == isDirty' failed.

In the interest of time we wiped the cluster clean and installed 1.7.1 since it looks like there were quite a few memcached fixes since the 1.6.5 version. Hopefully 1.7.1 is a bit forgiving of hardware failures.

I will have to check the text-based stats for the CAS Badval. It looks like it's no longer in the JSON response provided by the REST API. The other statistics we find useful is the GET misses.

Thanks again,
Dan

Sorry to hear about your troubles Dan, 1.7.1 should be much stabler and has fixed that specific "assert" issue.

Have you been unable to reproduce the left-over replicas or just not had a chance to try yet?

Anything you see in the UI is in the REST API, but there is a large amount of stats only available through 'mbstats'. To easily aggregate, you can point it to 11212, etc but if you want specific per-node stats, point it at 11210 as I showed earlier. Get misses are in there too.

Perry

I haven't had a chance to reproduce the left-over replicas issue, I've been too busy putting out fires. I'll try to look at it tomorrow.

I'll check out mbstats as well.

Thanks,
Dan

I haven't had time to test this in dev but it appears that this is happening in our production environment.

http://imageshack.us/photo/my-images/225/comparisoni.jpg/

Red line is replicas, blue is active. The one difference between this case and the one I mentioned before is that I loaded 140 million keys last week which exceeded the watermark, so some replicas were ejected from memory. About 120 million keys expired on Friday.

Vbuckets: http://imageshack.us/photo/my-images/845/vbucket.jpg/

Right now the stats seem really off since by now all of this active data will has been reloaded multiple times. The replica rate should show at least 100% (or more since the replica count is higher than active).

Thanks for the additional data here Dan. I'm working with the engineering team to reproduce this issue and see what the overall impact is as well as fixing it.

Perry

Great, let us know if there is any other data you need from us.

Dan

Hi Perry,

I am just checking in to see if you have been able to reproduce this issue or if we are the only ones seeing it.

Thanks,
Dan