"I'm not responsible for this vbucket" errors after membase server reboot

Hi Kevin,

NOT_MY_VBUCKET is the expected response if that server node is warming up. The map that the node uses to determine the vbuckets it is responsible for, lives in the sqlite database along with the keys/metadata/values that are being warmed up. The node will need to completely finish warming up before it can start server keys for any of the vbuckets that it is responsible for - it will also need to completely finish warming up before it can reference the vbucket map to understand what vbuckets it is responsible for in the first place - hence the error you are seeing.

In your scenario, after determining the node was taking too long to warm up, you would have been better off stopping the restart, doing a failover and re-adding the downed node/rebalancing afterwards.

You are probably wondering "how do I know that my node is taking to long to warm up?" or "how long will my node take to warm up?" .

You can use the info below to determine whether a warmup or a failover makes more sense.

To gauge warm up time for a given node, we generally look at a few metrics within the logs to get a sense of the rate at which Membase can process the items you have on disk. We look at:
curr_items - The number of items currently active on this node.
curr_items_tot - Total number of items this node knows about(active and replica)
ep_warmed_up - Number of items retrieved from disk - this should increase during warmup.
ep_warmup_oom - Number of out of memory exceptions received from the server while loading data into RAM
ep_warmup_thread - The status of the warmup thread.
ep_warmup_time - How long the warmup thread was running for in microseconds.

[root@localhost bin]# /opt/membase/bin/mbstats localhost:11210 all | egrep 'warm|curr'
curr_connections: 9
curr_items: 52287
curr_items_tot: 104051
ep_store_max_concurrency: 10
ep_warmed_up: 104051
ep_warmup: true
ep_warmup_dups: 0
ep_warmup_oom: 0
ep_warmup_thread: complete
ep_warmup_time: 1964924
vb_active_curr_items: 52287
vb_pending_curr_items: 0
vb_replica_curr_items: 51764

we can see that the warmup thread took 1.9 seconds and warmed up 104k items 52k of which are active. Usually we will divide the number of items warmed up by the warmup time to see the rate at which the server is processing items. Using this we can get a sense for how long it should take to fully warm up the node based on the number of total items.

hope this helps.

-Alex.

Hi Alex,
Thanks for the extremely helpful reply! What steps would you suggest be taken when a machine needs to be rebooted to apply Windows Updates, etc. Should we Remove the server, rebalance, perform the necessary maintenance, reboot, add server back to cluster, and rebalance? Does that sound like the best approach?

Thanks again for your insight. It's much appreciated!
-k

Hi Again Kevin,

not a problem, we're here to help.

For scheduled maintenance, the steps you have outlined are exactly what we would recommend.

thanks

-Alex.