All nodes in cluster in pending state after m/c restart

Last weekend, we stopped couchbase servers on individual node (service couchbase-server stop). Then stopped AWS m/c.
Today, when i started all servers, all nodes in cluster are in pending state. I can see bucket is loaded with all 7.6 million items.
How do i know why all nodes are in pending state ?
Could you please help.

Last time when i restarted, they came up within few minutes. Now it is almost 6-7 Hrs and they are still in pending state.

They are likely either still warming up, or are waiting for all the cluster nodes to come back online.

You can monitor warmup progress using cbstats warmup.

How long will this warm-up normally take? I dont see much CPU utilization either.

./cbstats 177.39.20.105:8091 warmup
Stats ‘warmup’ are not available from the requested engine. (Operation timed out)

I get this message using cbstats command.

As per the documentation, cbstats connects directly to the data port, i.e. 11210, not the cluster management port.

./cbstats 177.39.20.105:11210 warmup
Stats ‘warmup’ are not available from the requested engine. (Got empty data (remote died?).)

Are you running that command from the local machine? What do you get if you use 127.0.0.1:11210 as the address?

Its same.
./cbstats 127.0.0.1:11210 warmup
Stats ‘warmup’ are not available from the requested engine. (Got empty data (remote died?).)

Hmm ok. What version of CB are you running? What OS? Is there anything in the Logs tab in the UI?

Can you see if there’s any process listening on port 11210 (should be memcached).

Version: "3.0.1-1444-rel-community.
Yeah, memcached is listening on 11210.
All nodes are up now (green), but i have another problem as posted in Re-indexing only one node.

Thanks for your help !

Hi,

We are currently facing similar issues. The nodes are all in pend state after the servers restart.

Can you please let me know how did you manage to bring the nodes to healthy state

Regards…