Cannot enable auto-failover / delete bucket
Hi,
We have a 8 node cluster. And for some strange none of the admin commands are working. I cannot enable fail over, I cannot delete a bucket. Can't even remove server and rebalance. The shows the following errors:
On trying to enable fail over:
Server error during processing: ["web request failed",
{path,"/settings/autoFailover"},
{type,exit},
{what,
{noproc,
{gen_server,call,
[{global,auto_failover},
{enable_auto_failover,30,1}]}}},
{trace,
[{gen_server,call,2},
{menelaus_web,
handle_settings_auto_failover_post,1},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}]
On trying to delete bucket:
Server error during processing: ["web request failed",
{path,"/pools/default/buckets/sitep13n"},
{type,exit},
{what,
{noproc,
{gen_fsm,sync_send_event,
[{global,ns_orchestrator},
{delete_bucket,"sitep13n"},
infinity]}}},
{trace,
[{gen_fsm,sync_send_event,3},
{menelaus_web_buckets,
handle_bucket_delete,3},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}]
on trying to remove server and rebalance:
Server error during processing: ["web request failed",
{path,"/controller/rebalance"},
{type,exit},
{what,
{noproc,
{gen_fsm,sync_send_event,
[{global,ns_orchestrator},
{start_rebalance,
['ns_1@10.63.49.44','ns_1@10.63.49.250',
'ns_1@10.63.49.254',
'ns_1@10.63.52.163',
'ns_1@10.63.52.248',
'ns_1@10.63.53.244',
'ns_1@10.63.55.243'],
['ns_1@10.63.55.240'],
[]}]}}},
{trace,
[{gen_fsm,sync_send_event,2},
{menelaus_web,do_handle_rebalance,3},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}]
Any ideas what I can do? Do I have to bring down the entire cluster, delete the data directory manually and bring up the entire cluster again???
regards,
-Piyush
Hi Muthu,
Yes, the issues continued for 2 days. Finally, I did exactly what I mentioned in my OP. In fact, even delete bucket stopped working. So I had to bring down the cluster, delete the data files and literally redo the cluster.
Not an ideal solution everytime we run into this kind of a problem.
regards,
-Piyush
Hi,
Have you tried using the Command-line? Do you have any specific happening which lead you to face this issue.
Please provide us logs. you can run cbcollectinfo to check the logs. Anyother information you may find helpful do post it.
Thanks
Neo
Hi Neo,
At that point in time, no I did not try the command line options! Maybe I should have, but it didn't strike me. Also, I'll try to get logs as you mentioned and see if any other info is accessible.
thanks,
-Piyush
Hi Piyush - Please let us know in case if you are still facing issues.
Thanks
Muthu