rebalancing nodes getting stuck in 1->6 or 2->1 with 100k items

Farshid Ghods added a comment - 12/Jan/12 4:11 PM erros i se in the diag [ns_server:warn] [2012-01-12 12:37:50] [ ns_1@10.1.6.149 :ns_config_rep_merger:ns_config_rep:do_merge:276] config cas failed. Retrying [ns_server:warn] [2012-01-12 12:37:50] [ ns_1@10.1.6.149 :ns_config_rep_merger:ns_config_rep:do_merge:276] config cas failed. Retrying [ns_server:info] [2012-01-12 12:37:50] [ ns_1@10.1.6.149 :ns_config_events:ns_node_disco_conf_events:handle_event:56] ns_node_disco_conf_events config all [ns_server:warn] [2012-01-12 12:37:50] [ ns_1@10.1.6.149 :ns_config_rep_merger:ns_config_rep:do_merge:276] config cas failed. Retrying [ns_server:info] [2012-01-12 12:37:50] [ ns_1@10.1.6.149 :ns_config_rep:ns_config_rep:handle_info:181] Pushing config [ns_server:warn] [2012-01-12 12:37:50] [ ns_1@10.1.6.149 :ns_config_rep_merger:ns_config_rep:do_merge:276] config cas failed. Retrying [ns_server:info] [2012-01-12 12:37:50] [ ns_1@10.1.6.149 :ns_config_rep:ns_config_rep:handle_info:183] Pushing config done [ns_server:warn] [2012-01-12 12:37:50] [ ns_1@10.1.6.149 :ns_config_rep_merger:ns_config_rep:do_merge:276] config cas failed. Retrying [ns_server:warn] [2012-01-12 12:37:50] [ ns_1@10.1.6.149 :ns_config_rep_merger:ns_config_rep:do_merge:276] config cas failed. Retrying [ns_server:warn] [2012-01-12 12:37:50] [ ns_1@10.1.6.149 :ns_config_rep_merger:ns_config_rep:do_merge:276] config cas failed. Retrying [ns_server:warn] [2012-01-12 12:37:50] [ ns_1@10.1.6.149 :ns_config_rep_merger:ns_config_rep:do_merge:276] config cas failed. Retrying no_db_file} [couchdb:info] [2012-01-12 13:51:42] [ ns_1@10.1.6.149 :'capi_set_view_manager-default':couch_log:error:42] MC daemon: Error opening vb 11 in <<"default">>: {not_found,

Farshid Ghods added a comment - 12/Jan/12 4:27 PM memcached crashed i think more errors here Port server memcached on node ' ns_1@10.1.6.152 ' exited with status 134. Restarting. Messages: Trying to connect to mccouch: "localhost:11213" Connected to mccouch: "localhost:11213" Trying to connect to mccouch: "localhost:11213" Connected to mccouch: "localhost:11213" Extension support isn't implemented in this version of bucket_engine Preloaded 0 keys (with metadata) The second phase of warmup took 7344 (us). memcached: objectregistry.cc:104: static void ObjectRegistry::onDeleteItem(Item*): Assertion `stats.memOverhead.get() < ((size_t)1<<(sizeof(size_t)*8-1))' failed. ns_port_server000 ns_1@10.1.6.152 13:57:59 - Thu Jan 12, 2012 Port server memcached on node ' ns_1@10.1.6.154 ' exited with status 134. Restarting. Messages: Trying to connect to mccouch: "localhost:11213" Connected to mccouch: "localhost:11213" Trying to connect to mccouch: "localhost:11213" Connected to mccouch: "localhost:11213" Extension support isn't implemented in this version of bucket_engine Preloaded 0 keys (with metadata) The second phase of warmup took 2826 (us). memcached: objectregistry.cc:84: static void ObjectRegistry::onDeleteQueuedItem(QueuedItem*): Assertion `stats.memOverhead.get() < ((size_t)1<<(sizeof(size_t)*8-1))' failed.

Aleksey Kondratenko added a comment - 17/Jan/12 4:00 PM btw. cas failed message is very very likely harmless. We do optimistic concurrency when merging config changes. CAS failed simply means we will retry again and likely succeed.

Aleksey Kondratenko added a comment - 17/Jan/12 4:01 PM At least part of issue was set view manager race(s). Aliaksey is continuing work here.