[MB-5546] Increasing the default timeouts on ns_server to avoid rebalance failures due to ep-engine stats timeout issues in large cluster or clusters where some nodes are actively using swap

Wed, 9 Jan 2013 22:59:12 -0600

Related issues:-
~~MB-5360~~
~~MB-5352~~

We have multiple bugs related to the timeouts we are hitting on ns_server :-
1) When in swap
2) On windows even on a small cluster.

This bug is to recommend increasing the default timeouts.

We used the following timeouts on most of the params, its not all in one solution, but hopefully would cover basic secnarios.
ns_memcached_outer, 60000
ns_memcached_open_checkpoint, 60000
ns_memcached_outer_heavy, 60000
ns_memcached_outer_very_heavy, 120000
ns_memcached_connected, 10000
ebucketmigrator_connect, 60000

Summary, some error messages and fixes that worked:-
1) Rebalance exited with reason {exited} {'EXIT',<0.22700.12>,{timeout,{gen_server,call,[{'ns_memcached-default','ns_1@10.3.2.81'},{stats,<<"tap">>},30000]}}}}
Fix : adjust timeout value - 120sec - ns_memcached_outer_very_heavy
2) Rebalance exited with reason {exited,
{replicator_died,

Fix: Adjust timeout value - 120 sec - ns_memcached_outer_heavy

3) Rebalance exited with reason {exited,
{'EXIT',<0.24287.15>,
{timeout,
{gen_server,call,
[{'ns_memcached-default','ns_1@10.3.2.81'},
{stats,<<"tap">>},
30000]}}}}

Fix : Adjust timeout to 120sec
4) Rebalance exited with reason {{change_filter_failed,
{'EXIT',
{timeout,

Fix : Adjust timeout values -
ebucketmigrator_connect 120 secs
ns_memcached_connected 1 sec

Couchbase

[MB-5546] Increasing the default timeouts on ns_server to avoid rebalance failures due to ep-engine stats timeout issues in large cluster or clusters where some nodes are actively using swap