Details
Description
Ubuntu-64 bit, 2 node cluster, earlier version 1.7.0
Cluster nodes
10.1.4.83
10.1.4.82
One node upgrades successfully, other node fails with following error
Previous config.dat file is /opt/membase/var/lib/membase/config/config.dat.debsave
Target node: ns_1@10.1.4.83
ERROR: bucket escript: exception error: no function clause matching is configured but missing: /opt/membase/var/lib/membase/data/escript: exception error: no function clause matching
dpkg: error processing membase-server (--install):
subprocess installed post-installation script returned error exit status 1
This was not the case with 1.7.1-4.
Cluster nodes
10.1.4.83
10.1.4.82
One node upgrades successfully, other node fails with following error
Previous config.dat file is /opt/membase/var/lib/membase/config/config.dat.debsave
Target node: ns_1@10.1.4.83
ERROR: bucket escript: exception error: no function clause matching is configured but missing: /opt/membase/var/lib/membase/data/escript: exception error: no function clause matching
dpkg: error processing membase-server (--install):
subprocess installed post-installation script returned error exit status 1
This was not the case with 1.7.1-4.
-
- logs-10.1.4.98.out.gz
- 14/Aug/11 7:00 PM
- 145 kB
- Steve Yen
-
- logs-10.1.4.99.out.gz
- 14/Aug/11 7:00 PM
- 34 kB
- Steve Yen
Activity
- All
- Comments
- Work Log
- History
- Activity
- Gerrit Reviews
Hide
Steve Yen
added a comment -
The two nodes are (strangely) returning different results for mbdumpconfig.escript
The breaking node's mbdumpconfig.escript output is much shorter, and is missing the necessary buckets information.
The upgrade script (mbupgrade) is using mbdumpconfig.escript roughly like the following, in order to learn what buckets to upgrade (via mbdbupgrade)...
/opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat buckets membase
To get a full config.dat dump, you can do...
/opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat
When run on the breaking node, I see...
# /opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat | cat
[{nodes_wanted,['ns_1@10.1.4.99','ns_1@10.1.4.98']},
{otp,
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{cookie,mrqvjmauapsdnevj}]},
{directory,"/opt/membase/var/lib/membase/config"},
{{node,'ns_1@10.1.4.99',config_version},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]}|{1,7}]},
{{node,'ns_1@10.1.4.99',isasl},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},
{{node,'ns_1@10.1.4.99',memcached},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{port,11210},
{dbdir,"/opt/membase/var/lib/membase/data"},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},
{engines,
[{membase,
[{engine,"/opt/membase/lib/memcached/ep.so"},
{initfile,"/opt/membase/etc/membase/init.sql"},
{static_config_string,
"vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},
{memcached,
[{engine,"/opt/membase/lib/memcached/default_engine.so"},
{static_config_string,"vb0=true"}]}]},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.99',moxi},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{port,11211},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.99',ns_log},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{filename,"/opt/membase/var/lib/membase/data/ns_log"}]},
{{node,'ns_1@10.1.4.99',rest},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},{port,8091}]}]
The breaking node's mbdumpconfig.escript output is much shorter, and is missing the necessary buckets information.
The upgrade script (mbupgrade) is using mbdumpconfig.escript roughly like the following, in order to learn what buckets to upgrade (via mbdbupgrade)...
/opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat buckets membase
To get a full config.dat dump, you can do...
/opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat
When run on the breaking node, I see...
# /opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat | cat
[{nodes_wanted,['ns_1@10.1.4.99','ns_1@10.1.4.98']},
{otp,
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{cookie,mrqvjmauapsdnevj}]},
{directory,"/opt/membase/var/lib/membase/config"},
{{node,'ns_1@10.1.4.99',config_version},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]}|{1,7}]},
{{node,'ns_1@10.1.4.99',isasl},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},
{{node,'ns_1@10.1.4.99',memcached},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{port,11210},
{dbdir,"/opt/membase/var/lib/membase/data"},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},
{engines,
[{membase,
[{engine,"/opt/membase/lib/memcached/ep.so"},
{initfile,"/opt/membase/etc/membase/init.sql"},
{static_config_string,
"vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},
{memcached,
[{engine,"/opt/membase/lib/memcached/default_engine.so"},
{static_config_string,"vb0=true"}]}]},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.99',moxi},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{port,11211},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.99',ns_log},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{filename,"/opt/membase/var/lib/membase/data/ns_log"}]},
{{node,'ns_1@10.1.4.99',rest},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},{port,8091}]}]
Show
Steve Yen
added a comment - The two nodes are (strangely) returning different results for mbdumpconfig.escript
The breaking node's mbdumpconfig.escript output is much shorter, and is missing the necessary buckets information.
The upgrade script (mbupgrade) is using mbdumpconfig.escript roughly like the following, in order to learn what buckets to upgrade (via mbdbupgrade)...
/opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat buckets membase
To get a full config.dat dump, you can do...
/opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat
When run on the breaking node, I see...
# /opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat | cat
[{nodes_wanted,[' ns_1@10.1.4.99 ',' ns_1@10.1.4.98 ']},
{otp,
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},
{cookie,mrqvjmauapsdnevj}]},
{directory,"/opt/membase/var/lib/membase/config"},
{{node,' ns_1@10.1.4.99 ',config_version},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]}|{1,7}]},
{{node,' ns_1@10.1.4.99 ',isasl},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},
{path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},
{{node,' ns_1@10.1.4.99 ',memcached},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},
{port,11210},
{dbdir,"/opt/membase/var/lib/membase/data"},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},
{engines,
[{membase,
[{engine,"/opt/membase/lib/memcached/ep.so"},
{initfile,"/opt/membase/etc/membase/init.sql"},
{static_config_string,
"vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},
{memcached,
[{engine,"/opt/membase/lib/memcached/default_engine.so"},
{static_config_string,"vb0=true"}]}]},
{verbosity,[]}]},
{{node,' ns_1@10.1.4.99 ',moxi},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},
{port,11211},
{verbosity,[]}]},
{{node,' ns_1@10.1.4.99 ',ns_log},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},
{filename,"/opt/membase/var/lib/membase/data/ns_log"}]},
{{node,' ns_1@10.1.4.99 ',rest},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},{port,8091}]}]
Hide
Steve Yen
added a comment -
When run on the ok node, I see...
# /opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat
[{alerts,
[{email,[]},
{email_alerts,false},
{email_server,
[{user,undefined},
{pass,undefined},
{addr,undefined},
{port,undefined},
{encrypt,false}]},
{alerts,
[server_down,server_unresponsive,server_up,server_joined,
server_left,bucket_created,bucket_deleted,bucket_auth_failed]}]},
{auto_failover,[{enabled,false},{age,60},{max_nodes,1}]},
{buckets,
[{'_vclock',
[{'ns_1@10.1.4.98',{5,63480584070}},
{'ns_1@127.0.0.1',{2,63480583874}}]},
{configs,
[{"default",
[{num_replicas,1},
{ram_quota,314572800},
{auth_type,sasl},
{sasl_password,[]},
{type,membase},
{num_vbuckets,1024},
{servers,['ns_1@10.1.4.98','ns_1@10.1.4.99']},
{map,
[['ns_1@10.1.4.98','ns_1@10.1.4.99'],
['ns_1@10.1.4.98','ns_1@10.1.4.99'],
['ns_1@10.1.4.98','ns_1@10.1.4.99'],
['ns_1@10.1.4.98','ns_1@10.1.4.99'],
['ns_1@10.1.4.98','ns_1@10.1.4.99'],
['ns_1@10.1.4.98','ns_1@10.1.4.99'],
...snip...
['ns_1@10.1.4.99','ns_1@10.1.4.98'],
['ns_1@10.1.4.99','ns_1@10.1.4.98'],
['ns_1@10.1.4.99','ns_1@10.1.4.98'],
['ns_1@10.1.4.99','ns_1@10.1.4.98']]},
{fastForwardMap,undefined}]}]}]},
{directory,"/opt/membase/var/lib/membase/config"},
{memory_quota,[{'_vclock',[{'ns_1@127.0.0.1',{1,63480583874}}]}|300]},
{nodes_wanted,
[{'_vclock',[{'ns_1@10.1.4.98',{2,63480583964}}]},
'ns_1@10.1.4.98','ns_1@10.1.4.99']},
{otp,
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{cookie,mrqvjmauapsdnevj}]},
{port_servers,
[{moxi,"/opt/membase/bin/moxi",
["-Z",
{"port_listen=~B,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_max_errors=5,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200",
[port]},
"-z",
{"url=http://127.0.0.1:~B/pools/default/saslBucketsStreaming",
[{rest,port}]},
"-p","0","-Y","y","-O","stderr",
{"~s",[verbosity]}],
[{env,
[{"EVENT_NOSELECT","1"},
{"MOXI_SASL_PLAIN_USR",{"~s",[{ns_moxi_sup,rest_user,[]}]}},
{"MOXI_SASL_PLAIN_PWD",{"~s",[{ns_moxi_sup,rest_pass,[]}]}}]},
use_stdio,exit_status,port_server_send_eol,stderr_to_stdout,
stream]},
{memcached,"/opt/membase/bin/memcached",
["-X","/opt/membase/lib/memcached/stdin_term_handler.so","-p",
{"~B",[port]},
"-E","/opt/membase/lib/memcached/bucket_engine.so","-B","binary",
"-r","-c","10000","-e",
{"admin=~s;default_bucket_name=default;auto_create=false",
[admin_user]},
{"~s",[verbosity]}],
[{env,
[{"EVENT_NOSELECT","1"},
{"MEMCACHED_TOP_KEYS","100"},
{"ISASL_PWFILE",{"~s",[{isasl,path}]}},
{"ISASL_DB_CHECK_TIME","1"}]},
use_stdio,stderr_to_stdout,exit_status,port_server_send_eol,
stream]}]},
{rebalance_status,[{'_vclock',[{'ns_1@10.1.4.98',{1,63480584070}}]}|none]},
{replication,[{enabled,true}]},
{rest,[{port,8091}]},
{rest_creds,
[{'_vclock',[{'ns_1@127.0.0.1',{1,63480583874}}]},
{creds,[{"Administrator",[{password,"password"}]}]}]},
{uuid,<<"4252de8a-7faa-4001-15e8-014a000000b5">>},
{{node,'ns_1@10.1.4.98',config_version},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]}|{1,7}]},
{{node,'ns_1@10.1.4.98',isasl},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]},
{path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},
{{node,'ns_1@10.1.4.98',membership},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]}|active]},
{{node,'ns_1@10.1.4.98',memcached},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]},
{port,11210},
{dbdir,"/opt/membase/var/lib/membase/data"},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},
{engines,
[{membase,
[{engine,"/opt/membase/lib/memcached/ep.so"},
{initfile,"/opt/membase/etc/membase/init.sql"},
{static_config_string,
"vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},
{memcached,
[{engine,"/opt/membase/lib/memcached/default_engine.so"},
{static_config_string,"vb0=true"}]}]},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.98',moxi},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]},
{port,11211},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.98',ns_log},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]},
{filename,"/opt/membase/var/lib/membase/data/ns_log"}]},
{{node,'ns_1@10.1.4.98',rest},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]},{port,8091}]},
{{node,'ns_1@10.1.4.99',config_version},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]}|{1,7}]},
{{node,'ns_1@10.1.4.99',isasl},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},
{{node,'ns_1@10.1.4.99',membership},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583964}}]}|active]},
{{node,'ns_1@10.1.4.99',memcached},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{port,11210},
{dbdir,"/opt/membase/var/lib/membase/data"},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},
{engines,
[{membase,
[{engine,"/opt/membase/lib/memcached/ep.so"},
{initfile,"/opt/membase/etc/membase/init.sql"},
{static_config_string,
"vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},
{memcached,
[{engine,"/opt/membase/lib/memcached/default_engine.so"},
{static_config_string,"vb0=true"}]}]},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.99',moxi},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{port,11211},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.99',ns_log},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{filename,"/opt/membase/var/lib/membase/data/ns_log"}]},
{{node,'ns_1@10.1.4.99',rest},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},{port,8091}]}]
# /opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat
[{alerts,
[{email,[]},
{email_alerts,false},
{email_server,
[{user,undefined},
{pass,undefined},
{addr,undefined},
{port,undefined},
{encrypt,false}]},
{alerts,
[server_down,server_unresponsive,server_up,server_joined,
server_left,bucket_created,bucket_deleted,bucket_auth_failed]}]},
{auto_failover,[{enabled,false},{age,60},{max_nodes,1}]},
{buckets,
[{'_vclock',
[{'ns_1@10.1.4.98',{5,63480584070}},
{'ns_1@127.0.0.1',{2,63480583874}}]},
{configs,
[{"default",
[{num_replicas,1},
{ram_quota,314572800},
{auth_type,sasl},
{sasl_password,[]},
{type,membase},
{num_vbuckets,1024},
{servers,['ns_1@10.1.4.98','ns_1@10.1.4.99']},
{map,
[['ns_1@10.1.4.98','ns_1@10.1.4.99'],
['ns_1@10.1.4.98','ns_1@10.1.4.99'],
['ns_1@10.1.4.98','ns_1@10.1.4.99'],
['ns_1@10.1.4.98','ns_1@10.1.4.99'],
['ns_1@10.1.4.98','ns_1@10.1.4.99'],
['ns_1@10.1.4.98','ns_1@10.1.4.99'],
...snip...
['ns_1@10.1.4.99','ns_1@10.1.4.98'],
['ns_1@10.1.4.99','ns_1@10.1.4.98'],
['ns_1@10.1.4.99','ns_1@10.1.4.98'],
['ns_1@10.1.4.99','ns_1@10.1.4.98']]},
{fastForwardMap,undefined}]}]}]},
{directory,"/opt/membase/var/lib/membase/config"},
{memory_quota,[{'_vclock',[{'ns_1@127.0.0.1',{1,63480583874}}]}|300]},
{nodes_wanted,
[{'_vclock',[{'ns_1@10.1.4.98',{2,63480583964}}]},
'ns_1@10.1.4.98','ns_1@10.1.4.99']},
{otp,
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{cookie,mrqvjmauapsdnevj}]},
{port_servers,
[{moxi,"/opt/membase/bin/moxi",
["-Z",
{"port_listen=~B,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_max_errors=5,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200",
[port]},
"-z",
{"url=http://127.0.0.1:~B/pools/default/saslBucketsStreaming",
[{rest,port}]},
"-p","0","-Y","y","-O","stderr",
{"~s",[verbosity]}],
[{env,
[{"EVENT_NOSELECT","1"},
{"MOXI_SASL_PLAIN_USR",{"~s",[{ns_moxi_sup,rest_user,[]}]}},
{"MOXI_SASL_PLAIN_PWD",{"~s",[{ns_moxi_sup,rest_pass,[]}]}}]},
use_stdio,exit_status,port_server_send_eol,stderr_to_stdout,
stream]},
{memcached,"/opt/membase/bin/memcached",
["-X","/opt/membase/lib/memcached/stdin_term_handler.so","-p",
{"~B",[port]},
"-E","/opt/membase/lib/memcached/bucket_engine.so","-B","binary",
"-r","-c","10000","-e",
{"admin=~s;default_bucket_name=default;auto_create=false",
[admin_user]},
{"~s",[verbosity]}],
[{env,
[{"EVENT_NOSELECT","1"},
{"MEMCACHED_TOP_KEYS","100"},
{"ISASL_PWFILE",{"~s",[{isasl,path}]}},
{"ISASL_DB_CHECK_TIME","1"}]},
use_stdio,stderr_to_stdout,exit_status,port_server_send_eol,
stream]}]},
{rebalance_status,[{'_vclock',[{'ns_1@10.1.4.98',{1,63480584070}}]}|none]},
{replication,[{enabled,true}]},
{rest,[{port,8091}]},
{rest_creds,
[{'_vclock',[{'ns_1@127.0.0.1',{1,63480583874}}]},
{creds,[{"Administrator",[{password,"password"}]}]}]},
{uuid,<<"4252de8a-7faa-4001-15e8-014a000000b5">>},
{{node,'ns_1@10.1.4.98',config_version},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]}|{1,7}]},
{{node,'ns_1@10.1.4.98',isasl},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]},
{path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},
{{node,'ns_1@10.1.4.98',membership},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]}|active]},
{{node,'ns_1@10.1.4.98',memcached},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]},
{port,11210},
{dbdir,"/opt/membase/var/lib/membase/data"},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},
{engines,
[{membase,
[{engine,"/opt/membase/lib/memcached/ep.so"},
{initfile,"/opt/membase/etc/membase/init.sql"},
{static_config_string,
"vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},
{memcached,
[{engine,"/opt/membase/lib/memcached/default_engine.so"},
{static_config_string,"vb0=true"}]}]},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.98',moxi},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]},
{port,11211},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.98',ns_log},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]},
{filename,"/opt/membase/var/lib/membase/data/ns_log"}]},
{{node,'ns_1@10.1.4.98',rest},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583963}}]},{port,8091}]},
{{node,'ns_1@10.1.4.99',config_version},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]}|{1,7}]},
{{node,'ns_1@10.1.4.99',isasl},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},
{{node,'ns_1@10.1.4.99',membership},
[{'_vclock',[{'ns_1@10.1.4.98',{1,63480583964}}]}|active]},
{{node,'ns_1@10.1.4.99',memcached},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{port,11210},
{dbdir,"/opt/membase/var/lib/membase/data"},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},
{engines,
[{membase,
[{engine,"/opt/membase/lib/memcached/ep.so"},
{initfile,"/opt/membase/etc/membase/init.sql"},
{static_config_string,
"vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},
{memcached,
[{engine,"/opt/membase/lib/memcached/default_engine.so"},
{static_config_string,"vb0=true"}]}]},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.99',moxi},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{port,11211},
{verbosity,[]}]},
{{node,'ns_1@10.1.4.99',ns_log},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},
{filename,"/opt/membase/var/lib/membase/data/ns_log"}]},
{{node,'ns_1@10.1.4.99',rest},
[{'_vclock',[{'ns_1@10.1.4.99',{1,63480583964}}]},{port,8091}]}]
Show
Steve Yen
added a comment - When run on the ok node, I see...
# /opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat
[{alerts,
[{email,[]},
{email_alerts,false},
{email_server,
[{user,undefined},
{pass,undefined},
{addr,undefined},
{port,undefined},
{encrypt,false}]},
{alerts,
[server_down,server_unresponsive,server_up,server_joined,
server_left,bucket_created,bucket_deleted,bucket_auth_failed]}]},
{auto_failover,[{enabled,false},{age,60},{max_nodes,1}]},
{buckets,
[{'_vclock',
[{' ns_1@10.1.4.98 ',{5,63480584070}},
{' ns_1@127.0.0.1 ',{2,63480583874}}]},
{configs,
[{"default",
[{num_replicas,1},
{ram_quota,314572800},
{auth_type,sasl},
{sasl_password,[]},
{type,membase},
{num_vbuckets,1024},
{servers,[' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 ']},
{map,
[[' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],
[' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],
[' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],
[' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],
[' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],
[' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],
...snip...
[' ns_1@10.1.4.99 ',' ns_1@10.1.4.98 '],
[' ns_1@10.1.4.99 ',' ns_1@10.1.4.98 '],
[' ns_1@10.1.4.99 ',' ns_1@10.1.4.98 '],
[' ns_1@10.1.4.99 ',' ns_1@10.1.4.98 ']]},
{fastForwardMap,undefined}]}]}]},
{directory,"/opt/membase/var/lib/membase/config"},
{memory_quota,[{'_vclock',[{' ns_1@127.0.0.1 ',{1,63480583874}}]}|300]},
{nodes_wanted,
[{'_vclock',[{' ns_1@10.1.4.98 ',{2,63480583964}}]},
' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 ']},
{otp,
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},
{cookie,mrqvjmauapsdnevj}]},
{port_servers,
[{moxi,"/opt/membase/bin/moxi",
["-Z",
{"port_listen=~B,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_max_errors=5,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200",
[port]},
"-z",
{"url= http://127.0.0.1:~B/pools/default/saslBucketsStreaming ",
[{rest,port}]},
"-p","0","-Y","y","-O","stderr",
{"~s",[verbosity]}],
[{env,
[{"EVENT_NOSELECT","1"},
{"MOXI_SASL_PLAIN_USR",{"~s",[{ns_moxi_sup,rest_user,[]}]}},
{"MOXI_SASL_PLAIN_PWD",{"~s",[{ns_moxi_sup,rest_pass,[]}]}}]},
use_stdio,exit_status,port_server_send_eol,stderr_to_stdout,
stream]},
{memcached,"/opt/membase/bin/memcached",
["-X","/opt/membase/lib/memcached/stdin_term_handler.so","-p",
{"~B",[port]},
"-E","/opt/membase/lib/memcached/bucket_engine.so","-B","binary",
"-r","-c","10000","-e",
{"admin=~s;default_bucket_name=default;auto_create=false",
[admin_user]},
{"~s",[verbosity]}],
[{env,
[{"EVENT_NOSELECT","1"},
{"MEMCACHED_TOP_KEYS","100"},
{"ISASL_PWFILE",{"~s",[{isasl,path}]}},
{"ISASL_DB_CHECK_TIME","1"}]},
use_stdio,stderr_to_stdout,exit_status,port_server_send_eol,
stream]}]},
{rebalance_status,[{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480584070}}]}|none]},
{replication,[{enabled,true}]},
{rest,[{port,8091}]},
{rest_creds,
[{'_vclock',[{' ns_1@127.0.0.1 ',{1,63480583874}}]},
{creds,[{"Administrator",[{password,"password"}]}]}]},
{uuid,<<"4252de8a-7faa-4001-15e8-014a000000b5">>},
{{node,' ns_1@10.1.4.98 ',config_version},
[{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]}|{1,7}]},
{{node,' ns_1@10.1.4.98 ',isasl},
[{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]},
{path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},
{{node,' ns_1@10.1.4.98 ',membership},
[{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]}|active]},
{{node,' ns_1@10.1.4.98 ',memcached},
[{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]},
{port,11210},
{dbdir,"/opt/membase/var/lib/membase/data"},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},
{engines,
[{membase,
[{engine,"/opt/membase/lib/memcached/ep.so"},
{initfile,"/opt/membase/etc/membase/init.sql"},
{static_config_string,
"vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},
{memcached,
[{engine,"/opt/membase/lib/memcached/default_engine.so"},
{static_config_string,"vb0=true"}]}]},
{verbosity,[]}]},
{{node,' ns_1@10.1.4.98 ',moxi},
[{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]},
{port,11211},
{verbosity,[]}]},
{{node,' ns_1@10.1.4.98 ',ns_log},
[{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]},
{filename,"/opt/membase/var/lib/membase/data/ns_log"}]},
{{node,' ns_1@10.1.4.98 ',rest},
[{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]},{port,8091}]},
{{node,' ns_1@10.1.4.99 ',config_version},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]}|{1,7}]},
{{node,' ns_1@10.1.4.99 ',isasl},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},
{path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},
{{node,' ns_1@10.1.4.99 ',membership},
[{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583964}}]}|active]},
{{node,' ns_1@10.1.4.99 ',memcached},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},
{port,11210},
{dbdir,"/opt/membase/var/lib/membase/data"},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},
{engines,
[{membase,
[{engine,"/opt/membase/lib/memcached/ep.so"},
{initfile,"/opt/membase/etc/membase/init.sql"},
{static_config_string,
"vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},
{memcached,
[{engine,"/opt/membase/lib/memcached/default_engine.so"},
{static_config_string,"vb0=true"}]}]},
{verbosity,[]}]},
{{node,' ns_1@10.1.4.99 ',moxi},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},
{port,11211},
{verbosity,[]}]},
{{node,' ns_1@10.1.4.99 ',ns_log},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},
{filename,"/opt/membase/var/lib/membase/data/ns_log"}]},
{{node,' ns_1@10.1.4.99 ',rest},
[{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},{port,8091}]}]
Hide
Steve Yen
added a comment -
Are we sure we only see this on ubuntu 64?
Given what I know so far, I'd think we should also see this on 32-bit and other platforms, and it might be a ns_config issue in 1.7.0?
More notes...
Install 1.7.0 (http://builds.hq.northscale.net/releases/1.7.0/) on two nodes. Join them. Shut down the nodes. On each node, run...
/opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat
Output from erlwsh/shell ns_config:get(), however, does (correctly) show the same ns_config contents.
Given what I know so far, I'd think we should also see this on 32-bit and other platforms, and it might be a ns_config issue in 1.7.0?
More notes...
Install 1.7.0 (http://builds.hq.northscale.net/releases/1.7.0/) on two nodes. Join them. Shut down the nodes. On each node, run...
/opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat
Output from erlwsh/shell ns_config:get(), however, does (correctly) show the same ns_config contents.
Show
Steve Yen
added a comment - Are we sure we only see this on ubuntu 64?
Given what I know so far, I'd think we should also see this on 32-bit and other platforms, and it might be a ns_config issue in 1.7.0?
More notes...
Install 1.7.0 ( http://builds.hq.northscale.net/releases/1.7.0/ ) on two nodes. Join them. Shut down the nodes. On each node, run...
/opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat
Output from erlwsh/shell ns_config:get(), however, does (correctly) show the same ns_config contents.
Hide
Aleksey Kondratenko
added a comment -
fix is coming. But _before_ upgrading we'll need to force saving config on all nodes. Users will need to run:
rpc:eval_everywhere(ns_config, resave, []).
via erlwsh or via /diag/eval:
wget -O- -q --post-data='rpc:eval_everywhere(ns_config, resave, []).' --user=Administrator --password=asdasd http://dn2:8091/diag/eval
rpc:eval_everywhere(ns_config, resave, []).
via erlwsh or via /diag/eval:
wget -O- -q --post-data='rpc:eval_everywhere(ns_config, resave, []).' --user=Administrator --password=asdasd http://dn2:8091/diag/eval
Show
Aleksey Kondratenko
added a comment - fix is coming. But _before_ upgrading we'll need to force saving config on all nodes. Users will need to run:
rpc:eval_everywhere(ns_config, resave, []).
via erlwsh or via /diag/eval:
wget -O- -q --post-data='rpc:eval_everywhere(ns_config, resave, []).' --user=Administrator --password=asdasd http://dn2:8091/diag/eval
Show
Aleksey Kondratenko
added a comment - http://review.couchbase.org/8946
Hide
Karan Kumar
added a comment -
Aleksey,
How can we verify if this command has actually executed successfully. I want to verify if its completely fixed in 1.7.1.1
How can we verify if this command has actually executed successfully. I want to verify if its completely fixed in 1.7.1.1
Show
Karan Kumar
added a comment - Aleksey,
How can we verify if this command has actually executed successfully. I want to verify if its completely fixed in 1.7.1.1
Hide
Farshid Ghods
added a comment -
Integrated in single-node-windows-64-install #296 (See [http://qa.hq.northscale.net/job/single-node-windows-64-install/296/])
Force saving config before upgrade (MB-4186)
Farshid Ghods :
Files :
* pytests/upgradetests.py
Force saving config before upgrade (
Farshid Ghods :
Files :
* pytests/upgradetests.py
Show
Farshid Ghods
added a comment - Integrated in single-node-windows-64-install #296 (See [ http://qa.hq.northscale.net/job/single-node-windows-64-install/296/ ])
Force saving config before upgrade ( MB-4186 )
Farshid Ghods :
Files :
* pytests/upgradetests.py
Hide
Farshid Ghods
added a comment -
Integrated in github-ns-server-2-0 #117 (See [http://qa.hq.northscale.net/job/github-ns-server-2-0/117/])
save config after config cas.MB-4186
Steve Yen :
Files :
* src/ns_config.erl
save config after config cas.
Steve Yen :
Files :
* src/ns_config.erl
Show
Farshid Ghods
added a comment - Integrated in github-ns-server-2-0 #117 (See [ http://qa.hq.northscale.net/job/github-ns-server-2-0/117/ ])
save config after config cas. MB-4186
Steve Yen :
Files :
* src/ns_config.erl
Hide
Aleksey Kondratenko
added a comment -
Karan, verifying can be done by looking at config.dat size. After forced saving it should grow.
Show
Aleksey Kondratenko
added a comment - Karan, verifying can be done by looking at config.dat size. After forced saving it should grow.
Show
Aleksey Kondratenko
added a comment - seems to be fixed long time ago
Show
Karan Kumar
added a comment - Verified.
dpkg -r membase-server || true
rm -rf /opt/membase
dpkg -i $1
if [ "$2" = "" ]; then
/opt/membase/bin/membase cluster-init -c 127.0.0.1:8091 \
--cluster-init-username=Administrator \
--cluster-init-password=password \
--cluster-init-ramsize=300
/opt/membase/bin/membase bucket-create -c 127.0.0.1:8091 \
-u Administrator -p password \
--bucket=default \
--bucket-type=membase \
--bucket-port=11211 \
--bucket-password= \
--bucket-ramsize=300 \
--bucket-replica=1
else
/opt/membase/bin/membase rebalance -c $2 \
-u Administrator -p password \
--server-add=`/sbin/ifconfig eth0|grep inet|awk {'print $2'}|cut -d":" -f2|head -n 1`
fi