Upgrade from 1.7.0 to 1.7.1.1(1.7.1-5) fails with bucket_escript expection

Steve Yen added a comment - 14/Aug/11 6:36 PM #!/bin/bash dpkg -r membase-server || true rm -rf /opt/membase dpkg -i $1 if [ "$2" = "" ]; then    /opt/membase/bin/membase cluster-init -c 127.0.0.1:8091 \       --cluster-init-username=Administrator \       --cluster-init-password=password \       --cluster-init-ramsize=300    /opt/membase/bin/membase bucket-create -c 127.0.0.1:8091 \       -u Administrator -p password \       --bucket=default \       --bucket-type=membase \       --bucket-port=11211 \       --bucket-password= \       --bucket-ramsize=300 \       --bucket-replica=1 else    /opt/membase/bin/membase rebalance -c $2 \       -u Administrator -p password \       --server-add=`/sbin/ifconfig eth0|grep inet|awk {'print $2'}|cut -d":" -f2|head -n 1` fi

Steve Yen added a comment - 14/Aug/11 6:46 PM The two nodes are (strangely) returning different results for mbdumpconfig.escript The breaking node's mbdumpconfig.escript output is much shorter, and is missing the necessary buckets information. The upgrade script (mbupgrade) is using mbdumpconfig.escript roughly like the following, in order to learn what buckets to upgrade (via mbdbupgrade)...   /opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat buckets membase To get a full config.dat dump, you can do...   /opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat When run on the breaking node, I see... # /opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat | cat [{nodes_wanted,[' ns_1@10.1.4.99 ',' ns_1@10.1.4.98 ']},  {otp,      [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},       {cookie,mrqvjmauapsdnevj}]},  {directory,"/opt/membase/var/lib/membase/config"},  {{node,' ns_1@10.1.4.99 ',config_version},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]}|{1,7}]},  {{node,' ns_1@10.1.4.99 ',isasl},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},    {path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},  {{node,' ns_1@10.1.4.99 ',memcached},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},    {port,11210},    {dbdir,"/opt/membase/var/lib/membase/data"},    {admin_user,"_admin"},    {admin_pass,"_admin"},    {bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},    {engines,        [{membase,             [{engine,"/opt/membase/lib/memcached/ep.so"},              {initfile,"/opt/membase/etc/membase/init.sql"},              {static_config_string,                  "vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},         {memcached,             [{engine,"/opt/membase/lib/memcached/default_engine.so"},              {static_config_string,"vb0=true"}]}]},    {verbosity,[]}]},  {{node,' ns_1@10.1.4.99 ',moxi},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},    {port,11211},    {verbosity,[]}]},  {{node,' ns_1@10.1.4.99 ',ns_log},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},    {filename,"/opt/membase/var/lib/membase/data/ns_log"}]},  {{node,' ns_1@10.1.4.99 ',rest},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},{port,8091}]}]

Steve Yen added a comment - 14/Aug/11 6:48 PM When run on the ok node, I see... # /opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat [{alerts,      [{email,[]},       {email_alerts,false},       {email_server,           [{user,undefined},            {pass,undefined},            {addr,undefined},            {port,undefined},            {encrypt,false}]},       {alerts,           [server_down,server_unresponsive,server_up,server_joined,            server_left,bucket_created,bucket_deleted,bucket_auth_failed]}]},  {auto_failover,[{enabled,false},{age,60},{max_nodes,1}]},  {buckets,      [{'_vclock',           [{' ns_1@10.1.4.98 ',{5,63480584070}},            {' ns_1@127.0.0.1 ',{2,63480583874}}]},       {configs,           [{"default",             [{num_replicas,1},              {ram_quota,314572800},              {auth_type,sasl},              {sasl_password,[]},              {type,membase},              {num_vbuckets,1024},              {servers,[' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 ']},              {map,                  [[' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],                   [' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],                   [' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],                   [' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],                   [' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],                   [' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 '],   ...snip...                   [' ns_1@10.1.4.99 ',' ns_1@10.1.4.98 '],                   [' ns_1@10.1.4.99 ',' ns_1@10.1.4.98 '],                   [' ns_1@10.1.4.99 ',' ns_1@10.1.4.98 '],                   [' ns_1@10.1.4.99 ',' ns_1@10.1.4.98 ']]},              {fastForwardMap,undefined}]}]}]},  {directory,"/opt/membase/var/lib/membase/config"},  {memory_quota,[{'_vclock',[{' ns_1@127.0.0.1 ',{1,63480583874}}]}|300]},  {nodes_wanted,      [{'_vclock',[{' ns_1@10.1.4.98 ',{2,63480583964}}]},       ' ns_1@10.1.4.98 ',' ns_1@10.1.4.99 ']},  {otp,      [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},       {cookie,mrqvjmauapsdnevj}]},  {port_servers,      [{moxi,"/opt/membase/bin/moxi",           ["-Z",            {"port_listen=~B,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_max_errors=5,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200",             [port]},            "-z",            {"url= http://127.0.0.1:~B/pools/default/saslBucketsStreaming ",             [{rest,port}]},            "-p","0","-Y","y","-O","stderr",            {"~s",[verbosity]}],           [{env,                [{"EVENT_NOSELECT","1"},                 {"MOXI_SASL_PLAIN_USR",{"~s",[{ns_moxi_sup,rest_user,[]}]}},                 {"MOXI_SASL_PLAIN_PWD",{"~s",[{ns_moxi_sup,rest_pass,[]}]}}]},            use_stdio,exit_status,port_server_send_eol,stderr_to_stdout,            stream]},       {memcached,"/opt/membase/bin/memcached",           ["-X","/opt/membase/lib/memcached/stdin_term_handler.so","-p",            {"~B",[port]},            "-E","/opt/membase/lib/memcached/bucket_engine.so","-B","binary",            "-r","-c","10000","-e",            {"admin=~s;default_bucket_name=default;auto_create=false",             [admin_user]},            {"~s",[verbosity]}],           [{env,                [{"EVENT_NOSELECT","1"},                 {"MEMCACHED_TOP_KEYS","100"},                 {"ISASL_PWFILE",{"~s",[{isasl,path}]}},                 {"ISASL_DB_CHECK_TIME","1"}]},            use_stdio,stderr_to_stdout,exit_status,port_server_send_eol,            stream]}]},  {rebalance_status,[{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480584070}}]}|none]},  {replication,[{enabled,true}]},  {rest,[{port,8091}]},  {rest_creds,      [{'_vclock',[{' ns_1@127.0.0.1 ',{1,63480583874}}]},       {creds,[{"Administrator",[{password,"password"}]}]}]},  {uuid,<<"4252de8a-7faa-4001-15e8-014a000000b5">>},  {{node,' ns_1@10.1.4.98 ',config_version},   [{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]}|{1,7}]},  {{node,' ns_1@10.1.4.98 ',isasl},   [{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]},    {path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},  {{node,' ns_1@10.1.4.98 ',membership},   [{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]}|active]},  {{node,' ns_1@10.1.4.98 ',memcached},   [{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]},    {port,11210},    {dbdir,"/opt/membase/var/lib/membase/data"},    {admin_user,"_admin"},    {admin_pass,"_admin"},    {bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},    {engines,        [{membase,             [{engine,"/opt/membase/lib/memcached/ep.so"},              {initfile,"/opt/membase/etc/membase/init.sql"},              {static_config_string,                  "vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},         {memcached,             [{engine,"/opt/membase/lib/memcached/default_engine.so"},              {static_config_string,"vb0=true"}]}]},    {verbosity,[]}]},  {{node,' ns_1@10.1.4.98 ',moxi},   [{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]},    {port,11211},    {verbosity,[]}]},  {{node,' ns_1@10.1.4.98 ',ns_log},   [{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]},    {filename,"/opt/membase/var/lib/membase/data/ns_log"}]},  {{node,' ns_1@10.1.4.98 ',rest},   [{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583963}}]},{port,8091}]},  {{node,' ns_1@10.1.4.99 ',config_version},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]}|{1,7}]},  {{node,' ns_1@10.1.4.99 ',isasl},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},    {path,"/opt/membase/var/lib/membase/data/isasl.pw"}]},  {{node,' ns_1@10.1.4.99 ',membership},   [{'_vclock',[{' ns_1@10.1.4.98 ',{1,63480583964}}]}|active]},  {{node,' ns_1@10.1.4.99 ',memcached},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},    {port,11210},    {dbdir,"/opt/membase/var/lib/membase/data"},    {admin_user,"_admin"},    {admin_pass,"_admin"},    {bucket_engine,"/opt/membase/lib/memcached/bucket_engine.so"},    {engines,        [{membase,             [{engine,"/opt/membase/lib/memcached/ep.so"},              {initfile,"/opt/membase/etc/membase/init.sql"},              {static_config_string,                  "vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},         {memcached,             [{engine,"/opt/membase/lib/memcached/default_engine.so"},              {static_config_string,"vb0=true"}]}]},    {verbosity,[]}]},  {{node,' ns_1@10.1.4.99 ',moxi},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},    {port,11211},    {verbosity,[]}]},  {{node,' ns_1@10.1.4.99 ',ns_log},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},    {filename,"/opt/membase/var/lib/membase/data/ns_log"}]},  {{node,' ns_1@10.1.4.99 ',rest},   [{'_vclock',[{' ns_1@10.1.4.99 ',{1,63480583964}}]},{port,8091}]}]

Steve Yen added a comment - 14/Aug/11 6:54 PM The good node is 10.1.4.98. The breaking node is 10.1.4.99.

Steve Yen added a comment - 14/Aug/11 7:14 PM Are we sure we only see this on ubuntu 64? Given what I know so far, I'd think we should also see this on 32-bit and other platforms, and it might be a ns_config issue in 1.7.0? More notes... Install 1.7.0 ( http://builds.hq.northscale.net/releases/1.7.0/ ) on two nodes. Join them. Shut down the nodes. On each node, run...   /opt/membase/bin/escript /opt/membase/bin/mbdumpconfig.escript /opt/membase/var/lib/membase/config/config.dat Output from erlwsh/shell ns_config:get(), however, does (correctly) show the same ns_config contents.

Aleksey Kondratenko added a comment - 15/Aug/11 6:35 AM fix is coming. But _before_ upgrading we'll need to force saving config on all nodes. Users will need to run: rpc:eval_everywhere(ns_config, resave, []). via erlwsh or via /diag/eval: wget -O- -q --post-data='rpc:eval_everywhere(ns_config, resave, []).' --user=Administrator --password=asdasd http://dn2:8091/diag/eval

Aleksey Kondratenko added a comment - 15/Aug/11 6:48 AM http://review.couchbase.org/8946

Karan Kumar added a comment - 16/Aug/11 11:28 PM Aleksey, How can we verify if this command has actually executed successfully. I want to verify if its completely fixed in 1.7.1.1

Farshid Ghods added a comment - 17/Aug/11 4:58 PM Integrated in single-node-windows-64-install #296 (See [ http://qa.hq.northscale.net/job/single-node-windows-64-install/296/ ])     Force saving config before upgrade ( MB-4186 ) Farshid Ghods : Files : * pytests/upgradetests.py

Farshid Ghods added a comment - 23/Aug/11 2:36 AM Integrated in github-ns-server-2-0 #117 (See [ http://qa.hq.northscale.net/job/github-ns-server-2-0/117/ ])     save config after config cas. MB-4186 Steve Yen : Files : * src/ns_config.erl

Aleksey Kondratenko added a comment - 29/Nov/11 11:08 AM Karan, verifying can be done by looking at config.dat size. After forced saving it should grow.

Aleksey Kondratenko added a comment - 29/Nov/11 11:09 AM seems to be fixed long time ago

Karan Kumar added a comment - 12/Dec/11 4:19 PM Verified.