erlang's global naming facility apparently drops globally registered service with actual service still alive (was: impossible to change settings/autoFailover after rebalance)

Details

Type: Bug
Status: Open
Priority: Blocker
Resolution: Unresolved
Affects Version/s: 2.0
Fix Version/s: .major-release
Component/s: ns_server
Security Level: Public
Labels:
None

Operating System:
Centos 64-bit
Flagged:

Release Note

Description

build
./testrunner -i resources/jenkins/centos-64-5node-failover.ini get-logs=True -t autofailovertests.AutoFailoverTests.test_enable,replicas=2,keys-count=1000000,num-buckets=2
http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/480/consoleFull

test changes settings/autoFailover after rebalance

test logs:

2012-11-27 22:29:42,105 - INFO - MainProcess:MainThread - rest_client:_rebalance_progress - rebalance percentage : 99.5678770149 %
2012-11-27 22:29:44,114 - INFO - MainProcess:MainThread - rest_client:_rebalance_progress - rebalance percentage : 99.6821896236 %
2012-11-27 22:29:48,134 - INFO - MainProcess:MainThread - rest_client:monitorRebalance - rebalance progress took 1111.69762397 seconds
2012-11-27 22:29:48,135 - INFO - MainProcess:MainThread - rest_client:monitorRebalance - sleep for 10 seconds after rebalance...
2012-11-27 22:29:58,133 - INFO - MainProcess:MainThread - rest_client:update_autofailover_settings - settings/autoFailover params : enabled=true&timeout=30
2012-11-27 22:29:58,140 - ERROR - MainProcess:MainThread - rest_client:_http_request - http://10.1.3.114:8091/settings/autoFailover error 500 reason: unknown ["Unexpected server error, request logged."]

server logs:

[menelaus:warn,2012-11-27T22:53:28.404,ns_1@10.1.3.114:<0.21541.23>:menelaus_web:loop:430]Server error during processing: ["web request failed",
                                 {path,"/settings/autoFailover"},
                                 {type,exit},
                                 {what,
                                  {noproc,
                                   {gen_server,call,
                                    [{global,auto_failover},
                                     {enable_auto_failover,30,1}]}}},
                                 {trace,
                                  [{gen_server,call,2},
                                   {menelaus_web,
                                    handle_settings_auto_failover_post,1},
                                   {menelaus_web,loop,3},
                                   {mochiweb_http,headers,5},
                                   {proc_lib,init_p_do_apply,3}]}]

Alk, if we have to handle the error and retry again, please assign the ticket back to me

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Andrei Baranouski added a comment - 28/Nov/12 9:01 AM

https://s3.amazonaws.com/bugdb/jira/MB-7282/350d5885-93c9-4497-bc64-fd5c65d4c162-10.1.3.114-diag.txt.gz

Show

Andrei Baranouski added a comment - 28/Nov/12 9:01 AM https://s3.amazonaws.com/bugdb/jira/MB-7282/350d5885-93c9-4497-bc64-fd5c65d4c162-10.1.3.114-diag.txt.gz

Hide

Permalink

Farshid Ghods added a comment - 28/Nov/12 9:06 AM

does it work when you retry ?

Show

Farshid Ghods added a comment - 28/Nov/12 9:06 AM does it work when you retry ?

Hide

Permalink

Steve Yen added a comment - 28/Nov/12 1:18 PM

per bug-scrub, assigning back to andrei.

please try to reproduce it again (manually), and does it succeed if you retry to operation?

Show

Steve Yen added a comment - 28/Nov/12 1:18 PM per bug-scrub, assigning back to andrei. please try to reproduce it again (manually), and does it succeed if you retry to operation?

Hide

Permalink

Andrei Baranouski added a comment - 29/Nov/12 12:36 PM

I was not able to reproduce it manually, even with constant change autoFailover settings through script during the test. but the same problem was in 2.0.0-1965-rel run http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/476/consoleFull

Show

Andrei Baranouski added a comment - 29/Nov/12 12:36 PM I was not able to reproduce it manually, even with constant change autoFailover settings through script during the test. but the same problem was in 2.0.0-1965-rel run http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/476/consoleFull

Hide

Permalink

Steve Yen added a comment - 29/Nov/12 1:19 PM

per-bug-scrub - moved to 2.0.1.

apparently hard to reproduce manually.

Show

Steve Yen added a comment - 29/Nov/12 1:19 PM per-bug-scrub - moved to 2.0.1. apparently hard to reproduce manually.

Hide

Permalink

Farshid Ghods added a comment - 10/Dec/12 11:30 AM

deferring to 2.1 per bug scrub meeting ( Dipti & Farshid -December 7th )

Show

Farshid Ghods added a comment - 10/Dec/12 11:30 AM deferring to 2.1 per bug scrub meeting ( Dipti & Farshid -December 7th )

Hide

Permalink

Andrei Baranouski added a comment - 17/Dec/12 3:51 AM

reproduced again in some test against 2.0.1-103-rel build:

http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/491/consoleFull

Show

Andrei Baranouski added a comment - 17/Dec/12 3:51 AM reproduced again in some test against 2.0.1-103-rel build: http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/491/consoleFull

Hide

Permalink

Andrei Baranouski added a comment - 17/Dec/12 4:23 AM

Alk, if you have a chance , could you look today on 10.1.3.114 from the test, maybe this will help to quickly identify the problem:

resetCount( or update settings/autoFailover) started giving an 500 error after a certain point in the tests

curl -v --user Administrator:password -X POST http://10.1.3.114:8091/settings/autoFailover/resetCount
* About to connect() to 10.1.3.114 port 8091 (#0)
* Trying 10.1.3.114... connected
* Connected to 10.1.3.114 (10.1.3.114) port 8091 (#0)
* Server auth using Basic with user 'Administrator'
> POST /settings/autoFailover/resetCount HTTP/1.1
> Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==
> User-Agent: curl/7.21.4 (x86_64-unknown-linux-gnu) libcurl/7.21.4 OpenSSL/1.0.1 zlib/1.2.3.4
> Host: 10.1.3.114:8091
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Server: Couchbase Server 2.0.1-103-rel-enterprise
< Pragma: no-cache
< Date: Mon, 17 Dec 2012 10:14:05 GMT
< Content-Type: application/json
< Content-Length: 44
< Cache-Control: no-cache
<
* Connection #0 to host 10.1.3.114 left intact
* Closing connection #0

Show

Andrei Baranouski added a comment - 17/Dec/12 4:23 AM Alk, if you have a chance , could you look today on 10.1.3.114 from the test, maybe this will help to quickly identify the problem: resetCount( or update settings/autoFailover) started giving an 500 error after a certain point in the tests curl -v --user Administrator:password -X POST http://10.1.3.114:8091/settings/autoFailover/resetCount * About to connect() to 10.1.3.114 port 8091 (#0) * Trying 10.1.3.114... connected * Connected to 10.1.3.114 (10.1.3.114) port 8091 (#0) * Server auth using Basic with user 'Administrator' > POST /settings/autoFailover/resetCount HTTP/1.1 > Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA== > User-Agent: curl/7.21.4 (x86_64-unknown-linux-gnu) libcurl/7.21.4 OpenSSL/1.0.1 zlib/1.2.3.4 > Host: 10.1.3.114:8091 > Accept: */* > < HTTP/1.1 500 Internal Server Error < Server: Couchbase Server 2.0.1-103-rel-enterprise < Pragma: no-cache < Date: Mon, 17 Dec 2012 10:14:05 GMT < Content-Type: application/json < Content-Length: 44 < Cache-Control: no-cache < * Connection #0 to host 10.1.3.114 left intact * Closing connection #0

Hide

Permalink

Aleksey Kondratenko added a comment - 17/Dec/12 2:54 PM

Indeed a bug.

Looks like you rebalanced out master. And new master needs some time to be elected. Thus autofailover service is indeed not running anywhere.

Show

Aleksey Kondratenko added a comment - 17/Dec/12 2:54 PM Indeed a bug. Looks like you rebalanced out master. And new master needs some time to be elected. Thus autofailover service is indeed not running anywhere.

Hide

Permalink

Aleksey Kondratenko added a comment - 17/Dec/12 2:55 PM

As workaround I suggest you to wait a bit and retry. Or consider not doing this POST at all.

Show

Aleksey Kondratenko added a comment - 17/Dec/12 2:55 PM As workaround I suggest you to wait a bit and retry. Or consider not doing this POST at all.

Hide

Permalink

Andrei Baranouski added a comment - 18/Dec/12 2:39 AM

retry doesn't help here

Show

Andrei Baranouski added a comment - 18/Dec/12 2:39 AM retry doesn't help here

Hide

Permalink

Aleksey Kondratenko added a comment - 18/Dec/12 11:33 AM

May I see:

* code that does retry

* logs from that attempt where retry didn't help

Show

Aleksey Kondratenko added a comment - 18/Dec/12 11:33 AM May I see: * code that does retry * logs from that attempt where retry didn't help

Hide

Permalink

Aleksey Kondratenko added a comment - 18/Dec/12 11:34 AM

See above

Show

Aleksey Kondratenko added a comment - 18/Dec/12 11:34 AM See above

Hide

Permalink

Andrei Baranouski added a comment - 19/Dec/12 7:33 AM

now we don't use retries in our tests for such cases.
results from command POST above were gotten when suites failed with the same error and I tried to check it through curl. Ie was a long time since the server was idle, before I did this.
can't provide more info now because new tests were triggered.

Show

Andrei Baranouski added a comment - 19/Dec/12 7:33 AM now we don't use retries in our tests for such cases. results from command POST above were gotten when suites failed with the same error and I tried to check it through curl. Ie was a long time since the server was idle, before I did this. can't provide more info now because new tests were triggered.

Hide

Permalink

Aleksey Kondratenko added a comment - 19/Dec/12 7:20 PM

Ok, so there's possibly two bugs here. But I'll need logs for that case where it failed even long time after rebalance.

Show

Aleksey Kondratenko added a comment - 19/Dec/12 7:20 PM Ok, so there's possibly two bugs here. But I'll need logs for that case where it failed even long time after rebalance.

Hide

Permalink

Karen Zeller added a comment - 11/Jan/13 4:24 PM

Nominating for 2.0.1 RN

Show

Karen Zeller added a comment - 11/Jan/13 4:24 PM Nominating for 2.0.1 RN

Hide

Permalink

Aleksey Kondratenko added a comment - 11/Jan/13 4:38 PM

Nominating ?

Show

Aleksey Kondratenko added a comment - 11/Jan/13 4:38 PM Nominating ?

Hide

Permalink

Andrei Baranouski added a comment - 04/Feb/13 7:24 AM

reproduced again on build 147:
http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/529/consoleFull

all logs:
https://s3.amazonaws.com/bugdb/jira/MB-7282/529_run_logs.zip
or
http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/529/artifact/*zip*/archive.zip

Show

Andrei Baranouski added a comment - 04/Feb/13 7:24 AM reproduced again on build 147: http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/529/consoleFull all logs: https://s3.amazonaws.com/bugdb/jira/MB-7282/529_run_logs.zip or http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/529/artifact/*zip*/archive.zip

Hide

Permalink

Andrei Baranouski added a comment - 18/Feb/13 2:41 AM

build 159 http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/546/consoleFull

Show

Andrei Baranouski added a comment - 18/Feb/13 2:41 AM build 159 http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/546/consoleFull

Hide

Permalink

Andrei Baranouski added a comment - 25/Feb/13 3:31 AM

build 164 too

Show

Andrei Baranouski added a comment - 25/Feb/13 3:31 AM build 164 too

Hide

Permalink

Aleksey Kondratenko added a comment - 08/Apr/13 8:50 PM

Indeed a problem.

Appears to be something with global name service because autofailover service _is_ running we're just somehow unable to locate it.

Which would be a very bad news because that component is not directly our code but part of erlang's stdlib.

Show

Aleksey Kondratenko added a comment - 08/Apr/13 8:50 PM Indeed a problem. Appears to be something with global name service because autofailover service _is_ running we're just somehow unable to locate it. Which would be a very bad news because that component is not directly our code but part of erlang's stdlib.

Hide

Permalink

Aleksey Kondratenko added a comment - 08/Apr/13 8:50 PM

Raised to blocker as global component problem implies possibility of much worse issues.

Show

Aleksey Kondratenko added a comment - 08/Apr/13 8:50 PM Raised to blocker as global component problem implies possibility of much worse issues.

Hide

Permalink

Aleksey Kondratenko added a comment - 08/Apr/13 8:53 PM

rebalance is likely unrelated here. What is potentially related is other nodes connecting/disconnecting.

Show

Aleksey Kondratenko added a comment - 08/Apr/13 8:53 PM rebalance is likely unrelated here. What is potentially related is other nodes connecting/disconnecting.

Hide

Permalink

Andrei Baranouski added a comment - 25/Apr/13 2:34 PM

http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/596/consoleFull
2.0.2-773-rel

Show

Andrei Baranouski added a comment - 25/Apr/13 2:34 PM http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/596/consoleFull 2.0.2-773-rel

Hide

Permalink

Aleksey Kondratenko added a comment - 01/May/13 4:26 PM

See also http://www.couchbase.com/issues/browse/MB-7967

Show

Aleksey Kondratenko added a comment - 01/May/13 4:26 PM See also http://www.couchbase.com/issues/browse/MB-7967

Hide

Permalink

Aleksey Kondratenko added a comment - 01/May/13 4:27 PM

Workaround:

wget --user=Administrator --password=asdasd --post-data='rpc:call(mb_master:master_node(), erlang, apply ,[fun () -> erlang:exit(erlang:whereis(mb_master), kill) end, []]).' http://localhost:8091/diag/eval

Show

Aleksey Kondratenko added a comment - 01/May/13 4:27 PM Workaround: wget --user=Administrator --password=asdasd --post-data='rpc:call(mb_master:master_node(), erlang, apply ,[fun () -> erlang:exit(erlang:whereis(mb_master), kill) end, []]).' http://localhost:8091/diag/eval

People

Assignee:

Aleksey Kondratenko

Reporter:

Andrei Baranouski

Vote (0)

Watch (3)

Dates

Created:

28/Nov/12 8:56 AM

Updated:

02/May/13 5:31 PM

Agile

View on Board