Couchbase Server

rebalance-in fails on ec2: IP address seems to have changed. Unable to listen on 'ns_1@ec2-184-73-53-179.compute-1.amazonaws.com'.

Details

  • Type: Technical task Technical task
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Duplicate
  • Affects Version/s: 2.0-beta-2
  • Fix Version/s: 2.0.1
  • Component/s: ns_server
  • Security Level: Public
  • Labels:
    None
  • Environment:
    build 1940
    ec2 m2.2xlarge, 3->4 nodes, ubuntu 12
    10M x 2KB items + views

Description

Later saw "Resetting rebalance status since it's not really running" and "Fail Over Warning: Rebalance recommended, some data does not have the desired number of replicas!"

I tried to rebalance again and got:

Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.29378.114>,
{{{{badmatch,
{error,
{error,
<<"Partition 279 not in active nor passive set">>}}},
[{capi_set_view_manager,handle_call,3},
{gen_server,handle_msg,5},
{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]},
{gen_server,call,
['capi_set_view_manager-default',
{wait_index_updated,279},
infinity]}},
{gen_server,call,
[{'janitor_agent-default',
'ns_1@ec2-54-242-160-13.compute-1.amazonaws.com'},
{if_rebalance,<0.29306.114>,
{wait_index_updated,279}},
infinity]}}}}

Probably duplicate of MB-7115 (update: alk says not likely a dupe of MB-7115 (which is emfile related bug))

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Pavel Paulau added a comment -
When I manually clicked Rebalance button third time it managed to rebalance cluster.
Show
Pavel Paulau added a comment - When I manually clicked Rebalance button third time it managed to rebalance cluster.
Hide
Permalink
Steve Yen added a comment -
in bug-scrub mtg: still in investigation by alk.
Show
Steve Yen added a comment - in bug-scrub mtg: still in investigation by alk.
Hide
Permalink
Aleksey Kondratenko added a comment -
unable to listen is apparently problem (perhaps timeout) of dns. Because part of doing that check is running hostname through resolver.

Logs contain weird mb_master timeouts and re-elections.

I suspect network congestion because everything else looks quite healthy. No signs of swapping/paging. In fact there's lots of ram left on most nodes with notable exception of .13.

memcached requests occasionally take more than 500 milliseconds but rarely above 1 second, while we're seeing stats collection ticks to be dropped. Even for system stats collection. Which is best explained by troubles receiving those ticks from master node (which is 'wandering' accross cluster).

Need to rerun with atop data gathering.
Show
Aleksey Kondratenko added a comment - unable to listen is apparently problem (perhaps timeout) of dns. Because part of doing that check is running hostname through resolver. Logs contain weird mb_master timeouts and re-elections. I suspect network congestion because everything else looks quite healthy. No signs of swapping/paging. In fact there's lots of ram left on most nodes with notable exception of .13. memcached requests occasionally take more than 500 milliseconds but rarely above 1 second, while we're seeing stats collection ticks to be dropped. Even for system stats collection. Which is best explained by troubles receiving those ticks from master node (which is 'wandering' accross cluster). Need to rerun with atop data gathering.
Hide
Permalink
Steve Yen added a comment -
raised to blocker per bug scrub mtg.
Show
Steve Yen added a comment - raised to blocker per bug scrub mtg.
Hide
Permalink
Pavel Paulau added a comment -
atop:
https://s3.amazonaws.com/bugdb/jira/MB-7120/fbe951aa/atops_11102012.tar

diags:
https://s3.amazonaws.com/bugdb/jira/MB-7120/fbe951aa/ec2-174-129-74-54.compute-1.amazonaws.com-11102012-diag.zip

https://s3.amazonaws.com/bugdb/jira/MB-7120/fbe951aa/ec2-184-73-53-179.compute-1.amazonaws.com-11102012-diag.zip

https://s3.amazonaws.com/bugdb/jira/MB-7120/fbe951aa/ec2-50-17-133-35.compute-1.amazonaws.com-11102012-diag.zip

https://s3.amazonaws.com/bugdb/jira/MB-7120/fbe951aa/ec2-54-242-160-13.compute-1.amazonaws.com-11102012-diag.zip

Show
Pavel Paulau added a comment - atop: https://s3.amazonaws.com/bugdb/jira/MB-7120/fbe951aa/atops_11102012.tar diags: https://s3.amazonaws.com/bugdb/jira/MB-7120/fbe951aa/ec2-174-129-74-54.compute-1.amazonaws.com-11102012-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-7120/fbe951aa/ec2-184-73-53-179.compute-1.amazonaws.com-11102012-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-7120/fbe951aa/ec2-50-17-133-35.compute-1.amazonaws.com-11102012-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-7120/fbe951aa/ec2-54-242-160-13.compute-1.amazonaws.com-11102012-diag.zip
Hide
Permalink
Aleksey Kondratenko added a comment -
We found that atop data wasn't complete due to apparently some cron job that killed all running atops at 6:40. Pavel has fixed it and next run will hopefully be more helpful.

As for bug itself and it's relation to other similar bugs. We cannot yet exclude some network contention. atop will help us here, hopefully. Otherwise it indeed looks like duplicate of other issues. We still don't understand however why in this case only mb_master is affected. Notably in first run (and only first run) we've also supposedly had dns issue that caused original title.
Show
Aleksey Kondratenko added a comment - We found that atop data wasn't complete due to apparently some cron job that killed all running atops at 6:40. Pavel has fixed it and next run will hopefully be more helpful. As for bug itself and it's relation to other similar bugs. We cannot yet exclude some network contention. atop will help us here, hopefully. Otherwise it indeed looks like duplicate of other issues. We still don't understand however why in this case only mb_master is affected. Notably in first run (and only first run) we've also supposedly had dns issue that caused original title.
Hide
Permalink
Steve Yen added a comment -
reassigned to get fresh run with full atop results.
Show
Steve Yen added a comment - reassigned to get fresh run with full atop results.
Hide
Permalink
Ronnie Sun added a comment - - edited
Atop replay:

https://s3.amazonaws.com/bugdb/jira/MB-7120/atop.gz

diag:

https://s3.amazonaws.com/bugdb/jira/MB-7120/diag.gz
Show
Ronnie Sun added a comment - - edited Atop replay: https://s3.amazonaws.com/bugdb/jira/MB-7120/atop.gz diag: https://s3.amazonaws.com/bugdb/jira/MB-7120/diag.gz
Hide
Permalink
Steve Yen added a comment -
bug-scrub...

moved to 2.0.1.

farshid reports that hitting Rebalance manually again succeeds.
Show
Steve Yen added a comment - bug-scrub... moved to 2.0.1. farshid reports that hitting Rebalance manually again succeeds.
Hide
Permalink
Aleksey Kondratenko added a comment -
I was expecting reproduction of original "Unable to listen.." issue here while gathering atop. But I see no traces of this problem in latest diags
Show
Aleksey Kondratenko added a comment - I was expecting reproduction of original "Unable to listen.." issue here while gathering atop. But I see no traces of this problem in latest diags
Hide
Permalink
Dipti Borkar added a comment -
Bug scrub: Need to repro
Show
Dipti Borkar added a comment - Bug scrub: Need to repro
Hide
Permalink
Pavel Paulau added a comment -
"IP address seems to have changed. Unable to listen on 'ns_1@ec2-54-242-160-13.compute-1.amazonaws.com'."

"IP address seems to have changed. Unable to listen on 'ns_1@ec2-54-242-160-13.compute-1.amazonaws.com'. (repeated 1 times)"

Diags:
https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-50-17-133-35.compute-1.amazonaws.com-11232012-144-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-184-73-53-179.compute-1.amazonaws.com-11232012-146-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-54-242-160-13.compute-1.amazonaws.com-11232012-148-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-174-129-74-54.compute-1.amazonaws.com-11232012-149-diag.zip

atops:
https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-174-129-74-54.compute-1.amazonaws.com.atop
https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-184-73-53-179.compute-1.amazonaws.com.atop
https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-50-17-133-35.compute-1.amazonaws.com.atop
https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-54-242-160-13.compute-1.amazonaws.com.atop
Show
Pavel Paulau added a comment - "IP address seems to have changed. Unable to listen on ' ns_1@ec2-54-242-160-13.compute-1.amazonaws.com '." "IP address seems to have changed. Unable to listen on ' ns_1@ec2-54-242-160-13.compute-1.amazonaws.com '. (repeated 1 times)" Diags: https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-50-17-133-35.compute-1.amazonaws.com-11232012-144-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-184-73-53-179.compute-1.amazonaws.com-11232012-146-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-54-242-160-13.compute-1.amazonaws.com-11232012-148-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-174-129-74-54.compute-1.amazonaws.com-11232012-149-diag.zip atops: https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-174-129-74-54.compute-1.amazonaws.com.atop https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-184-73-53-179.compute-1.amazonaws.com.atop https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-50-17-133-35.compute-1.amazonaws.com.atop https://s3.amazonaws.com/bugdb/jira/MB-7120/928cc9a4/ec2-54-242-160-13.compute-1.amazonaws.com.atop
Hide
Permalink
Aleksey Kondratenko added a comment -
MB-7182
Show
Aleksey Kondratenko added a comment - MB-7182

People

  • Assignee:
    Aleksey Kondratenko
    Reporter:
    Pavel Paulau
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: