Couchbase Server

memcached and moxi getting killed on issuing a stop rebalance on a 2 node cluster. [ Swap rebalance 1 node in, 1 node out]

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.8.1
  • Fix Version/s: 1.8.1
  • Component/s: ns_server
  • Security Level: Public
  • Labels:
    None
  • Environment:
    Build 918
    2 Buckets, bucket1, bucket2
    1024 vbuckets
    Each node : 24G ram.

Description

Setup
1.Cluster(nodes 94, 95) has 25.1M items per bucket.
2. Mutate items with much large value, causing fragmentation.
3. Node 94 is in heavy swap - 84%.
4. Resident ratio on node 95 has dropped to < 1 percent.
5. Restart node 95.
6. Issue rebalance - add node 97.
7. Stop rebalance

Seeing the following output messages

Port server moxi on node 'ns_1@10.3.2.97' exited with status 137. Restarting. Messages: 2012-06-20 16:12:14: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-06-20 16:12:14: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)

Port server memcached on node 'ns_1@10.3.2.97' exited with status 137. Restarting. Messages: TAP (Producer) eq_tapq:rebalance_483 - Clear the tap queues by force

On re-issuing rebalance, with remove node94. Rebalance fails with

Rebalance exited with reason {wait_for_memcached_failed,"bucket2",
['ns_1@10.3.2.97']}


Attaching the logs from all the nodes https://s3.amazonaws.com/bugdb/jira/bug-cluster-swap/bug.tar

Attached the current screenshot

The live cluster can be accessed at http://10.3.2.94:8091/index.html#sec=log&serversTab=0

Activity

Ascending order - Click to sort in descending order
Farshid Ghods made changes -
Field Original Value New Value
Assignee Aleksey Kondratenko [ alkondratenko ] Karan Kumar [ karan ]
Hide
Permalink
Karan Kumar added a comment -
Node 97 has no buckets at all. Rebalance could not even create bucket on this node.

[ns_server:error] [2012-06-20 18:29:32] [ns_1@10.3.2.95:ns_doctor:ns_doctor:update_status:154] The following buckets became not ready on node 'ns_1@10.3.2.97': ["bucket1",
                                                                  "bucket2"], those of them are active ["bucket1",
                                                                                                        "bucket2"]



[error_logger:error] [2012-06-20 18:29:28] [ns_1@10.3.2.95:error_logger:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
     Supervisor: {local,'ns_vbm_sup-bucket2'}
     Context: shutdown
     Reason: reached_max_restart_intensity
     Offender: [{pid,<0.5948.0>},

Show
Karan Kumar added a comment - Node 97 has no buckets at all. Rebalance could not even create bucket on this node. [ns_server:error] [2012-06-20 18:29:32] [ ns_1@10.3.2.95 :ns_doctor:ns_doctor:update_status:154] The following buckets became not ready on node ' ns_1@10.3.2.97 ': ["bucket1",                                                                   "bucket2"], those of them are active ["bucket1",                                                                                                         "bucket2"] [error_logger:error] [2012-06-20 18:29:28] [ ns_1@10.3.2.95 :error_logger:ale_error_logger_handler:log_report:72] =========================SUPERVISOR REPORT=========================      Supervisor: {local,'ns_vbm_sup-bucket2'}      Context: shutdown      Reason: reached_max_restart_intensity      Offender: [{pid,<0.5948.0>},
Hide
Permalink
Karan Kumar added a comment -
Once again, cluster thinks it has one 1 node (.97) as its member, but the node is not at all part of it.
Show
Karan Kumar added a comment - Once again, cluster thinks it has one 1 node (.97) as its member, but the node is not at all part of it.
Karan Kumar made changes -
Assignee Karan Kumar [ karan ] Aleksey Kondratenko [ alkondratenko ]
Hide
Permalink
Ketaki Gangal added a comment -
From Aleksey -

The moxi/memcached was killed due to a Jenkins attempt to killall -9 on this VM.

Jun 20 18:29:27 ubu-1803 sudo: jenkins : TTY=pts/0 ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/killall -9 memcached
Jun 20 18:29:27 ubu-1803 sudo: cannot execute /usr/sbin/sendmail: No such file or directory
Jun 20 18:29:27 ubu-1803 sudo: cannot execute /usr/sbin/sendmail: No such file or directory
Jun 20 18:29:27 ubu-1803 sudo: jenkins : TTY=pts/0 ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/killall -9 moxi
Jun 20 18:29:27 ubu-1803 sudo: cannot execute /usr/sbin/sendmail: No such file or directory
Jun 20 18:29:27 ubu-1803 sudo: jenkins : TTY=pts/0 ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/killall -9 vbucketmigrator
Jun 20 18:29:27 ubu-1803 sudo: jenkins : TTY=pts/0 ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/killall -9 couchdb
Jun 20 18:29:27 ubu-1803 sudo: cannot execute /usr/sbin/sendmail: No such file or directory
Jun 20 18:29:28 ubu-1803 sudo: cannot execute /usr/sbin/sendmail: No such file or directory
Jun 20 18:29:28 ubu-1803 sudo: jenkins : TTY=pts/0 ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/killall -9 epmd
Show
Ketaki Gangal added a comment - From Aleksey - The moxi/memcached was killed due to a Jenkins attempt to killall -9 on this VM. Jun 20 18:29:27 ubu-1803 sudo: jenkins : TTY=pts/0 ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/killall -9 memcached Jun 20 18:29:27 ubu-1803 sudo: cannot execute /usr/sbin/sendmail: No such file or directory Jun 20 18:29:27 ubu-1803 sudo: cannot execute /usr/sbin/sendmail: No such file or directory Jun 20 18:29:27 ubu-1803 sudo: jenkins : TTY=pts/0 ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/killall -9 moxi Jun 20 18:29:27 ubu-1803 sudo: cannot execute /usr/sbin/sendmail: No such file or directory Jun 20 18:29:27 ubu-1803 sudo: jenkins : TTY=pts/0 ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/killall -9 vbucketmigrator Jun 20 18:29:27 ubu-1803 sudo: jenkins : TTY=pts/0 ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/killall -9 couchdb Jun 20 18:29:27 ubu-1803 sudo: cannot execute /usr/sbin/sendmail: No such file or directory Jun 20 18:29:28 ubu-1803 sudo: cannot execute /usr/sbin/sendmail: No such file or directory Jun 20 18:29:28 ubu-1803 sudo: jenkins : TTY=pts/0 ; PWD=/home/jenkins ; USER=root ; COMMAND=/usr/bin/killall -9 epmd
Hide
Permalink
Ketaki Gangal added a comment -
The issue occured due to another user/job trying to kill memcached process on this node.
Show
Ketaki Gangal added a comment - The issue occured due to another user/job trying to kill memcached process on this node.
Ketaki Gangal made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ep-engine-2-0 #327 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/327/])
    MB-5630 Notify change_vb_filter completion to the client side. (Revision 8d5a967812c88bf27e97ba913d4161d970a7ad43)

     Result = SUCCESS
Chiyoung Seo :
Files :
* tapconnection.hh
* ep_testsuite.cc
* tapconnmap.cc
* command_ids.h
* tapconnection.cc
Show
Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #327 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/327/ ])      MB-5630 Notify change_vb_filter completion to the client side. (Revision 8d5a967812c88bf27e97ba913d4161d970a7ad43)      Result = SUCCESS Chiyoung Seo : Files : * tapconnection.hh * ep_testsuite.cc * tapconnmap.cc * command_ids.h * tapconnection.cc
Peter Wansch made changes -
Sprint Status Current Sprint
Farshid Ghods made changes -
Status Resolved [ 5 ] Closed [ 6 ]

People

  • Assignee:
    Aleksey Kondratenko
    Reporter:
    Ketaki Gangal
Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: