Couchbase

[MB-8333] [DOC 2.0.2] [windows] unable to add a 1.8.0 node to 2.0.2 cluster

Wed, 22 May 2013 16:53:33 -0500

2 nodes cluster (2.0.2), trying to add a 1.8.0 node

2013-05-22 00:40:06.346 ns_cluster:5:info:message(ns_1@172.23.105.74) - Failed to add node 172.23.105.72:8091 to cluster. This node cannot add another node ('ns_1@172.23.105.72') because of cluster version compatibility mismatch. Cluster works in [2, 0] mode and node only supports [1, 8]

[MB-8338] [windows] moxi crashed when create bucket with error: bad json configuration

Wed, 22 May 2013 16:08:28 -0500

Install couchbase server 2.0.2-811 on one node windows 2012 64-bit

Create default bucket, see moxi crash right after bucket created with error bad json configuration

Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: WARNING: curl error: transfer closed with outstanding read data remaining from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
2013-05-22 13:11:46: (agent_config.c.705) ERROR: bad JSON configuration from http://127.0.0.1:8091/pools/default/saslBucketsStreaming: Number of vBuckets must be a power of two > 0 and <= 65536 ({
"name": "default",
"nodeLocator": "vbucket",
"saslPassword": "",
"nodes": [{
"hostname": "127.0.0.1:8091",
"ports": {
"direct": 11210,
"proxy": 11211
}
}],
"vBucketServerMap": {
"hashAlgorithm": "CRC",
"numReplicas": 1,
"serverList": ["127.0.0.1:11210"],
"vBucketMap": []
}
})
EOL on stdin. Exiting ns_log000 ns_1@127.0.0.1 13:11:26 - Wed May 22, 2013
Bucket "default" loaded on node 'ns_1@127.0.0.1' in 0 seconds. ns_memcached001 ns_1@127.0.0.1 13:11:16 - Wed May 22, 2013
Created bucket "default" of type: membase
[{num_replicas,1},
{replica_index,false},
{ram_quota,2571108352},
{auth_type,sasl},
{flush_enabled,false}] menelaus_web012 ns_1@127.0.0.1 13:11:16 - Wed May 22, 2013

after moxi restarts, I could use cbworkloadgen.exe to load and read items from bucket

Link to manifest file for this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-811-rel.setup.exe.manifest.xml

[MB-8313] Table 7.4. Administration — Standard couchbase Tool Options should be updated with new options

Wed, 22 May 2013 15:35:32 -0500

http://www.couchbase.com/docs/couchbase-manual-2.0.pdf

Table 7.4. Administration — Standard couchbase Tool Options doesn't contains the full list of available options.

Actual info we can get from from './couchbase-cli -h":

CLUSTER:
  --cluster=HOST[:PORT] or -c HOST[:PORT]

OPTIONS:
  -u USERNAME, --user=USERNAME admin username of the cluster
  -p PASSWORD, --password=PASSWORD admin password of the cluster
  -o KIND, --output=KIND KIND is json or standard
  -d, --debug

server-add OPTIONS:
  --server-add=HOST[:PORT] server to be added
  --server-add-username=USERNAME admin username for the
                                    server to be added
  --server-add-password=PASSWORD admin password for the
                                    server to be added

server-readd OPTIONS:
  --server-add=HOST[:PORT] server to be added
  --server-add-username=USERNAME admin username for the
                                    server to be added
  --server-add-password=PASSWORD admin password for the
                                    server to be added

rebalance OPTIONS:
  --server-add* see server-add OPTIONS
  --server-remove=HOST[:PORT] the server to be removed

failover OPTIONS:
  --server-failover=HOST[:PORT] server to failover

cluster-* OPTIONS:
  --cluster-username=USER new admin username
  --cluster-password=PASSWORD new admin password
  --cluster-port=PORT new cluster REST/http port
  --cluster-ramsize=RAMSIZEMB per node ram quota in MB

node-init OPTIONS:
  --node-init-data-path=PATH per node path to store data
  --node-init-index-path=PATH per node path to store index

bucket-* OPTIONS:
  --bucket=BUCKETNAME bucket to act on
  --bucket-type=TYPE memcached or couchbase
  --bucket-port=PORT supports ASCII protocol and is auth-less
  --bucket-password=PASSWORD standard port, exclusive with bucket-port
  --bucket-ramsize=RAMSIZEMB ram quota in MB
  --bucket-replica=COUNT replication count
  --enable-flush=[0|1] enable/disable flush
  --enable-index-replica=[0|1] enable/disable index replicas
  --wait wait for bucket create to be complete before returning
  --force force to execute command without asking for confirmation
  --data-only compact datbase data only
  --view-only compact view data only

setting-compacttion OPTIONS:
  --compaction-db-percentage=PERCENTAGE at which point database compaction is triggered
  --compaction-db-size=SIZE[MB] at which point database compaction is triggered
  --compaction-view-percentage=PERCENTAGE at which point view compaction is triggered
  --compaction-view-size=SIZE[MB] at which point view compaction is triggered
  --compaction-period-from=HH:MM allow compaction time period from
  --compaction-period-to=HH:MM allow compaction time period to
  --enable-compaction-abort=[0|1] allow compaction abort when time expires
  --enable-compaction-parallel=[0|1] allow parallel compaction for database and view

setting-notification OPTIONS:
  --enable-notification=[0|1] allow notification

setting-alert OPTIONS:
  --enable-email-alert=[0|1] allow email alert
  --email-recipients=RECIPIENT email recipents, separate addresses with , or ;
  --email-sender=SENDER sender email address
  --email-user=USER email server username
  --email-password=PWD email server password
  --email-host=HOST email server host
  --email-port=PORT email server port
  --enable-email-encrypt=[0|1] email encrypt
  --alert-auto-failover-node node was auto failover
  --alert-auto-failover-max-reached maximum number of auto failover nodes was reached
  --alert-auto-failover-node-down node wasn't auto failover as other nodes are down at the same time
  --alert-auto-failover-cluster-small node wasn't auto fail over as cluster was too small
  --alert-ip-changed node ip address has changed unexpectedly
  --alert-disk-space disk space used for persistent storgage has reached at least 90% capacity
  --alert-meta-overhead metadata overhead is more than 50%
  --alert-meta-oom bucket memory on a node is entirely used for metadata
  --alert-write-failed writing data to disk for a specific bucket has failed

setting-autofailover OPTIONS:
  --enable-auto-failover=[0|1] allow auto failover
  --auto-failover-timeout=TIMEOUT (>=30) specify timeout that expires to trigger auto failover

setting-xdcr OPTIONS:
  --max-concurrent-reps=[32] maximum concurrent replications per bucket, 8 to 256.
  --checkpoint-interval=[1800] intervals between checkpoints, 60 to 14400 seconds.
  --worker-batch-size=[500] doc batch size, 500 to 10000.
  --doc-batch-size=[2048]KB document batching size, 10 to 100000 KB
  --failure-restart-interval=[30] interval for restarting failed xdcr, 1 to 300 seconds
  --optimistic-replication-threshold=[256] document body size threshold (bytes) to trigger optimistic replication

xdcr-setup OPTIONS:
  --create create a new xdcr configuration
  --edit modify existed xdcr configuration
  --delete delete existed xdcr configuration
  --xdcr-cluster-name=CLUSTERNAME cluster name
  --xdcr-hostname=HOSTNAME remote host name to connect to
  --xdcr-username=USERNAME remote cluster admin username
  --xdcr-password=PASSWORD remtoe cluster admin password

xdcr-replicate OPTIONS:
  --create create and start a new replication
  --delete stop and cancel a replication
  --xdcr-from-bucket=BUCKET local bucket name to replicate from
  --xdcr-clucter-name=CLUSTERNAME remote cluster to replicate to
  --xdcr-to-bucket=BUCKETNAME remote bucket to replicate to

[MB-8311] Table 7.3. Administration — couchbase Tool Commands should be updated with new features

Wed, 22 May 2013 15:35:00 -0500

according to http://www.couchbase.com/docs/couchbase-manual-2.0.pdf

Table 7.3. Administration — couchbase Tool Commands
Command Description
server-list List all servers in a cluster
server-info Show details on one server
server-add Add one or more servers to the cluster
server-readd Re-add a server that was failed over to the cluster
rebalance Start a cluster rebalancingCommand-line Interface for Administration
169
Command Description
rebalance-stop Stop current cluster rebalancing
rebalance-status Show status of current cluster rebalancing
failover Failover one or more servers
cluster-init Set the username, password and port of the cluster
node-init Set node specific parameters
bucket-list List all buckets in a cluster
bucket-create Add a new bucket to the cluster
bucket-edit Modify an existing bucket
bucket-delete Delete an existing bucket
bucket-flush Flush a given bucket
help Show longer usage/help and examples

but the full list should be for 2.0.2 as:

./couchbase-cli
usage: couchbase-cli COMMAND CLUSTER [OPTIONS]

CLUSTER is --cluster=HOST[:PORT] or -c HOST[:PORT]

COMMANDs include
  server-list list all servers in a cluster
  server-info show details on one server
  server-add add one or more servers to the cluster
  server-readd readd a server that was failed over
  rebalance start a cluster rebalancing
  rebalance-stop stop current cluster rebalancing
  rebalance-status show status of current cluster rebalancing
  failover failover one or more servers
  cluster-init set the username,password and port of the cluster
  cluster-edit modify cluster settings
  node-init set node specific parameters
  bucket-list list all buckets in a cluster
  bucket-create add a new bucket to the cluster
  bucket-edit modify an existing bucket
  bucket-delete delete an existing bucket
  bucket-flush flush all data from disk for a given bucket
  bucket-compact compact database and index data
  setting-compaction set auto compaction settings
  setting-notification set notification settings
  setting-alert set email alert settings
  setting-autofailover set auto failover settings
  setting-xdcr set xdcr related settings
  xdcr-setup set up XDCR connection
  xdcr-replicate xdcr operations
  help show longer usage/help and examples

[MB-8232] [Doc'd] Multi-reader/writer

Wed, 22 May 2013 14:30:21 -0500

[Doc'd] Multi-reader/writer

[MB-8335] Move RAM stuff to RAM chapter from Disk Storage

Wed, 22 May 2013 14:11:59 -0500

As a separate task, I think we need to take a look at how we are describing what goes on with RAM and disk. The "Disk Storage" section covers a lot of the RAM stuff...

[MB-7852] [Doc'd] Complete productization of Replica Read API

Wed, 22 May 2013 13:43:10 -0500

Replica Read API implementation is ready but testing and related documentation have not completed up to the product-feature ready state.

[MB-8329] ui doesn't show document content

Wed, 22 May 2013 12:53:26 -0500

earlier documents page shown id and content, now content is omitted

[MB-8332] moxi crashed when delete non default bucket

Wed, 22 May 2013 12:48:16 -0500

Test on build 2.0.2-809 on windows 8 64bit
Create default bucket,
Delete default bucket ==> no moxi crashed

Create sasl bucket
Delete sasl bucket ==> moxi crashed as show in log

Deleted bucket "saslbucket"
menelaus_web011 ns_1@127.0.0.1 18:43:37 - Tue May 21, 2013
Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: 2013-05-21 18:40:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2013-05-21 18:40:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)
2013-05-21 18:40:36: (agent_config.c.705) ERROR: bad JSON configuration from http://127.0.0.1:8091/pools/default/saslBucketsStreaming: Number of vBuckets must be a power of two > 0 and <= 65536 ({
"name": "default",
"nodeLocator": "vbucket",
"saslPassword": "",
"nodes": [{
"hostname": "127.0.0.1:8091",
"ports": {
"direct": 11210,
"proxy": 11211
}
}],
"vBucketServerMap": {
"hashAlgorithm": "CRC",
"numReplicas": 1,
"serverList": ["127.0.0.1:11210"],
"vBucketMap": []
}
})
2013-05-21 18:43:11: (agent_config.c.705) ERROR: bad JSON configuration from http://127.0.0.1:8091/pools/default/saslBucketsStreaming: Number of vBuckets must be a power of two > 0 and <= 65536 ({
"name": "saslbucket",
"nodeLocator": "vbucket",
"saslPassword": "",
"nodes": [{
"hostname": "127.0.0.1:8091",
"ports": {
"direct": 11210,
"proxy": 11211
}
}],
"vBucketServerMap": {
"hashAlgorithm": "CRC",
"numReplicas": 1,
"serverList": ["127.0.0.1:11210"],
"vBucketMap": []
}
})
EOL on stdin. Exiting ns_log000 ns_1@127.0.0.1 18:43:37 - Tue May 21, 2013
Shutting down bucket "saslbucket" on 'ns_1@127.0.0.1' for deletion ns_memcached002 ns_1@127.0.0.1 18:43:33 - Tue May 21, 2013
Bucket "saslbucket" loaded on node 'ns_1@127.0.0.1' in 0 seconds. ns_memcached001 ns_1@127.0.0.1 18:43:09 - Tue May 21, 2013
Created bucket "saslbucket" of type: membase
[{num_replicas,1},
{replica_index,false},
{ram_quota,2515533824},
{auth_type,sasl},
{autocompaction,false},
{flush_enabled,false}] menelaus_web012 ns_1@127.0.0.1 18:43:09 - Tue May 21, 2013
Shutting down bucket "default" on 'ns_1@127.0.0.1' for deletion ns_memcached002 ns_1@127.0.0.1 18:41:59 - Tue May 21, 2013
Bucket "default" loaded on node 'ns_1@127.0.0.1' in 0 seconds. ns_memcached001 ns_1@127.0.0.1 18:40:34 - Tue May 21, 2013
Created bucket "default" of type: membase
[{num_replicas,1},
{replica_index,false},
{ram_quota,2515533824},
{auth_type,sasl},
{autocompaction,false},
{flush_enabled,false}] menelaus_web012 ns_1@127.0.0.1 18:40:34 - Tue May 21, 2013
Deleted bucket "new-bucket-97c90481-0efb-4ad3-84ed-8a0bc00799ad"
(repeated 1 times) menelaus_web011 ns_1@127.0.0.1 18:40:27 - Tue May 21, 2013
Client-side error-report for user "Administrator" on node 'ns_1@127.0.0.1':
User-Agent:Python-httplib2/$Rev: 259 $
2013-05-21 18:39:58.017545 : test_non_default_moxi finished
menelaus_web102 ns_1@127.0.0.1 18:40:02 - Tue May 21, 2013
Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: 2013-05-21 18:39:43: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2013-05-21 18:39:43: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)
EOL on stdin. Exiting ns_log000 ns_1@127.0.0.1 18:40:02 - Tue May 21, 2013
Shutting down bucket "new-bucket-97c90481-0efb-4ad3-84ed-8a0bc00799ad" on 'ns_1@127.0.0.1' for deletion

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-809-rel.setup.exe.manifest.xml

[MB-8010] Couchbase Logo update on Windows Installer

Wed, 22 May 2013 11:01:59 -0500

Couchbase Logo update on Windows Installer

[MB-8215] [windows] firewalled node is seen as healthy

Wed, 22 May 2013 07:24:34 -0500

.29 node was firewalled
Domain Profile Settings:
----------------------------------------------------------------------
State ON
Firewall Policy BlockInbound,AllowOutbound
LocalFirewallRules N/A (GPO-store only)
LocalConSecRules N/A (GPO-store only)
InboundUserNotification Disable
RemoteManagement Disable
UnicastResponseToMulticast Enable

Logging:
LogAllowedConnections Disable
LogDroppedConnections Disable
FileName %systemroot%\system32\LogFiles\Firewall\pfirewall.log
MaxFileSize 4096

Private Profile Settings:
----------------------------------------------------------------------
State ON
Firewall Policy BlockInbound,AllowOutbound
LocalFirewallRules N/A (GPO-store only)
LocalConSecRules N/A (GPO-store only)
InboundUserNotification Disable
RemoteManagement Disable
UnicastResponseToMulticast Enable

Logging:
LogAllowedConnections Disable
LogDroppedConnections Disable
FileName %systemroot%\system32\LogFiles\Firewall\pfirewall.log
MaxFileSize 4096

Public Profile Settings:
----------------------------------------------------------------------
State ON
Firewall Policy BlockInbound,AllowOutbound
LocalFirewallRules N/A (GPO-store only)
LocalConSecRules N/A (GPO-store only)
InboundUserNotification Disable
RemoteManagement Disable
UnicastResponseToMulticast Enable

Logging:
LogAllowedConnections Disable
LogDroppedConnections Disable
FileName %systemroot%\system32\LogFiles\Firewall\pfirewall.log
MaxFileSize 4096

Ok.

[MB-8294] Destination node with XDCR (heavy DGM) goes into pending.

Wed, 22 May 2013 07:24:28 -0500

Destination node with XDCR (heavy DGM) goes into pending.

Setup Information:

- Cluster Config : 4 nodes, OS: Centos 6.3, CPU : 6 Core, RAM : 16G, Disk : 500G
- XDCR Topology : Unidirectional ( Master(3) -> Slave(1))
- 4 buckets AbRegNums(2GB), MsgsCalls(2GB), RevAB(3GB), UserInfo(2GB)
- Data Loaded using Viber Workload with 6.7M(RR ~45%), 0.1M(RR 100%), 11M(RR 100%), 12M(RR ~30%) data on the 4 buckets
- Unidirectional XDCR setup to 1 node(master -> slave) for 3 buckets AbRegNums, RevAB, UserInfo

Destination node goes into pending state(screenshot attached). Both beam.smp and memcached are running on the node.

There are lot of crash reports like below but there are no views/design docs in the system.

[error_logger:error,2013-05-16T1:40:30.265,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
  crasher:
    initial call: set_view_update_daemon:init/1
    pid: <0.9511.707>
    registered_name: set_view_update_daemon
    exception exit: {noproc,
                        {gen_server,call,
                            ['capi_set_view_manager-UserInfo',
                             {foreach_doc,
                                 #Fun<capi_ddoc_replication_srv.2.102018441>},
                             infinity]}}
      in function gen_server:terminate/6
    ancestors: [ns_server_sup,ns_server_cluster_sup,<0.58.0>]
    messages: []
    links: [<0.298.0>,<0.9512.707>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 121393
    stack_size: 24
    reductions: 7537
  neighbours:

=========================CRASH REPORT=========================
  crasher:
    initial call: compaction_daemon:-spawn_bucket_compactor/3-fun-2-/0
    pid: <0.9508.707>
    registered_name: []
    exception exit: {noproc,
                        {gen_server,call,
                            ['capi_set_view_manager-RevAB',
                             {foreach_doc,
                                 #Fun<capi_ddoc_replication_srv.1.36030090>},
                             infinity]}}
      in function gen_server:call/3
      in call from capi_ddoc_replication_srv:foreach_live_ddoc_id/2
      in call from capi_ddoc_replication_srv:fetch_ddoc_ids/1
      in call from compaction_daemon:'-spawn_bucket_compactor/3-fun-2-'/4
    ancestors: [compaction_daemon,ns_server_sup,ns_server_cluster_sup,
                  <0.58.0>]
    messages: []
    links: [<0.412.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 28657
    stack_size: 24
    reductions: 30607
  neighbours:

The pending node is in the same state and can be used for investigation:
coconut-h20804.hq.couchbase.com

[MB-7269] cbtransfer/cbrestore throws BadStatusLine exception when using wrong port number

Wed, 22 May 2013 04:59:30 -0500

Thanks to Tony Fonager....

When cbtransfer is invoked incorrectly against the wrong port (11211 rather than 8091), the tool should give a more polite, useful error message rather than an ugly stack trace.

[root@cbnode05 bin]# ./cbtransfer /root/Backup/cbtransfer http://192.168.0.75:11211 -b quizmo -B quizmo -u username -p password
Traceback (most recent call last):
  File "/opt/couchbase/lib/python/cbtransfer", line 33, in <module>
    pump_transfer.exit_handler(pump_transfer.Transfer().main(sys.argv))
  File "/opt/couchbase/lib/python/pump_transfer.py", line 85, in main
    sink_class, sink).run()
  File "/opt/couchbase/lib/python/pump.py", line 100, in run
    rv, source_map, sink_map = self.check_endpoints()
  File "/opt/couchbase/lib/python/pump.py", line 150, in check_endpoints
    rv, sink_map = self.sink_class.check(self.opts, self.sink_spec, source_map)
  File "/opt/couchbase/lib/python/pump_cb.py", line 71, in check
    rv, sink_map = pump.rest_couchbase(opts, spec)
  File "/opt/couchbase/lib/python/pump.py", line 879, in rest_couchbase
    rest_request_json(host, int(port), user, pswd, path)
  File "/opt/couchbase/lib/python/pump.py", line 856, in rest_request_json
    reason=reason)
  File "/opt/couchbase/lib/python/pump.py", line 829, in rest_request
    resp = conn.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine

[MB-8023] Port server moxi on node 'ns_1@10.3.121.97' exited with status 1.

Tue, 21 May 2013 20:54:52 -0500

I reboot one vms install couchbase server 2.0.2-746.
After the vm is up and couchbase server started, I saw error "Port server moxi on node 'ns_1@10.3.121.97' exited with status 1." in log page

Port server moxi on node 'ns_1@10.3.121.97' exited with status 1. Restarting. Messages: 2013-04-02 15:48:43: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2013-04-02 15:48:43: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)
2013-04-02 15:48:45: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.3.121.97 15:48:43 - Tue Apr 2, 2013
Port server moxi on node 'ns_1@10.3.121.97' exited with status 1. Restarting. Messages: 2013-04-02 15:48:38: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2013-04-02 15:48:38: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)
2013-04-02 15:48:40: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.3.121.97 15:48:38 - Tue Apr 2, 2013
Service moxi exited on node 'ns_1@10.3.121.97' in 0.01s
supervisor_cushion001 ns_1@10.3.121.97 15:48:33 - Tue Apr 2, 2013
Port server moxi on node 'ns_1@10.3.121.97' exited with status 1. Restarting. Messages: 2013-04-02 15:48:33: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2013-04-02 15:48:33: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)
2013-04-02 15:48:35: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.3.121.97 15:48:33 - Tue Apr 2, 2013
Port server moxi on node 'ns_1@10.3.121.97' exited with status 1. Restarting. Messages: 2013-04-02 15:48:28: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2013-04-02 15:48:28: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)
2013-04-02 15:48:30: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.3.121.97 15:48:28 - Tue Apr 2, 2013

I will attach collect info file soon

[MB-8325] ns_server should collect various stats about supervised processes

Tue, 21 May 2013 19:22:05 -0500

ns_server should collect various stats about supervised processes

[MB-7621] [Mac README]No way to install CLI tools as per readme.tx instructions

Tue, 21 May 2013 18:27:13 -0500

From the readme.txt in the Mac download:
  * After installing the command-line tools (via the item in the menu),
    moving the app will break the symbolic links that were created in
    /usr/local/bin (or wherever you installed the tools into.)

There is no "item in the menu" with which to install the tools. How does one do so?

We also need docs on the Mac installation in general.

[MB-8240] memslap vbucketkeygen vbuckettool is not working with MAC for latest build

Tue, 21 May 2013 17:42:33 -0500

Chishengs-MacBook-Pro:tools chisheng$ ./memslap -h
dyld: Library not loaded: /opt/couchbase/lib/libmemcached.6.dylib
  Referenced from: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/./memslap
  Reason: image not found
Trace/BPT trap: 5

Chishengs-MacBook-Pro:tools chisheng$ ./vbuckettool -h
dyld: Library not loaded: /opt/couchbase/lib/libvbucket.1.dylib
  Referenced from: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/./vbuckettool
  Reason: image not found
Trace/BPT trap: 5

Chishengs-MacBook-Pro:tools chisheng$ ./vbucketkeygen -h
dyld: Library not loaded: /opt/couchbase/lib/libvbucket.1.dylib
  Referenced from: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/./vbucketkeygen
  Reason: image not found
Trace/BPT trap: 5

[MB-8331] Need to move to 5.8.5.cb1

Dependency — Tue, 21 May 2013 14:15:58 -0500

We need to move to our patched Erlang version 5.8.5.cb1 as we decided not to move to newer version of Erlang that has Filipe's changes upstream.

[MB-8318] memcached crashed in CouchKVStore::setVBucketState

Tue, 21 May 2013 10:53:38 -0500

http://qa.hq.northscale.net/job/cen-ubu-64-2.0-new-rebalance-mixed-cluster-P1/9/consoleFull

1.8.1 & 2.0.2-807 ( mixed cluster)
1_8_1:
10.5.2.11
10.6.2.81
10.6.2.82

2.0.2-807-rel:
10.3.121.119
10.3.121.121
10.3.121.122
10.3.121.123

on 10.3.121.121( root/couchbase):

[root@localhost tmp]# ls -la
total 1345100
drwxrwxrwt 10 root root 4096 May 20 01:09 .
drwxr-xr-x 24 root root 4096 May 18 09:40 ..
drwx------ 2 root root 4096 May 20 00:00 atop.d
drwxr-xr-x 2 root root 4096 May 18 03:24 backup
-rw------- 1 couchbase couchbase 698519552 May 19 17:13 core.memcached.14586
-rw------- 1 couchbase couchbase 947982336 May 19 13:57 core.memcached.3876
-rw-r--r-- 1 root root 129834728 May 18 00:32 couchbase-server-enterprise_x86_64_2.0.2-807-rel.rpm
drwxrwxrwt 2 root root 4096 May 18 09:40 .font-unix
-rw------- 1 root root 66 May 18 00:33 .gdm6DAMUW
srw-rw-rw- 1 root root 0 May 18 09:40 .gdm_socket
drwxrwxrwt 2 root root 4096 May 18 09:40 .ICE-unix
drwx------ 2 root root 4096 Apr 2 12:32 keyring-iUmHcT
drwx------ 2 couchbase couchbase 4096 Apr 11 2012 keyring-JYEPdw
drwx------ 2 root root 4096 May 18 09:40 vmware-root
-r--r--r-- 1 root root 11 May 18 09:40 .X0-lock
drwxrwxrwt 2 root root 4096 May 18 09:40 .X11-unix
[root@localhost tmp]# gdb /opt/couchbase/bin/memcached core.memcached.14586
GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 15231]
[New Thread 20939]
[New Thread 20938]
[New Thread 20937]
[New Thread 15236]
[New Thread 15235]
[New Thread 15234]
[New Thread 15233]
[New Thread 15232]
[New Thread 15230]
[New Thread 14593]
[New Thread 14592]
[New Thread 14591]
[New Thread 14590]
[New Thread 14589]
[New Thread 14588]
[New Thread 14587]
[New Thread 14586]

warning: .dynamic section for "/usr/lib64/libstdc++.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff41d5f000
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x0000003018630265 in raise () from /lib64/libc.so.6
(gdb) t a a bt

Thread 18 (Thread 0x2b6944d5f220 (LWP 14586)):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772000, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772000, flags=<value optimized out>) at event.c:1558
#3 0x00000000004097d6 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7926

Thread 17 (Thread 14587):
#0 0x00000030186c678b in read () from /lib64/libc.so.6
#1 0x000000301866cd57 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x000000301866d71e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x0000003018662804 in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00000030186616a9 in fgets () from /lib64/libc.so.6
#5 0x00002b6944d60939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#6 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#7 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 16 (Thread 14588):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0xef0a040) at extensions/loggers/file_logger.c:368
#2 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#3 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 15 (Thread 14589):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772500, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772500, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0d900) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 14 (Thread 14590):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772280, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0d9f8) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 13 (Thread 14591):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772c80, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772c80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0daf0) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 12 (Thread 14592):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772a00, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772a00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0dbe8) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 11 (Thread 14593):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772780, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0dce0) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 10 (Thread 15230):
#0 0x000000301869a541 in nanosleep () from /lib64/libc.so.6
#1 0x00000030186cded4 in usleep () from /lib64/libc.so.6
#2 0x00002aaaaaf32805 in updateStatsThread (arg=0xef0a4c0) at src/memory_tracker.cc:31
#3 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#4 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 9 (Thread 15232):

#0 0x000000301920d524 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003019208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003019208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002aaaaaf336fa in Mutex::acquire (this=0x137ed110) at src/mutex.cc:79
#4 0x00002aaaaaf79dc3 in lock (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00002aaaaaf70163 in CouchKVStore::setVBucketState (this=0x195f6300, vbucketId=1, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#8 0x00002aaaaaf71069 in CouchKVStore::snapshotVBuckets (this=0x195f6300, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#9 0x00002aaaaaf021f3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2ca5b800, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#10 0x00002aaaaaf51cff in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#11 0x00002aaaaaf371c1 in ExecutorThread::run (this=0x1381da00) at src/scheduler.cc:153
#12 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#13 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#14 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 8 (Thread 15233):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x1381d860) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x1381d860) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 7 (Thread 15234):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x13844ea0) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x13844ea0) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 6 (Thread 15235):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x13844d00) at src/syncobject.hh:57
---Type <return> to continue, or q <return> to quit---
#2 ExecutorThread::run (this=0x13844d00) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 5 (Thread 15236):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x13844b60) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x13844b60) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 4 (Thread 20937):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf0ebdf in wait (this=0x13770400) at src/syncobject.hh:57
#2 wait (this=0x13770400) at src/syncobject.hh:73
#3 wait (this=0x13770400) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x13770400) at src/ep_engine.cc:3377
#5 0x00002aaaaaf0ecc3 in EvpNotifyPendingConns (arg=0x13770400) at src/ep_engine.cc:1153
#6 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#7 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 3 (Thread 20938):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef31c8 in wait (this=0x1428d440, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x1428d440, d=...) at src/dispatcher.cc:342
#3 0x00002aaaaaef5d2a in Dispatcher::run (this=0x137b2a80) at src/dispatcher.cc:184
#4 0x00002aaaaaef64ed in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
#5 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 2 (Thread 20939):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef31c8 in wait (this=0x1428ccf0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x1428ccf0, d=...) at src/dispatcher.cc:342
#3 0x00002aaaaaef5d2a in Dispatcher::run (this=0x137b2fc0) at src/dispatcher.cc:184
#4 0x00002aaaaaef64ed in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
#5 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x4651e940 (LWP 15231)):
#0 0x0000003018630265 in raise () from /lib64/libc.so.6
#1 0x0000003018631d10 in abort () from /lib64/libc.so.6
#2 0x00002aaaaaf7a779 in BinaryPacketHandler::implicitResponse (this=0x38fa) at src/couch-kvstore/couch-notifier.cc:41
#3 0x00002aaaaaf77009 in CouchNotifier::handleResponse (this=0x137ed000, res=0x137fc000) at src/couch-kvstore/couch-notifier.cc:216
#4 0x00002aaaaaf794eb in CouchNotifier::processInput (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:571
#5 0x00002aaaaaf79185 in waitOnce (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:677
#6 CouchNotifier::selectBucket (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:725
#7 0x00002aaaaaf7966f in CouchNotifier::processInput (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:608
#8 0x00002aaaaaf79f75 in waitOnce (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:677
#9 CouchNotifier::notify_update (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:755
#10 0x00002aaaaaf70163 in CouchKVStore::setVBucketState (this=0x195f6000, vbucketId=0, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#11 0x00002aaaaaf71069 in CouchKVStore::snapshotVBuckets (this=0x195f6000, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
---Type <return> to continue, or q <return> to quit---
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#12 0x00002aaaaaf021f3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2ca5b800, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#13 0x00002aaaaaf51cff in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#14 0x00002aaaaaf371c1 in ExecutorThread::run (this=0x1381dba0) at src/scheduler.cc:153
#15 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#16 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#17 0x00000030186d44bd in clone () from /lib64/libc.so.6
(gdb) quit

[MB-8009] Couchbase Logo update on UI

Tue, 21 May 2013 02:52:38 -0500

Couchbase Logo update on UI

[MB-8154] ep_queue_size is not decreasing 20 mins after load was stopped

Tue, 21 May 2013 02:42:50 -0500

1 default bucket, 3 replicas, 4 nodes cluster
loaded 1M items

after load ep_queue_size is still 16 for 20 mins.

attaching logs

[MB-8246] [system test] Rebalance exited with reason timeout waiting for backfill determination

Tue, 21 May 2013 02:41:56 -0500

Cluster ip is 172.23.105.23
1. create 8 nodes cluster, each node has 12G RAM, HHD
2. create 2 buckets default and saslbucket, with memory quota 6G and 4G
3. Run the KV use case for 1 day:
loading 35M items to each bucket, access the data 4k ops/sec with 5% create, 5% delete, 5%expire, 5% update, 80 gets for several hours. Then with the same work load, run some rebalance and failover operations. Works good.
4 Continue the workload for another day, then try to rebalance in one node, rebalance exit with time out.

Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.7961.56>,
{{badmatch,
[{'EXIT',
{timeout,
{gen_server,call,
[<12869.26294.2>,had_backfill,30000]}}}]},
[{ns_single_vbucket_mover,
'-wait_backfill_determination/1-fun-1-',
1}]}}}
ns_orchestrator002 ns_1@172.23.105.23 14:36:20 - Fri May 10, 2013
<0.7953.56> exited with {unexpected_exit,
{'EXIT',<0.7961.56>,
{{badmatch,
[{'EXIT',
{timeout,
{gen_server,call,
[<12869.26294.2>,had_backfill,30000]}}}]},
[{ns_single_vbucket_mover,
'-wait_backfill_determination/1-fun-1-',1}]}}}

The link for diags is https://s3.amazonaws.com/bugdb/jira/MB-8246/ns-diag-20130510162640.txt.zip

[MB-8165] timeout error appeared while trying to read replica from a bucket with active resident ratio 30%

Tue, 21 May 2013 02:39:38 -0500

test to reproduce
newmemcapable.GetrTests.getr_dgm_test,nodes_init=3,GROUP=P1,,dgm_run=true,wait_timeout=250 -p get-delays=true

attaching logs and delays

[MB-8309] online_upgrade_swap_rebalance 1.8.1->2.0.2: Rebalance exited with reason {{change_filter_failed,{'EXIT',{{unexpected_reason,killed}

Mon, 20 May 2013 17:15:48 -0500

http://qa.hq.northscale.net/job/centos-32-2.0-upgrade-P1/4/consoleFull

./testrunner -i /tmp/centos-32-2.0-upgrade.ini upgrade_version=2.0.2-805-rel,get-cbcollect-info=True,GROUP=P1 -t newupgradetests.MultiNodesUpgradeTests.online_upgrade_swap_rebalance,initial_version=1.8.1-942-rel,standard_buckets=1,items=500000,max_verify=1000,GROUP=1_8;ONLINE;WINDOWS;P1

steps:
1)10.3.3.151, 10.3.3.152 with 1.8.1 build
10.3.3.153 with 2.0.2-805-rel

2)[2013-05-17 01:11:42,198] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.151&user=Administrator&knownNodes=ns_1%4010.3.3.152%2Cns_1%4010.3.3.151%2Cns_1%4010.3.3.153
...
[2013-05-17 01:26:15,908] - [task:340] INFO - rebalancing was completed with progress: 100.0% in 873.699863911 sec

3)[2013-05-17 01:26:37,009] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.152&user=Administrator&knownNodes=ns_1%4010.3.3.152%2Cns_1%4010.3.3.153
...

[2013-05-17 01:31:07,549] - [rest_client:1014] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2013-05-17 01:31:07,550] - [rest_client:1015] INFO - Latest logs from UI:
[2013-05-17 01:31:07,601] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 2, u'text': u"Rebalance exited with reason {{change_filter_failed,\n {'EXIT',\n {{unexpected_reason,killed},\n [{misc,executing_on_new_process,1},\n {ns_vbm_sup,local_change_vbucket_filter,4},\n {rpc,local_call,3},\n {ns_vbm_sup,change_vbucket_filter,4},\n {ns_vbm_sup,'-set_replicas/3-fun-2-',5},\n {lists,foldl,3},\n {ns_vbm_sup,set_replicas,3},\n {ns_vbm_sup,\n '-set_replicas_on_nodes/3-fun-1-',3}]}}},\n [{ns_vbm_sup,change_vbucket_filter,4},\n {ns_vbm_sup,'-set_replicas/3-fun-2-',5},\n {lists,foldl,3},\n {ns_vbm_sup,set_replicas,3},\n {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',\n 3},\n {lists,foreach,2},\n {janitor_agent,\n do_bulk_set_vbucket_state_old_style,4},\n {ns_vbucket_mover,\n update_replication_post_move,3}]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368779444497.0, u'type': u'info'}
[2013-05-17 01:31:07,602] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 43 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779418787.0, u'type': u'critical'}
[2013-05-17 01:31:07,603] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779358861.0, u'type': u'critical'}
[2013-05-17 01:31:07,603] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 179 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779358787.0, u'type': u'critical'}
[2013-05-17 01:31:07,604] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779299138.0, u'type': u'critical'}
[2013-05-17 01:31:07,604] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 154 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779298786.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779238798.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 132 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779238787.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779184396.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Bucket "standard_bucket0" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368779184144.0, u'type': u'info'}

[MB-7209] [RN 2.0.2 + Doc]2.0.0 cluster restore with data and index

Mon, 20 May 2013 15:09:46 -0500

In our document showing how to restore from one cluster to another identical cluster (http://www.couchbase.com/docs/couchbase-manual-1.8/couchbase-backup-restore-restore.html at session 5.2.2.2. Restoring to a different cluster),
it shows only restore data.
In 2.0.0, we have @indexes directory to store index. What is the instruction to do cluster restore (data and indexes) from one cluster to another cluster.

[MB-8238] NO menu bar to access menu commands after installation and click the Couchbase icon on MAC

Mon, 20 May 2013 14:13:09 -0500

1. Intall Couchbase server on MAC Lion OS
2. Click Couchbase Icon no access menu poped out

For previous build like 755, we have this access menu. The screen shot is from 755

[MB-8317] Rebalance fails when ejecting out nodes in a 1.8.1 & 2.0.2 mixed cluster while data is being inserted/mutated

Mon, 20 May 2013 13:30:23 -0500

1.8.1 & 2.0.2 mixed cluster: Rebalance exited with reason {{change_filter_failed,{'EXIT',{noproc,[{erlang,link,[<17560.12069.16>]},{ns_vbm_new_sup,'-perform_vbucket_filter_change/6-fun-1-'

very similar bug ~~MB-8309~~

1.8.1 & 2.0.2-807 ( mixed cluster)http://qa.hq.northscale.net/job/cen-ubu-64-2.0-new-rebalance-mixed-cluster-P1/9/consoleFull

./testrunner -i /tmp/rebalance_in.ini wait_timeout=180,GROUP=P1,EXCLUDE_GROUP=FROM_2_0,get-cbcollect-info=True -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops,nodes_out=3,replicas=2,items=50000,doc_ops=create;update;delete,GROUP=OUT;P1

1_8_1:
10.5.2.11
10.6.2.81
10.6.2.82

2.0.2-807-rel:
10.3.121.119
10.3.121.121
10.3.121.122
10.3.121.123

2013-05-19 14:30:01,445] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.121.119%2Cns_1%4010.3.121.122%2Cns_1%4010.6.2.81%2Cns_1%4010.3.121.123%2Cns_1%4010.6.2.82%2Cns_1%4010.5.2.11%2Cns_1%4010.3.121.121
[2013-05-19 14:30:02,062] - [rest_client:929] INFO - rebalance operation started
[2013-05-19 14:30:02,080] - [rest_client:1031] INFO - rebalance percentage : 0 %
[2013-05-19 14:30:12,106] - [task:340] INFO - rebalancing was completed with progress: 100% in 10.0423641205 sec
....
[2013-05-19 14:30:47,707] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.121.123%2Cns_1%4010.3.121.122%2Cns_1%4010.3.121.121&user=Administrator&knownNodes=ns_1%4010.3.121.119%2Cns_1%4010.3.121.122%2Cns_1%4010.6.2.81%2Cns_1%4010.3.121.123%2Cns_1%4010.6.2.82%2Cns_1%4010.5.2.11%2Cns_1%4010.3.121.121
[2013-05-19 14:30:47,720] - [rest_client:929] INFO - rebalance operation started
[2013-05-19 14:30:47,730] - [rest_client:1031] INFO - rebalance percentage : 0 %
[2013-05-19 14:30:57,756] - [rest_client:1014] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2013-05-19 14:30:57,757] - [rest_client:1015] INFO - Latest logs from UI:
[2013-05-19 14:30:57,815] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 0, u'text': u"Current master is older (before 2.0.1) and I'll try to takeover (repeated 1 times)", u'shortText': u'message', u'module': u'mb_master', u'tstamp': 1368999050547.0, u'type': u'warning'}
[2013-05-19 14:30:57,816] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 2, u'text': u"Rebalance exited with reason {{change_filter_failed,\n {'EXIT',\n {noproc,\n [{erlang,link,[<17560.12069.16>]},\n {ns_vbm_new_sup,\n '-perform_vbucket_filter_change/6-fun-1-',\n 7},\n {misc,'-executing_on_new_process/1-fun-0-',\n 3}]}}},\n [{ns_vbm_sup,change_vbucket_filter,4},\n {ns_vbm_sup,'-set_replicas/3-fun-2-',5},\n {lists,foldl,3},\n {ns_vbm_sup,set_replicas,3},\n {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',\n 3},\n {lists,foreach,2},\n {janitor_agent,\n do_bulk_set_vbucket_state_old_style,4},\n {ns_vbucket_mover,handle_call,3}]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368999050224.0, u'type': u'info'}
[2013-05-19 14:30:57,818] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368999049409.0, u'type': u'critical'}
[2013-05-19 14:30:57,819] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368999049155.0, u'type': u'info'}
[2013-05-19 14:30:57,820] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'module': u'ns_rebalancer', u'tstamp': 1368999048756.0, u'type': u'info'}
[2013-05-19 14:30:57,821] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.3.121.119','ns_1@10.6.2.81',\n 'ns_1@10.6.2.82','ns_1@10.5.2.11'], EjectNodes = ['ns_1@10.3.121.123',\n 'ns_1@10.3.121.122',\n 'ns_1@10.3.121.121']\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368999048675.0, u'type': u'info'}
[2013-05-19 14:30:57,822] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.5.2.11', u'code': 102, u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@10.5.2.11\':\nUser-Agent:Python-httplib2/$Rev: 259 $\n2013-05-19 14:30:19.598873 : rebalance_out_with_ops started \n', u'shortText': u'client-side error report', u'module': u'menelaus_web', u'tstamp': 1368999020803.0, u'type': u'warning'}
[2013-05-19 14:30:57,823] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.5.2.11', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.5.2.11\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368999014887.0, u'type': u'info'}
[2013-05-19 14:30:57,825] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.6.2.81', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.6.2.81\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368999013947.0, u'type': u'info'}
[2013-05-19 14:30:57,827] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.6.2.82', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.6.2.82\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368999013933.0, u'type': u'info'}

[MB-8285] DevGuide: metadata reported as 150 bytes needs to be revised

Mon, 20 May 2013 13:24:53 -0500

Found a few places where the metadata size is still reading 150 bytes and should be reduced to 60 for 2.0.1 and 54 for 2.0.2...maybe we can setup some sort of linking or monitoring as this will continue to change in future versions and customers are stumbling across the old values

http://www.couchbase.com/docs/couchbase-devguide-2.0/cb-store-operations.html
http://www.couchbase.com/docs/couchbase-devguide-2.0/cb-store-operations.html
http://www.couchbase.com/docs/couchbase-devguide-2.0/couchbase-keys.html
\http://www.couchbase.com/docs/couchbase-devguide-2.0/cb-store-operations.html

[MB-8296] win32: rebalance failed after Port server memcached on node \'babysitter_of_ns_1@127.0.0.1\' exited with status 255

Mon, 20 May 2013 12:15:45 -0500

http://qa.hq.northscale.net/job/windows32_rebalance-kv/9/consoleFull

./testrunner -i /tmp/4-w-32.ini get-cbcollect-info=True -t rebalancetests.IncrementalRebalanceOut.test_load,replica=1,delete-ratio=0,expiry-ratio=0

password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.2.179%2Cns_1%4010.3.2.185%2Cns_1%4010.3.2.178%2Cns_1%4010.3.2.183
[2013-05-16 04:40:53,799] - [rest_client:929] INFO - rebalance operation started
[2013-05-16 04:40:53,991] - [rest_client:1031] INFO - rebalance percentage : 0 %
[2013-05-16 04:40:55,037] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:40:57,191] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:40:58,428] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:40:59,674] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:00,790] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:02,902] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:04,931] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:06,048] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:08,102] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:10,423] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:11,575] - [rest_client:1014] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2013-05-16 04:41:11,575] - [rest_client:1015] INFO - Latest logs from UI:
[2013-05-16 04:41:11,736] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.178', u'code': 2, u'text': u"Rebalance exited with reason {bulk_set_vbucket_state_failed,\n [{'ns_1@10.3.2.179',\n {'EXIT',\n {{badmatch,wrong_rebalancer_pid},\n [{janitor_agent,set_vbucket_state,7},\n {janitor_agent,\n '-bulk_set_vbucket_state/4-fun-0-',4},\n {misc,'-parallel_map/3-fun-0-',4}]}}}]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368704449014, u'type': u'info'}
[2013-05-16 04:41:11,736] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.178', u'code': 0, u'text': u"<0.8010.5> exited with {bulk_set_vbucket_state_failed,\n [{'ns_1@10.3.2.179',\n {'EXIT',\n {{badmatch,wrong_rebalancer_pid},\n [{janitor_agent,set_vbucket_state,7},\n {janitor_agent,\n '-bulk_set_vbucket_state/4-fun-0-',4},\n {misc,'-parallel_map/3-fun-0-',4}]}}}]}", u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368704446155, u'type': u'critical'}
[2013-05-16 04:41:11,736] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.179', u'code': 0, u'text': u'Port server memcached on node \'babysitter_of_ns_1@127.0.0.1\' exited with status 255. Restarting. Messages: Thu May 16 04:29:32.201380 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Schedule the backfill for vbucket 427\nThu May 16 04:29:32.201380 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nThu May 16 04:29:32.208215 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nThu May 16 04:29:32.213098 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 427\nThu May 16 04:29:32.222863 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Backfill is completed with VBuckets 427, \nThu May 16 04:29:32.222863 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 427\nThu May 16 04:29:32.267782 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.183\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nThu May 16 04:29:32.267782 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.183\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nThu May 16 04:29:32.315630 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:32.553896 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - disconnected, keep alive for 300 seconds\nThu May 16 04:29:32.565614 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Connection is closed by force\nThu May 16 04:29:32.693536 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nThu May 16 04:29:32.693536 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nThu May 16 04:29:32.693536 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - Sending TAP_VBUCKET_SET with vbucket 427 and state "pending"\nThu May 16 04:29:32.694512 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1608"\nThu May 16 04:29:32.697442 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Clear the tap queues by force\nThu May 16 04:29:32.705254 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 774, cookie 044D50E0\nThu May 16 04:29:32.708183 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - VBucket <427> is going dead to complete vbucket takeover\nThu May 16 04:29:32.715995 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - Sending TAP_VBUCKET_SET with vbucket 427 and state "active"\nThu May 16 04:29:32.719901 Pacific Daylight Time 3: (bucket-0) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_427>\nThu May 16 04:29:32.720878 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - disconnected\nThu May 16 04:29:32.723807 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:rebalance_427"\nThu May 16 04:29:32.724784 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - Clear the tap queues by force\nThu May 16 04:29:32.742361 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.183\' - disconnected, keep alive for 300 seconds\nThu May 16 04:29:32.751149 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.183\' - Connection is closed by force\nThu May 16 04:29:32.817551 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1609 - disconnected\nThu May 16 04:29:32.983556 Pacific Daylight Time 3: (bucket-0) Deletion of vbucket 427 was completed.\nThu May 16 04:29:33.097807 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1612 - disconnected\nThu May 16 04:29:33.214010 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1610"\nThu May 16 04:29:33.214010 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1609"\nThu May 16 04:29:33.214010 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.183\' - Clear the tap queues by force\nThu May 16 04:29:33.214010 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:33.214010 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1612"\nThu May 16 04:29:33.335096 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1611 - disconnected\nThu May 16 04:29:33.348767 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1611"\nThu May 16 04:29:33.401498 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 772, cookie 044D4960\nThu May 16 04:29:33.546020 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1614 - disconnected\nThu May 16 04:29:33.734485 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1613 - disconnected\nThu May 16 04:29:33.883889 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1613"\nThu May 16 04:29:33.883889 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1614"\nThu May 16 04:29:33.883889 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:33.900490 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1615 - disconnected\nThu May 16 04:29:34.011811 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1615"\nThu May 16 04:29:34.022552 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 771, cookie 044D4D20\nThu May 16 04:29:34.139732 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:34.308667 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1616 - disconnected\nThu May 16 04:29:34.412176 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1616"\nThu May 16 04:29:34.445377 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 770, cookie 044D5860\nThu May 16 04:29:34.524473 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1618 - disconnected\nThu May 16 04:29:34.547909 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1617 - disconnected\nThu May 16 04:29:34.793011 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1617"\nThu May 16 04:29:34.793011 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1618"\nThu May 16 04:29:34.793011 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:34.805705 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 769, cookie 044D5860\nThu May 16 04:29:34.942415 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1619 - disconnected\nThu May 16 04:29:34.976593 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1619"\nThu May 16 04:29:35.003935 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1620 - disconnected\nThu May 16 04:29:35.162128 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1621 - disconnected\nThu May 16 04:29:35.178728 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1620"\nThu May 16 04:29:35.178728 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1621"\nThu May 16 04:29:35.178728 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:35.595694 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:31:37.747055 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:31:37.747055 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=234 rev=2\nThu May 16 04:34:13.673599 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:13.673599 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=269 rev=4\nThu May 16 04:34:27.262573 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:27.262573 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=702 rev=3\nThu May 16 04:34:37.990402 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:37.990402 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=816 rev=4\nThu May 16 04:34:41.872966 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:41.872966 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=865 rev=4\nThu May 16 04:40:07.491773 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - disconnected, keep alive for 300 seconds\nThu May 16 04:40:07.538645 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1027 - disconnected\nThu May 16 04:40:07.594306 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1027"\nThu May 16 04:40:07.600165 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_4 - disconnected\nThu May 16 04:40:07.602118 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_18 - disconnected\nThu May 16 04:40:07.623601 Pacific Daylight Time 3: (bucket-0) Shutting down tap connections!\nThu May 16 04:40:07.623601 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_4"\nThu May 16 04:40:07.624577 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.183"\nThu May 16 04:40:07.625554 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.178"\nThu May 16 04:40:07.625554 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.183 - Clear the tap queues by force\nThu May 16 04:40:07.625554 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_18"\nThu May 16 04:40:07.626530 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.185"\nThu May 16 04:40:07.626530 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.178 - Clear the tap queues by force\nThu May 16 04:40:07.627507 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Clear the tap queues by force\nThu May 16 04:40:33.087881 Pacific Daylight Time 3: (bucket-0) Warning: failed to load the engine session stats due to IO exception "basic_ios::clear"\nThu May 16 04:40:33.087881 Pacific Daylight Time 3: (bucket-0) Failed to load mutation log, falling back to key dump\nThu May 16 04:40:33.087881 Pacific Daylight Time 3: Extension support isn\'t implemented in this version of bucket_engine\nThu May 16 04:40:33.088858 Pacific Daylight Time 3: (bucket-0) metadata loaded in 1236 usec\nThu May 16 04:40:33.089834 Pacific Daylight Time 3: (bucket-0) warmup completed in 1709 usec\nThu May 16 04:40:35.654123 Pacific Daylight Time 3: (bucket-0) Connection closed by mccouch\nThu May 16 04:40:35.654123 Pacific Daylight Time 3: (bucket-0) Resetting connection to mccouch, lastReceivedCommand = notify_vbucket_update lastSentCommand = notify_vbucket_update currentCommand =notify_vbucket_update\nMUTEX ERROR: Failed to acquire lock: Invalid argument\n\nThis application has requested the Runtime to terminate it in an unusual way.\nPlease contact the application\'s support team for more information.\nThu May 16 04:40:36.013475 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1622 - Reset vbucket 938 was completed succecssfully.\nThu May 16 04:40:36.444112 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1623 - Reset vbucket 682 was completed succecssfully.\nThu May 16 04:40:43.419251 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1624 - Reset vbucket 937 was completed succecssfully.', u'shortText': u'message', u'module': u'ns_log', u'tstamp': 1368704444183, u'type': u'info'}
[2013-05-16 04:41:11,737] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.179', u'code': 4, u'text': u"Control connection to memcached on 'ns_1@10.3.2.179' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368704444167, u'type': u'info'}
[2013-05-16 04:41:11,737] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.183', u'code': 0, u'text': u'Port server memcached on node \'babysitter_of_ns_1@127.0.0.1\' exited with status 255. Restarting. Messages: Thu May 16 04:29:15.442145 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3313 - disconnected\nThu May 16 04:29:15.624751 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 87, cookie 044041E0\nThu May 16 04:29:15.669670 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3317 - disconnected\nThu May 16 04:29:15.862040 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3316 - disconnected\nThu May 16 04:29:16.100306 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3320 - disconnected\nThu May 16 04:29:16.126672 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3313"\nThu May 16 04:29:16.127648 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3316"\nThu May 16 04:29:16.128625 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:16.133507 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3317"\nThu May 16 04:29:16.135460 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3320"\nThu May 16 04:29:16.220416 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 86, cookie 04405860\nThu May 16 04:29:16.252640 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3315 - disconnected\nThu May 16 04:29:17.141255 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3315"\nThu May 16 04:29:17.169574 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 433, cookie 04404B40\nThu May 16 04:29:17.269177 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:17.290660 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3319 - disconnected\nThu May 16 04:29:17.524043 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3319"\nThu May 16 04:29:17.652941 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3321 - disconnected\nThu May 16 04:29:17.821876 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3318 - disconnected\nThu May 16 04:29:18.312079 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3318"\nThu May 16 04:29:18.312079 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3321"\nThu May 16 04:29:18.312079 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:19.443842 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 432, cookie 04404B40\nThu May 16 04:29:19.913539 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 85, cookie 044052C0\nThu May 16 04:29:19.920374 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3322 - disconnected\nThu May 16 04:29:19.941857 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 431, cookie 04405680\nThu May 16 04:29:20.106886 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3323 - disconnected\nThu May 16 04:29:20.277773 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3325 - disconnected\nThu May 16 04:29:20.415460 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3322"\nThu May 16 04:29:20.416436 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3323"\nThu May 16 04:29:20.416436 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3325"\nThu May 16 04:29:20.416436 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:20.758211 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3324 - disconnected\nThu May 16 04:29:21.398795 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3326 - disconnected\nThu May 16 04:29:21.637061 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3324"\nThu May 16 04:29:21.637061 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3326"\nThu May 16 04:29:21.936847 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3327 - disconnected\nThu May 16 04:29:22.640903 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3327"\nThu May 16 04:29:23.123294 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3328 - disconnected\nThu May 16 04:29:23.646698 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3328"\nThu May 16 04:29:24.432781 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 427, cookie 04405860\nThu May 16 04:29:24.720848 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3329 - disconnected\nThu May 16 04:29:25.651453 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3329"\nThu May 16 04:31:32.042777 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:31:32.042777 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=70 rev=3\nThu May 16 04:32:06.188053 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:32:06.188053 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=345 rev=3\nThu May 16 04:32:14.777347 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:32:14.777347 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=391 rev=3\nThu May 16 04:32:32.463715 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:32:32.463715 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=504 rev=3\nThu May 16 04:32:43.120259 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:32:43.120259 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=567 rev=3\nThu May 16 04:32:43.849705 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:32:43.849705 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=574 rev=4\nThu May 16 04:34:08.487842 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:08.487842 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=154 rev=4\nThu May 16 04:34:40.688906 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:40.689883 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=454 rev=4\nThu May 16 04:34:41.123449 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:41.123449 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=456 rev=4\nThu May 16 04:34:51.585670 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:51.586646 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=511 rev=5\nThu May 16 04:34:52.812154 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:52.812154 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=518 rev=4\nThu May 16 04:34:59.385952 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:59.386928 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=550 rev=5\nThu May 16 04:39:57.487966 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - disconnected, keep alive for 300 seconds\nThu May 16 04:39:57.588545 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.179 - disconnected, keep alive for 300 seconds\nThu May 16 04:39:57.780916 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_2904 - disconnected\nThu May 16 04:39:57.783845 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.178 - disconnected, keep alive for 300 seconds\nThu May 16 04:39:57.788728 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1541 - disconnected\nThu May 16 04:39:57.826811 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_131 - disconnected\nThu May 16 04:39:57.913720 Pacific Daylight Time 3: (bucket-0) Shutting down tap connections!\nThu May 16 04:39:57.913720 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.178"\nThu May 16 04:39:57.913720 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_131"\nThu May 16 04:39:57.913720 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.178 - Clear the tap queues by force\nThu May 16 04:39:57.914696 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1541"\nThu May 16 04:39:57.914696 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.179"\nThu May 16 04:39:57.916649 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_2904"\nThu May 16 04:39:57.916649 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.179 - Clear the tap queues by force\nThu May 16 04:39:57.916649 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.185"\nThu May 16 04:39:57.919579 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Clear the tap queues by force\nThu May 16 04:40:23.080161 Pacific Daylight Time 3: (bucket-0) Warning: failed to load the engine session stats due to IO exception "basic_ios::clear"\nThu May 16 04:40:23.080161 Pacific Daylight Time 3: (bucket-0) Failed to load mutation log, falling back to key dump\nThu May 16 04:40:23.080161 Pacific Daylight Time 3: Extension support isn\'t implemented in this version of bucket_engine\nThu May 16 04:40:23.081137 Pacific Daylight Time 3: (bucket-0) metadata loaded in 1162 usec\nThu May 16 04:40:23.082114 Pacific Daylight Time 3: (bucket-0) warmup completed in 1738 usec\nThu May 16 04:40:26.572125 Pacific Daylight Time 3: (bucket-0) Connection closed by mccouch\nThu May 16 04:40:26.572125 Pacific Daylight Time 3: (bucket-0) Resetting connection to mccouch, lastReceivedCommand = notify_vbucket_update lastSentCommand = notify_vbucket_update currentCommand =notify_vbucket_update\nMUTEX ERROR: Failed to acquire lock: Invalid argument\n\nThis application has requested the Runtime to terminate it in an unusual way.\nPlease contact the application\'s support team for more information.\nThu May 16 04:40:26.785978 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3330 - Reset vbucket 596 was completed succecssfully.\nThu May 16 04:40:37.310695 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3331 - Reset vbucket 595 was completed succecssfully.\nThu May 16 04:40:37.419087 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3331 - disconnected\nThu May 16 04:40:37.419087 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3330 - disconnected\nThu May 16 04:40:38.198334 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3330"\nThu May 16 04:40:38.198334 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3331"', u'shortText': u'message', u'module': u'ns_log', u'tstamp': 1368704438866, u'type': u'info'}
[2013-05-16 04:41:11,738] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.183', u'code': 4, u'text': u"Control connection to memcached on 'ns_1@10.3.2.183' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368704438788, u'type': u'info'}
[2013-05-16 04:41:11,738] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.185', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.185\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368704436520, u'type': u'info'}
[2013-05-16 04:41:11,739] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.185', u'code': 3, u'text': u'Node ns_1@10.3.2.185 joined cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1368704435895, u'type': u'info'}
[2013-05-16 04:41:11,741] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.185', u'code': 4, u'text': u"Node 'ns_1@10.3.2.185' saw that node 'ns_1@10.3.2.183' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1368704435895, u'type': u'info'}
[2013-05-16 04:41:11,742] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.185', u'code': 4, u'text': u"Node 'ns_1@10.3.2.185' saw that node 'ns_1@10.3.2.179' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1368704435707, u'type': u'info'}
ERROR

[MB-8003] investigate whether hot-fix commits were included in later releases

Mon, 20 May 2013 10:53:02 -0500

commits should be merged forward once they are found to be good.

This is especaily true for changes that are included in hot-fixes.

Track the changes that were included in hot-fixes and verify which releases they are int.

[MB-7938] 2.0.2 memcached crashes in EventuallyPersistentStore::flushVBucket

Mon, 20 May 2013 08:39:19 -0500

http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions/196/consoleFull

./testrunner -i /tmp/rebalance_regression.ini wait_timeout=100,get-cbcollect-info=True -t swaprebalance.SwapRebalanceFailedTests.test_failover_swap_rebalance,replica=2,num-buckets=2,num-swap=2,keys-count=1000000,swap-orchestrator=True

test logs:
[2013-03-18 08:35:42,154] - [data_helper:289] INFO - creating direct client 10.3.121.94:11210 bucket-1
[2013-03-18 08:35:42,432] - [rest_client:913] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2013-03-18 08:35:42,432] - [rest_client:914] INFO - Latest logs from UI:
[2013-03-18 08:35:42,565] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.93', u'code': 1, u'text': u"Node 'ns_1@10.3.121.93' is leaving cluster.", u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1363621308275, u'type': u'info'}
[2013-03-18 08:35:42,567] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.93', u'code': 4, u'text': u'Node ns_1@10.3.121.93 left cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1363621307949, u'type': u'info'}
[2013-03-18 08:35:42,567] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.95', u'code': 4, u'text': u"Control connection to memcached on 'ns_1@10.3.121.95' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1363621307350, u'type': u'info'}
[2013-03-18 08:35:42,570] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.93', u'code': 2, u'text': u"Rebalance exited with reason {{bulk_set_vbucket_state_failed,\n [{'ns_1@10.3.121.95',\n {'EXIT',\n {{{{unexpected_reason,\n {{badmatch,{error,closed}},\n [{mc_binary,quick_stats_recv,3},\n {mc_binary,\n mass_get_last_closed_checkpoint_loop,\n 5},\n {mc_binary,\n mass_get_last_closed_checkpoint,3},\n {ebucketmigrator_srv,handle_call,3},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]}},\n [{misc,executing_on_new_process,1},\n {tap_replication_manager,\n change_vbucket_filter,4},\n {tap_replication_manager,\n '-do_set_incoming_replication_map/3-lc$^5/1-5-',\n 2},\n {tap_replication_manager,\n do_set_incoming_replication_map,3},\n {tap_replication_manager,handle_call,3},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n ['tap_replication_manager-bucket-1',\n {change_vbucket_replication,531,\n 'ns_1@10.3.121.96'},\n infinity]}},\n {gen_server,call,\n [{'janitor_agent-bucket-1',\n 'ns_1@10.3.121.95'},\n {if_rebalance,<0.13276.97>,\n {update_vbucket_state,531,replica,\n undefined,'ns_1@10.3.121.96'}},\n infinity]}}}},\n {'ns_1@10.3.121.94',\n {'EXIT',\n {{{{unexpected_reason,\n {{badmatch,{error,closed}},\n [{mc_binary,quick_stats_recv,3},\n {mc_binary,quick_stats_loop,5},\n {mc_binary,quick_stats,5},\n {mc_client_binary,\n get_zero_open_checkpoint_vbuckets,3},\n {ebucketmigrator_srv,handle_call,3},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]}},\n [{misc,executing_on_new_process,1},\n {tap_replication_manager,\n change_vbucket_filter,4},\n {tap_replication_manager,\n '-do_set_incoming_replication_map/3-lc$^5/1-5-',\n 2},\n {tap_replication_manager,\n do_set_incoming_replication_map,3},\n {tap_replication_manager,handle_call,3},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n ['tap_replication_manager-bucket-1',\n {change_vbucket_replication,531,\n 'ns_1@10.3.121.95'},\n infinity]}},\n {gen_server,call,\n [{'janitor_agent-bucket-1',\n 'ns_1@10.3.121.94'},\n {if_rebalance,<0.13276.97>,\n {update_vbucket_state,531,replica,\n undefined,'ns_1@10.3.121.95'}},\n infinity]}}}}]},\n [{janitor_agent,bulk_set_vbucket_state,4},\n {ns_vbucket_mover,\n update_replication_post_move,3},\n {ns_vbucket_mover,on_move_done,2},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1363621307292, u'type': u'info'}
[2013-03-18 08:35:42,571] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.95', u'code': 0, u'text': u'Port server memcached on node \'ns_1@10.3.121.95\' exited with status 134. Restarting. Messages: Mon Mar 18 08:41:35.578716 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_451 - Reset vbucket 538 was completed succecssfully.\nMon Mar 18 08:41:35.704327 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_452 - disconnected\nMon Mar 18 08:41:35.805105 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_450"\nMon Mar 18 08:41:35.805157 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_452"\nMon Mar 18 08:41:35.805446 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:35.830322 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 538, cookie 0x6ccf080\nMon Mar 18 08:41:36.347550 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_451 - disconnected\nMon Mar 18 08:41:36.444078 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_453 - Reset vbucket 870 was completed succecssfully.\nMon Mar 18 08:41:36.479457 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_451"\nMon Mar 18 08:41:36.480162 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:36.714685 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 870, cookie 0x6ca6000\nMon Mar 18 08:41:36.976727 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_453 - disconnected\nMon Mar 18 08:41:37.019371 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_454 - Reset vbucket 537 was completed succecssfully.\nMon Mar 18 08:41:37.206467 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_455 - disconnected\nMon Mar 18 08:41:37.271568 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 537, cookie 0x6ca6000\nMon Mar 18 08:41:37.285754 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_453"\nMon Mar 18 08:41:37.285827 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_455"\nMon Mar 18 08:41:37.286268 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:37.643780 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_456 - Reset vbucket 869 was completed succecssfully.\nMon Mar 18 08:41:37.799981 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 869, cookie 0x6cf1b80\nMon Mar 18 08:41:37.837434 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_456 - disconnected\nMon Mar 18 08:41:37.907834 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_454 - disconnected\nMon Mar 18 08:41:37.984234 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_457 - Reset vbucket 868 was completed succecssfully.\nMon Mar 18 08:41:38.035570 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_454"\nMon Mar 18 08:41:38.035669 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_456"\nMon Mar 18 08:41:38.036087 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:38.180437 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_458 - disconnected\nMon Mar 18 08:41:38.250083 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_458"\nMon Mar 18 08:41:38.383886 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:38.544533 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 868, cookie 0x6ca62c0\nMon Mar 18 08:41:38.692906 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_459 - Reset vbucket 536 was completed succecssfully.\nMon Mar 18 08:41:38.933451 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_457 - disconnected\nMon Mar 18 08:41:38.934109 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 536, cookie 0x6ca7080\nMon Mar 18 08:41:39.241149 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_460 - Reset vbucket 867 was completed succecssfully.\nMon Mar 18 08:41:39.396965 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_461 - disconnected\nMon Mar 18 08:41:39.472257 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_457"\nMon Mar 18 08:41:39.472345 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_461"\nMon Mar 18 08:41:39.506632 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:39.553585 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_459 - disconnected\nMon Mar 18 08:41:39.572591 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 867, cookie 0x6ca6000\nMon Mar 18 08:41:39.579114 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_459"\nMon Mar 18 08:41:39.695310 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_460 - disconnected\nMon Mar 18 08:41:39.900652 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_460"\nMon Mar 18 08:41:39.900985 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:40.151512 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_462 - disconnected\nMon Mar 18 08:41:40.266294 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_462"\nMon Mar 18 08:41:40.266461 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:40.856053 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_463 - Reset vbucket 866 was completed succecssfully.\nMon Mar 18 08:41:41.091008 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 866, cookie 0x6ca6000\nMon Mar 18 08:41:41.323419 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_463 - disconnected\nMon Mar 18 08:41:41.342314 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_463"\nMon Mar 18 08:41:41.371404 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_464 - Reset vbucket 535 was completed succecssfully.\nMon Mar 18 08:41:41.661043 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_465 - disconnected\nMon Mar 18 08:41:41.717840 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 535, cookie 0x6ca62c0\nMon Mar 18 08:41:41.742040 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_465"\nMon Mar 18 08:41:41.742448 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:41.954347 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_466 - Reset vbucket 865 was completed succecssfully.\nMon Mar 18 08:41:42.166879 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 865, cookie 0x6cf1600\nMon Mar 18 08:41:42.269048 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_464 - disconnected\nMon Mar 18 08:41:42.340677 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_464"\nMon Mar 18 08:41:42.340711 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:42.384967 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_467 - Reset vbucket 864 was completed succecssfully.\nMon Mar 18 08:41:42.413198 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_466 - disconnected\nMon Mar 18 08:41:42.441804 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_466"\nMon Mar 18 08:41:42.684453 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 864, cookie 0x6ccf080\nMon Mar 18 08:41:42.818667 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_468 - disconnected\nMon Mar 18 08:41:42.875395 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_467 - disconnected\nMon Mar 18 08:41:42.904356 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_469 - Reset vbucket 534 was completed succecssfully.\nMon Mar 18 08:41:42.978903 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_467"\nMon Mar 18 08:41:42.978974 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_468"\nMon Mar 18 08:41:42.979235 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:43.243646 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_470 - disconnected\nMon Mar 18 08:41:43.403811 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:43.403750 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_470"\nMon Mar 18 08:41:43.494580 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 534, cookie 0x6ccf600\nMon Mar 18 08:41:43.580734 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_471 - Reset vbucket 533 was completed succecssfully.\nMon Mar 18 08:41:43.893154 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 533, cookie 0x6ca6000\nMon Mar 18 08:41:44.327385 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_472 - Reset vbucket 532 was completed succecssfully.\nMon Mar 18 08:41:44.412144 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_469 - disconnected\nMon Mar 18 08:41:44.495170 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_469"\nMon Mar 18 08:41:44.502353 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_471 - disconnected\nMon Mar 18 08:41:44.516347 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_471"\nMon Mar 18 08:41:44.516369 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:44.519367 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 532, cookie 0x6ca62c0\nMon Mar 18 08:41:44.855524 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:45.087666 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_473 - Reset vbucket 863 was completed succecssfully.\nMon Mar 18 08:41:45.320777 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_472 - disconnected\nMon Mar 18 08:41:45.440897 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 863, cookie 0x6cceb00\nMon Mar 18 08:41:45.445425 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_472"\nMon Mar 18 08:41:45.446922 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.94 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:45.531819 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_474 - Reset vbucket 531 was completed succecssfully.\nMon Mar 18 08:41:45.566986 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_473 - disconnected\nMon Mar 18 08:41:45.641574 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_473"\nMon Mar 18 08:41:45.822884 PDT 3: (bucket-1) Notified the completion of checkpoint persistence for vbucket 531, cookie 0x6ccf600\nMon Mar 18 08:41:45.850503 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_475 - disconnected\nMon Mar 18 08:41:45.932729 PDT 3: (bucket-1) Schedule cleanup of "eq_tapq:anon_475"\nMon Mar 18 08:41:45.932798 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_ns_1@10.3.121.96 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nMon Mar 18 08:41:46.151290 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_476 - Reset vbucket 530 was completed succecssfully.\nMon Mar 18 08:41:46.393310 PDT 3: (bucket-1) TAP (Consumer) eq_tapq:anon_474 - disconnected\nmemcached: src/ep.cc:1790: virtual void PersistenceCallback::callback(mutation_result&): Assertion `stats->diskQueueSize < ((size_t)1<<(sizeof(size_t)*8-1))\' failed.', u'shortText': u'message', u'module': u'ns_port_server', u'tstamp': 1363621307102, u'type': u'info'}
[2013-03-18 08:35:42,577] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.93', u'code': 0, u'text': u'Bucket "bucket-1" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1363620900761, u'type': u'info'}
[2013-03-18 08:35:42,578] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.95', u'code': 1, u'text': u'Bucket "bucket-1" loaded on node \'ns_1@10.3.121.95\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1363620899668, u'type': u'info'}
[2013-03-18 08:35:42,579] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.96', u'code': 1, u'text': u'Bucket "bucket-1" loaded on node \'ns_1@10.3.121.96\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1363620899353, u'type': u'info'}
[2013-03-18 08:35:42,580] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.94', u'code': 5, u'text': u"Node 'ns_1@10.3.121.94' saw that node 'ns_1@10.3.121.98' went down.", u'shortText': u'node down', u'module': u'ns_node_disco', u'tstamp': 1363620899198, u'type': u'warning'}
[2013-03-18 08:35:42,585] - [rest_client:915] ERROR - {u'node': u'ns_1@10.3.121.95', u'code': 5, u'text': u"Node 'ns_1@10.3.121.95' saw that node 'ns_1@10.3.121.98' went down.", u'shortText': u'node down', u'module': u'ns_node_disco', u'tstamp': 1363620899195, u'type': u'warning'}
ERROR

root@ubuntu1104-64:/tmp# sudo gdb /opt/couchbase/bin/memcached core.memcached.30589
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 30607]
[New Thread 30597]
[New Thread 30611]
[New Thread 30589]
[New Thread 30604]
[New Thread 30598]
[New Thread 30602]
[New Thread 30600]
[New Thread 30603]
[New Thread 30601]
[New Thread 30606]
[New Thread 30610]
[New Thread 30609]
[New Thread 30608]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x00007f9d01257d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) t a a bt

Thread 14 (Thread 30608):
#0 0x00007f9d015c7f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9cfca013d8 in wait (this=0x24ee2d0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x24ee2d0, d=...) at src/dispatcher.cc:328
#3 0x00007f9cfca042ca in Dispatcher::run (this=0x6d9b880) at src/dispatcher.cc:171
#4 0x00007f9cfca04b7d in launch_dispatcher_thread (arg=0x6d9b8d4) at src/dispatcher.cc:28
#5 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 13 (Thread 30609):
#0 0x00007f9d015c7f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9cfca013d8 in wait (this=0x24ee240, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x24ee240, d=...) at src/dispatcher.cc:328
#3 0x00007f9cfca042ca in Dispatcher::run (this=0x6d9b6c0) at src/dispatcher.cc:171
#4 0x00007f9cfca04b7d in launch_dispatcher_thread (arg=0x6d9b714) at src/dispatcher.cc:28
#5 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 12 (Thread 30610):
#0 0x00007f9d015c7f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9cfca013d8 in wait (this=0x24ee5a0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x24ee5a0, d=...) at src/dispatcher.cc:328
#3 0x00007f9cfca042ca in Dispatcher::run (this=0x6d9b500) at src/dispatcher.cc:171
#4 0x00007f9cfca04b7d in launch_dispatcher_thread (arg=0x6d9b554) at src/dispatcher.cc:28
#5 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 11 (Thread 30606):
#0 0x00007f9d012d44ed in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d01305914 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f9cfca416a5 in updateStatsThread (arg=<value optimized out>) at src/memory_tracker.cc:31
#3 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 10 (Thread 30601):
#0 0x00007f9d0130d633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d020dbf36 in epoll_dispatch (base=0x6d4e280, tv=<value optimized out>) at epoll.c:404
#2 0x00007f9d020c7394 in event_base_loop (base=0x6d4e280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x24e94f8) at daemon/thread.c:301
#4 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 9 (Thread 30603):
#0 mc_get_allocation_size (ptr=0x7977200) at daemon/alloc_hooks.c:115
#1 0x00007f9cfca41264 in DeleteHook (ptr=0x7977200) at src/memory_tracker.cc:56
#2 0x00007f9d01801402 in MallocHook::InvokeDeleteHookSlow(void const*) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#3 0x00007f9d017f455a in MallocHook::InvokeDeleteHook(void const*) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#4 0x00007f9d018077f6 in tc_delete () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#5 0x00007f9d00fb3e19 in std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_stringstream() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
---Type <return> to continue, or q <return> to quit---
#6 0x00007f9cfca3809a in void core_engine::add_casted_stat<unsigned long>(char const*, unsigned long, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) ()
   from /opt/couchbase/lib/memcached/ep.so

#7 0x00007f9cfca28af0 in addCheckpointStat (this=<value optimized out>, vb=...) at src/ep_engine.cc:2858
#8 EventuallyPersistentEngine::StatCheckpointVisitor::visitBucket (this=<value optimized out>, vb=...) at src/ep_engine.cc:2842

#9 0x00007f9cfca09adc in EventuallyPersistentStore::visit (this=<value optimized out>, visitor=...) at src/ep.cc:2417
#10 0x00007f9cfca2d315 in EventuallyPersistentEngine::doCheckpointStats (this=0x6da2000, cookie=0x6cf18c0, add_stat=0x40d1d0 <append_stats>, stat_key=<value optimized out>, nkey=<value optimized out>)
    at src/ep_engine.cc:2868
#11 0x00007f9cfca2fdcc in EventuallyPersistentEngine::getStats (this=0x6da2000, cookie=0x6cf18c0, stat_key=0x6cf5018 "checkpointn_ns_1@10.3.121.94", nkey=10, add_stat=0x40d1d0 <append_stats>)
    at src/ep_engine.cc:3328
#12 0x00007f9cfca30416 in EvpGetStats (handle=0x6da2000, cookie=0x6cf18c0, stat_key=0x6cf5018 "checkpointn_ns_1@10.3.121.94", nkey=10, add_stat=<value optimized out>) at src/ep_engine.cc:193
#13 0x00007f9cff4e1c20 in bucket_get_stats (handle=<value optimized out>, cookie=0x6cf18c0, stat_key=0x6cf5018 "checkpointn_ns_1@10.3.121.94", nkey=10, add_stat=0x40d1d0 <append_stats>)
    at bucket_engine.c:1720
#14 0x00000000004113f6 in process_bin_stat (c=0x6cf18c0) at daemon/memcached.c:2199
#15 0x0000000000411d65 in complete_nread_binary (c=0x7977200) at daemon/memcached.c:3708
#16 complete_nread (c=0x7977200) at daemon/memcached.c:3820
#17 conn_nread (c=0x7977200) at daemon/memcached.c:5673
#18 0x00000000004068a5 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x6cf18c0) at daemon/memcached.c:5936
#19 0x00007f9d020c748c in event_process_active_single_queue (base=0x6d4ea00, flags=<value optimized out>) at event.c:1308
#20 event_process_active (base=0x6d4ea00, flags=<value optimized out>) at event.c:1375
#21 event_base_loop (base=0x6d4ea00, flags=<value optimized out>) at event.c:1572
#22 0x0000000000414b94 in worker_libevent (arg=0x24e96e8) at daemon/thread.c:301
#23 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#24 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#25 0x0000000000000000 in ?? ()

Thread 8 (Thread 30600):
#0 0x00007f9d015ca955 in __lll_unlock_wake () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9d015c6e7a in _L_unlock_1177 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f9d015c6da3 in pthread_mutex_unlock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007f9cfca42636 in Mutex::release (this=0xed948a8) at src/mutex.cc:94
#4 0x00007f9cfc9fab10 in unlock (this=<value optimized out>, qi=..., vbucket=...) at src/locks.hh:58
#5 ~LockHolder (this=<value optimized out>, qi=..., vbucket=...) at src/locks.hh:41
#6 CheckpointManager::queueDirty (this=<value optimized out>, qi=..., vbucket=...) at src/checkpoint.cc:717
#7 0x00007f9cfca07423 in EventuallyPersistentStore::queueDirty (this=0x6d9e480, vb=..., key=..., vbid=<value optimized out>, op=queue_op_set, seqno=<value optimized out>, tapBackfill=false)
    at src/ep.cc:2168
#8 0x00007f9cfca0bc2d in EventuallyPersistentStore::setWithMeta (this=0x6d9e480, itm=..., cas=<value optimized out>, cookie=<value optimized out>, force=<value optimized out>,
    allowExisting=<value optimized out>, nru=3 '\003') at src/ep.cc:1399
#9 0x00007f9cfca2ac53 in EventuallyPersistentEngine::tapNotify(const void *, void *, uint16_t, uint8_t, uint16_t, <anonymous enum>, uint32_t, const void *, size_t, uint32_t, uint32_t, uint64_t, const void *, size_t, uint16_t) (this=0x6da2000, cookie=0x6ccf080, engine_specific=<value optimized out>, nengine=<value optimized out>, tap_flags=<value optimized out>, tap_event=TAP_MUTATION, tap_seqno=974,
    key=0x1102c031, nkey=25, flags=0, exptime=0, cas=15671462831700303, data=0x1102c04a, ndata=369, vbucket=530) at src/ep_engine.cc:2067
#10 0x00007f9cfca2b368 in EvpTapNotify(ENGINE_HANDLE *, const void *, void *, uint16_t, uint8_t, uint16_t, <anonymous enum>, uint32_t, const void *, size_t, uint32_t, uint32_t, uint64_t, const void *, size_t, uint16_t) (handle=0x6da2000, cookie=0x6ccf080, engine_specific=0x1102c028, nengine=65535, ttl=254 '\376', tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=974, key=0x1102c031, nkey=25, flags=0,
    exptime=0, cas=15671462831700303, data=0x1102c04a, ndata=369, vbucket=<value optimized out>) at src/ep_engine.cc:1037
#11 0x00007f9cff4e0a04 in bucket_tap_notify (handle=<value optimized out>, cookie=0x6ccf080, engine_specific=0x1102c028, nengine=65535, ttl=254 '\376', tap_flags=23, tap_event=TAP_MUTATION,
    tap_seqno=974, key=0x1102c031, nkey=25, flags=0, exptime=0, cas=15671462831700303, data=0x1102c04a, ndata=369, vbucket=<value optimized out>) at bucket_engine.c:1942
#12 0x0000000000409eb2 in process_bin_tap_packet (event=TAP_MUTATION, c=0x6ccf080) at daemon/memcached.c:3031
#13 0x00000000004120c3 in process_bin_packet (c=0x6ccf080) at daemon/memcached.c:3117
#14 complete_nread_binary (c=0x6ccf080) at daemon/memcached.c:3738
#15 complete_nread (c=0x6ccf080) at daemon/memcached.c:3820
#16 conn_nread (c=0x6ccf080) at daemon/memcached.c:5673
#17 0x00000000004068a5 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x6ccf080) at daemon/memcached.c:5936
#18 0x00007f9d020c748c in event_process_active_single_queue (base=0x6d4e500, flags=<value optimized out>) at event.c:1308
#19 event_process_active (base=0x6d4e500, flags=<value optimized out>) at event.c:1375
#20 event_base_loop (base=0x6d4e500, flags=<value optimized out>) at event.c:1572
#21 0x0000000000414b94 in worker_libevent (arg=0x24e9400) at daemon/thread.c:301
---Type <return> to continue, or q <return> to quit---
#22 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#23 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#24 0x0000000000000000 in ?? ()

Thread 7 (Thread 30602):
#0 0x00007f9d0130d633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d020dbf36 in epoll_dispatch (base=0x6d4ec80, tv=<value optimized out>) at epoll.c:404
#2 0x00007f9d020c7394 in event_base_loop (base=0x6d4ec80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x24e95f0) at daemon/thread.c:301
#4 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 6 (Thread 30598):
#0 0x00007f9d015c7f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9d00102176 in logger_thead_main (arg=<value optimized out>) at extensions/loggers/file_logger.c:368
#2 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

Thread 5 (Thread 30604):
#0 0x00007f9d0130d633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d020dbf36 in epoll_dispatch (base=0x6d4e780, tv=<value optimized out>) at epoll.c:404
#2 0x00007f9d020c7394 in event_base_loop (base=0x6d4e780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x24e97e0) at daemon/thread.c:301
#4 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 4 (Thread 30589):
#0 0x00007f9d0130d633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d020dbf36 in epoll_dispatch (base=0x6d4e000, tv=<value optimized out>) at epoll.c:404
#2 0x00007f9d020c7394 in event_base_loop (base=0x6d4e000, flags=<value optimized out>) at event.c:1558
#3 0x000000000040c2e1 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7918

Thread 3 (Thread 30611):
#0 0x00007f9d015c7f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9cfca2090f in wait (this=0x6da2000) at src/syncobject.hh:57
#2 wait (this=0x6da2000) at src/syncobject.hh:73
#3 wait (this=0x6da2000) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x6da2000) at src/ep_engine.cc:3406
#5 0x00007f9cfca209f3 in EvpNotifyPendingConns (arg=0x6da2000) at src/ep_engine.cc:1139
#6 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 2 (Thread 30597):
#0 0x00007f9d012fe2ed in read () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d01299798 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f9d0129a7be in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f9d0128e8fa in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f9d0128d7ca in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f9d00b05b19 in fgets (arg=<value optimized out>) at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#7 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#8 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 1 (Thread 30607):
#0 0x00007f9d01257d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9d0125bab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f9d012507c5 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f9cfca1d8db in PersistenceCallback::callback(std::pair<int, long>&) () from /opt/couchbase/lib/memcached/ep.so
#4 0x00007f9cfca78724 in CouchKVStore::commitCallback(CouchRequest **, int, <anonymous enum>) (this=0x6dbc000, committedReqs=<value optimized out>, numReqs=10, errCode=COUCHSTORE_SUCCESS)
    at src/couch-kvstore/couch-kvstore.cc:1655
#5 0x00007f9cfca7c07c in CouchKVStore::commit2couchstore (this=0x6dbc000) at src/couch-kvstore/couch-kvstore.cc:1488
#6 0x00007f9cfca7c25a in CouchKVStore::commit (this=0x777d) at src/couch-kvstore/couch-kvstore.cc:871
#7 0x00007f9cfca0ee06 in EventuallyPersistentStore::flushVBucket (this=0x6d9e480, vbid=<value optimized out>) at src/ep.cc:1977
#8 0x00007f9cfca3ca1a in doFlush (this=0x6dba5a0, d=..., tid=...) at src/flusher.cc:215
#9 Flusher::step (this=0x6dba5a0, d=..., tid=...) at src/flusher.cc:153
#10 0x00007f9cfca042ca in Dispatcher::run (this=0x6d9aa80) at src/dispatcher.cc:171
#11 0x00007f9cfca04b7d in launch_dispatcher_thread (arg=0x777d) at src/dispatcher.cc:28
#12 0x00007f9d015c2d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007f9d0130cfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#14 0x0000000000000000 in ?? ()

[MB-8284] Rebalance fails with reason replicator_died

Mon, 20 May 2013 08:22:08 -0500

Rebalance failures are seen in Centos64 sanity job:
http://qa.hq.northscale.net/job/CouchbaseServer-SanityTest-4Node-Centos64/73/consoleFull

** Reason for termination ==
** {exited,
       {'EXIT',<0.28006.6>,
           {replicator_died,
               {'EXIT',<17674.28481.2>,{badmatch,{error,closed}}}}}}

[ns_server:info,2013-05-15T3:24:33.837,ns_1@10.3.3.32:janitor_agent-default<0.23324.6>:janitor_agent:handle_info:761]Undoing temporary vbucket states caused by rebalance
[user:info,2013-05-15T3:24:33.837,ns_1@10.3.3.32:<0.30432.2>:ns_orchestrator:handle_info:403]Rebalance exited with reason {exited,
                              {'EXIT',<0.28006.6>,
                               {replicator_died,
                                {'EXIT',<17674.28481.2>,
                                 {badmatch,{error,closed}}}}}}

ns_server:error,2013-05-15T3:24:33.584,ns_1@10.3.3.32:<0.28006.6>:ns_replicas_builder:build_replicas_main:90]Got premature exit from one of ebucketmigrators: {'EXIT',<17674.28481.2>,
                                                  {badmatch,{error,closed}}}
[error_logger:error,2013-05-15T3:24:33.585,ns_1@10.3.3.32:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
  crasher:
    initial call: ebucketmigrator_srv:init/1
    pid: <17674.28481.2>
    registered_name: []
    exception error: no match of right hand side value {error,closed}
      in function mc_client_binary:cmd_binary_vocal_recv/5
      in call from mc_client_binary:set_vbucket/3
      in call from ebucketmigrator_srv:'-init/1-lc$^0/1-0-'/3
      in call from ebucketmigrator_srv:init/1
    ancestors: [<0.28006.6>,<0.28000.6>,<0.27823.6>,<0.26685.6>]
    messages: []
    links: [#Port<17674.80060>,<0.28006.6>,#Port<17674.80059>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 24
    reductions: 1725
  neighbours:

Attaching collect_info

[MB-8243] memcached crashed in Configuration::getCouchBucket

Mon, 20 May 2013 04:42:27 -0500

I see crash from http://qa.hq.northscale.net/job/CouchbaseServer-SanityTest-4Node-
Centos64/63/consoleFull run on build 2.0.2-795-rel

seems it's the test:
./testrunner -i fournode.ini -t viewquerytests.ViewQueryTests.test_employee_dataset_all_queries,limit=1000,docs-per-day=2,wait_persistence=true

but this job doesn't generate collect info after failures, and I have logs that have been obtained much later

gdb /opt/couchbase/bin/memcached core.memcached.7524
GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 7705]
[New Thread 8584]
[New Thread 8583]
[New Thread 8582]
[New Thread 7709]
[New Thread 7708]
[New Thread 7707]
[New Thread 7706]
[New Thread 7704]
[New Thread 7703]
[New Thread 7702]
[New Thread 7701]
[New Thread 7539]
[New Thread 7538]
[New Thread 7537]
[New Thread 7536]
[New Thread 7535]
[New Thread 7533]
[New Thread 7532]
[New Thread 7524]

warning: .dynamic section for "/usr/lib64/libstdc++.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff9dbe8000
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x0000003866c30285 in raise () from /lib64/libc.so.6
(gdb) t aa bt
A syntax error in expression, near `bt'.
(gdb) t a a bt

Thread 20 (Thread 0x2af656d41220 (LWP 7524)):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6000, tv=<value optimized out>) at epoll.c:404
#2 0x00002af6568b2e44 in event_base_loop (base=0x222b6000, flags=<value optimized out>) at event.c:1558
#3 0x00000000004097d6 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7926

Thread 19 (Thread 7532):
#0 0x0000003866cc545b in read () from /lib64/libc.so.6
#1 0x0000003866c6b677 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x0000003866c6c03e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x0000003866c61124 in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x0000003866c5ffc9 in fgets () from /lib64/libc.so.6
#5 0x00002af656d42939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#6 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#7 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 18 (Thread 7533):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0x1da4e040) at extensions/loggers/file_logger.c:368
#2 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#3 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 17 (Thread 7535):
#0 0x00002af656b17f6f in tcmalloc::CentralFreeList::Populate() () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#1 0x00002af656b17c93 in tcmalloc::CentralFreeList::FetchFromSpansSafe() () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#2 0x00002af656b17bcc in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#3 0x00002af656b1ccd5 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#4 0x00002af656b12054 in tcmalloc::ThreadCache::Allocate(unsigned long, unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#5 0x00002af656b10b60 in (anonymous namespace)::do_malloc(unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#6 0x00002af656b10892 in (anonymous namespace)::do_malloc_or_cpp_alloc(unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#7 0x00002af656b10bec in (anonymous namespace)::do_calloc(unsigned long, unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#8 0x00002af656b24764 in tc_calloc () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#9 0x00002aaaaaef94b9 in HashTable (this=<value optimized out>, i=918, newState=vbucket_state_replica, st=..., checkpointConfig=..., checkpointId=1, initState=vbucket_state_dead)
    at src/stored-value.hh:814
#10 VBucket::VBucket (this=<value optimized out>, i=918, newState=vbucket_state_replica, st=..., checkpointConfig=..., checkpointId=1, initState=vbucket_state_dead) at src/vbucket.hh:126
#11 0x00002aaaaaefb932 in EventuallyPersistentStore::setVBucketState (this=0x2a424400, vbid=918, to=vbucket_state_replica) at src/ep.cc:795
#12 0x00002aaaaaf1e11d in setVBucketState (h=0x2230e900, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.h:508
#13 setVBucket (h=0x2230e900, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.cc:738
#14 processUnknownCommand (h=0x2230e900, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.cc:884
#15 0x00002aaaaaf1e99c in EvpUnknownCommand (handle=<value optimized out>, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.cc:1021
#16 0x00002aaaaacc4dd4 in bucket_unknown_command (handle=<value optimized out>, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at bucket_engine.c:2475
#17 0x0000000000411b4e in process_bin_unknown_packet (c=0x222858c0) at daemon/memcached.c:2882
#18 process_bin_packet (c=0x222858c0) at daemon/memcached.c:3170
#19 complete_nread_binary (c=0x222858c0) at daemon/memcached.c:3744
#20 complete_nread (c=0x222858c0) at daemon/memcached.c:3826
#21 conn_nread (c=0x222858c0) at daemon/memcached.c:5679
#22 0x0000000000405ec5 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x222858c0) at daemon/memcached.c:5942
#23 0x00002af6568b2f3c in event_process_active_single_queue (base=0x222b6500, flags=<value optimized out>) at event.c:1308
#24 event_process_active (base=0x222b6500, flags=<value optimized out>) at event.c:1375
#25 event_base_loop (base=0x222b6500, flags=<value optimized out>) at event.c:1572
#26 0x0000000000414604 in worker_libevent (arg=0x1da51900) at daemon/thread.c:301
#27 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#28 0x0000003866cd325d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 16 (Thread 7536):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6280, tv=<value optimized out>) at epoll.c:404
#2 0x00002af6568b2e44 in event_base_loop (base=0x222b6280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0x1da519f8) at daemon/thread.c:301
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 15 (Thread 7537):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6c80, tv=<value optimized out>) at epoll.c:404
#2 0x00002af6568b2e44 in event_base_loop (base=0x222b6c80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0x1da51af0) at daemon/thread.c:301
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 14 (Thread 7538):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6a00, tv=<value optimized out>) at epoll.c:404
#2 0x00002af6568b2e44 in event_base_loop (base=0x222b6a00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0x1da51be8) at daemon/thread.c:301
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 13 (Thread 7539):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6780, tv=<value optimized out>) at epoll.c:404
#2 0x00002af6568b2e44 in event_base_loop (base=0x222b6780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0x1da51ce0) at daemon/thread.c:301
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 12 (Thread 7701):
#0 0x0000003866c99221 in nanosleep () from /lib64/libc.so.6
#1 0x0000003866cccba4 in usleep () from /lib64/libc.so.6
#2 0x00002aaaaaf35125 in updateStatsThread (arg=0x1da4e4c0) at src/memory_tracker.cc:31
#3 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 11 (Thread 7702):
#0 0x000000386740d594 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003867408e8a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003867408d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002aaaaaf3601a in Mutex::acquire (this=0x223310f0) at src/mutex.cc:79
#4 0x00002aaaaaf7d303 in lock (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26595800, vbucketId=936, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#8 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26595800, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#9 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#10 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
---Type <return> to continue, or q <return> to quit---
#11 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22361ba0) at src/scheduler.cc:153
#12 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22361ba0) at src/scheduler.cc:34
#13 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#14 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 10 (Thread 7703):
#0 0x000000386740d594 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003867408e8a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003867408d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002aaaaaf3601a in Mutex::acquire (this=0x223310f0) at src/mutex.cc:79
#4 0x00002aaaaaf7d303 in lock (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26595200, vbucketId=937, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#8 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26595200, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#9 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#10 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#11 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22361a00) at src/scheduler.cc:153
#12 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22361a00) at src/scheduler.cc:34
#13 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#14 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 9 (Thread 7704):
#0 0x000000386740d594 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003867408e8a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003867408d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002aaaaaf3601a in Mutex::acquire (this=0x223310f0) at src/mutex.cc:79
#4 0x00002aaaaaf7d303 in lock (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26594c00, vbucketId=934, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#8 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26594c00, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#9 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#10 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#11 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22361860) at src/scheduler.cc:153
#12 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22361860) at src/scheduler.cc:34
#13 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#14 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 8 (Thread 7706):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf39631 in wait (this=0x22388d00) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x22388d00) at src/scheduler.cc:139
#3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388d00) at src/scheduler.cc:34
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 7 (Thread 7707):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf39631 in wait (this=0x22388b60) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x22388b60) at src/scheduler.cc:139
#3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388b60) at src/scheduler.cc:34
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 6 (Thread 7708):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf39631 in wait (this=0x223889c0) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x223889c0) at src/scheduler.cc:139
#3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x223889c0) at src/scheduler.cc:34
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 5 (Thread 7709):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf39631 in wait (this=0x22388820) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x22388820) at src/scheduler.cc:139
#3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388820) at src/scheduler.cc:34
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 4 (Thread 8582):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf0fe0f in wait (this=0x2230e900) at src/syncobject.hh:57
#2 wait (this=0x2230e900) at src/syncobject.hh:73
#3 wait (this=0x2230e900) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x2230e900) at src/ep_engine.cc:3379
#5 0x00002aaaaaf0fef3 in EvpNotifyPendingConns (arg=0x2230e900) at src/ep_engine.cc:1153
#6 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#7 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 3 (Thread 8583):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef3688 in wait (this=0x222aa1b0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x222aa1b0, d=...) at src/dispatcher.cc:342
#3 0x00002aaaaaef61ea in Dispatcher::run (this=0x22313c00) at src/dispatcher.cc:184
#4 0x00002aaaaaef69ad in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
#5 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 2 (Thread 8584):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef3688 in wait (this=0x222ab0e0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x222ab0e0, d=...) at src/dispatcher.cc:342
#3 0x00002aaaaaef61ea in Dispatcher::run (this=0x22313500) at src/dispatcher.cc:184
#4 0x00002aaaaaef69ad in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
#5 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x49688940 (LWP 7705)):
#0 0x0000003866c30285 in raise () from /lib64/libc.so.6
#1 0x0000003866c31d30 in abort () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#2 0x00002aaaaaf36070 in Mutex::acquire (this=0x2230e828) at src/mutex.cc:83
#3 0x00002aaaaaf83fe8 in lock (this=0x2230e828, key="couch_bucket") at src/locks.hh:48
#4 LockHolder (this=0x2230e828, key="couch_bucket") at src/locks.hh:26
#5 Configuration::getString (this=0x2230e828, key="couch_bucket") at src/configuration.cc:38
#6 0x00002aaaaaf8e5eb in Configuration::getCouchBucket (this=0x2230e828) at src/generated_configuration.cc:71
#7 0x00002aaaaaf7c59e in CouchNotifier::selectBucket (this=0x22331000) at src/couch-kvstore/couch-notifier.cc:721
#8 0x00002aaaaaf7cc0f in CouchNotifier::processInput (this=0x22331000) at src/couch-kvstore/couch-notifier.cc:606
#9 0x00002aaaaaf7c199 in maybeProcessInput (this=0x22331000, rh=0x22385540) at src/couch-kvstore/couch-notifier.cc:546
#10 CouchNotifier::sendCommand (this=0x22331000, rh=0x22385540) at src/couch-kvstore/couch-notifier.cc:439
#11 0x00002aaaaaf7d478 in CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:774
#12 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26594600, vbucketId=935, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#13 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26594600, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#14 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#15 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#16 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22388ea0) at src/scheduler.cc:153
#17 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388ea0) at src/scheduler.cc:34
#18 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#19 0x0000003866cd325d in clone () from /lib64/libc.so.6

core file:
root(couchbase)@10.3.3.30:/tmp/core.memcached.7524

[MB-7849] cbtransfer crashes with Python 2.4 and couchstore-files as source

Sun, 19 May 2013 04:49:24 -0500

Steps to reproduce (build 170, VM to reproduce - 10.3.3.95):
1. Create default bucket and load 100k items using mcsoda.
2. Use the below command to transfer data to file:
/opt/couchbase/bin/cbtransfer -v -v -v couchstore-files:///opt/couchbase/var/lib/couchbase/data/ /tmp/backup
(there is sufficient disk space in /tmp)

Getting the below error:
[root@caper-012 ~]# /opt/couchbase/bin/cbtransfer -v -v -v couchstore-files:///opt/couchbase/var/lib/couchbase/data/ /tmp/backup
2013-03-01 03:19:31,359: mt cbtransfer...
2013-03-01 03:19:31,360: mt source : couchstore-files:///opt/couchbase/var/lib/couchbase/data/
2013-03-01 03:19:31,360: mt sink : /tmp/backup
2013-03-01 03:19:31,360: mt opts : {'username': None, 'source_vbucket_state': 'active', 'destination_vbucket_state': 'active', 'verbose': 3, 'dry_run': False, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'nmv_retry': 1.0, 'cbb_max_mb': 100000.0, 'try_xwm': 1.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'batch_max_size': 1000.0, 'report': 5.0, 'recv_min_bytes': 4096.0}, 'single_node': False, 'bucket_destination': None, 'destination_operation': None, 'threads': 4, 'key': None, 'password': None, 'id': None, 'bucket_source': None}
2013-03-01 03:19:31,361: mt source_class: <class 'pump_sfd.SFDSource'>
2013-03-01 03:19:31,395: mt sink_class: <class 'pump_bfd.BFDSink'>
2013-03-01 03:19:31,395: mt source_buckets: default
2013-03-01 03:19:31,395: mt bucket: default
2013-03-01 03:19:31,396: mt source_nodes: N/A
2013-03-01 03:19:31,411: mt enqueueing node: N/A
2013-03-01 03:19:31,411: w0 node: N/A
2013-03-01 03:19:31,509: s0 create_db: /tmp/backup/bucket-default/node-N%2FA/data-0000.cbb
2013-03-01 03:19:31,509: s0 connect_db: /tmp/backup/bucket-default/node-N%2FA/data-0000.cbb
....................python: Objects/obmalloc.c:765: PyObject_Malloc: Assertion `bp != ((void *)0)' failed.
Aborted

With less number of items e.g. 10k, this works fine.

[MB-8270] node down with error net_kernal_terminated

Fri, 17 May 2013 17:39:20 -0500

Environment:
  4 windows server r2 64-bit with 4 core, 4GB RAM for each server

Couchbase:
  couchbase server version 2.0.2-801

Run sanity test on windows. When investigate log file for bug ~~MB-8231~~, I see error saying node down with reason "net_kernel_terminated"

2013-05-13 17:59:28: (agent_config.c.705) ERROR: bad JSON configuration from http://127.0.0.1:8091/pools/default/saslBucketsStreaming: Number of vBuckets must be a power of two > 0 and <= 65536 ({
"name": "default",
"nodeLocator": "vbucket",
"saslPassword": "",
"nodes": [{
"hostname": "127.0.0.1:8091",
"ports": {
"direct": 11210,
"proxy": 11211
}
}],
"vBucketServerMap": {
"hashAlgorithm": "CRC",
"numReplicas": 1,
"serverList": ["127.0.0.1:11210"],
"vBucketMap": []
}
})
EOL on stdin. Exiting
[menelaus:info,2013-05-13T18:02:35.839,ns_1@127.0.0.1:<0.20352.3>:menelaus_web_buckets:handle_bucket_delete:345]Deleted bucket "default"

[ns_server:debug,2013-05-13T18:02:36.980,ns_1@127.0.0.1:ns_config_log<0.284.0>:ns_config_log:log_common:111]config change:
memory_quota ->
2184
[ns_server:debug,2013-05-13T18:02:36.980,ns_1@127.0.0.1:ns_config_rep<0.308.0>:ns_config_rep:do_push_keys:317]Replicating some config keys ([memory_quota]..)
[cluster:debug,2013-05-13T18:02:38.136,ns_1@127.0.0.1:ns_cluster<0.273.0>:ns_cluster:handle_call:135]handling add_node("10.3.2.142", 8091, ..)

[cluster:info,2013-05-13T18:02:38.136,ns_1@127.0.0.1:ns_cluster<0.273.0>:ns_cluster:do_change_address:315]Decided to change address to "10.3.2.143"

[user:warn,2013-05-13T18:02:38.136,nonode@nohost:ns_node_disco<0.301.0>:ns_node_disco:handle_info:168]Node nonode@nohost saw that node 'ns_1@127.0.0.1' went down. Details: [{nodedown_reason,
                                                                        net_kernel_terminated}]
[ns_server:info,2013-05-13T18:02:38.136,nonode@nohost:dist_manager<0.263.0>:dist_manager:handle_call:249]Adjusted IP to "10.3.2.143"
[ns_server:info,2013-05-13T18:02:38.136,nonode@nohost:dist_manager<0.263.0>:dist_manager:bringup:227]Attempting to bring up net_kernel with name 'ns_1@10.3.2.143'
[error_logger:info,2013-05-13T18:02:38.136,nonode@nohost:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================PROGRESS REPORT=========================
          supervisor: {local,net_sup}
             started: [{pid,<0.4435.4>},
                       {name,erl_epmd},
                       {mfargs,{erl_epmd,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,2000},
                       {child_type,worker}]

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-801-rel.setup.exe.manifest.xml

Link to collect info of all node https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_2/2013_05/4nodes-202-801_reb_hang_20130513-185157.tgz

[MB-8312] we don't have Table 7.3. and Table 7.4. (clouhbase-cli stuff)in online documentation

Fri, 17 May 2013 14:34:42 -0500

I see that they are presented in http://www.couchbase.com/docs/couchbase-manual-2.0.pdf
Table 7.3. Administration — couchbase Tool Commands
Table 7.4. Administration — Standard couchbase Tool Options

but missed in online documentation for '7.4. couchbase-cli Tool' in http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-cmdline-couchbase-cli.html

[MB-8199] [2.0.2 - RN + Docs] many requests to views causes resource leak, crash

Fri, 17 May 2013 14:28:51 -0500

In response to many view requests against the scatter/gather view merger, a node can allocate so many resources that it will fail to recover.

In one case, this did cause many timeouts in the log leading to max_restart_intensity:
[error_logger:error,2013-04-25T15:23:26.047,ns_1@10.128.16.171:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
     Supervisor: {local,ns_node_disco_sup}
     Context: shutdown
     Reason: reached_max_restart_intensity
     Offender: [{pid,<0.17237.774>},
                  {name,ns_config_rep},
                  {mfargs,{ns_config_rep,start_link,[]}},
                  {restart_type,permanent},
                  {shutdown,1000},
                  {child_type,worker}]

[MB-8310] Couchdb gerrit changes for master must use manifest 2.1-unstable.xml

Fri, 17 May 2013 06:28:32 -0500

2.1-stable.xml was abandoned, and any couchdb master gerrit change depends on couchstore revision more recent than the one listed in 2.1-stable.xml - this means the jobs couchdb-gerrit-views-master and couchdb-gerrit-views-pre-merge-master always fail.

[MB-8153] [Doc'd] cbworkloadgen shows error import sqlite3 module

Thu, 16 May 2013 21:17:32 -0500

Install couchbase server 2.0.2-773
cbworkloadge does not work

Error:

root@cen-0408 thuan]# /opt/couchbase/bin/cbworkloadgen -h
Error: could not import sqlite3 module
[root@cen-0408 thuan]# cd /opt/couchbase/bin
[root@cen-0408 bin]# ./cbworkloadgen -h
Error: could not import sqlite3 module

[root@cen-0408 bin]# ./cbworkloadgen
Error: could not import sqlite3 module

[MB-8019] [2.0.2 RN + doc?] healthchecker - refresh for the new stats available in 2.0.0

Thu, 16 May 2013 20:55:22 -0500

[2.0.2 RN + doc?] healthchecker - refresh for the new stats available in 2.0.0

[MB-8242] Lazy computed outbound XDCR stats per replication are incorrect

Thu, 16 May 2013 19:53:11 -0500

1. Setup the Longevity cluster, West Master to East Master and West Master to West3 cluster on bucket0.
2. Load items,
3. Wait for replication to catch up for both the outbound replication streams.

Notice that"Percentage complete" stats are exactly identical (incorrect) for the 2 replications.
( See attached screenshot)

1. Percentage Complete - Although it is 100 percent replicated, the replication2 shows the same number as the replication1 which is still replicating

Cluster here : http://guinep-s10501:8091

[MB-8273] System test : observed slow swap-rebalance in 2.0.2 ( 15 hours+)

Thu, 16 May 2013 19:23:49 -0500

1. Setup a 6 node cluster with 2 buckets.
2. Each bucket has 1 ddoc and 2 views.
3. Load 60M items on both buckets, data-size 512 bytes consistent
4. Start mutations on the cluster for 2 hours.
5. Wait for initial indexing to complete.
6. Swap rebalance 1-1 node.

- Seeing very high swap ~ 5G on most of the nodes.
- Further inspection of process using swap shows memcached is using most of swap space.
- The memory stats show very high fragmentation
- Views are highly fragmented ( 91 percent) across the cluster.

]# pgrep memcached
9551
# grep --color VmSwap /proc/9551/status
VmSwap: 5219956 kB

[root@orange-11601 ~]# /opt/couchbase/bin/cbstats localhost:11210 -b default allocator
NOTE: SMALL MEMORY MODEL IS IN USE, PERFORMANCE MAY SUFFER.
------------------------------------------------
MALLOC: 14060977616 (13409.6 MiB) Bytes in use by application
MALLOC: + 263405568 ( 251.2 MiB) Bytes in page heap freelist
MALLOC: + 5191139360 ( 4950.7 MiB) Bytes in central cache freelist
MALLOC: + 0 ( 0.0 MiB) Bytes in transfer cache freelist
MALLOC: + 2662928 ( 2.5 MiB) Bytes in thread cache freelists
MALLOC: + 137932952 ( 131.5 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 19656118424 (18745.5 MiB) Actual memory used (physical + swap)
MALLOC: + 126885888 ( 121.0 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 19783004312 (18866.5 MiB) Virtual address space used
MALLOC:
MALLOC: 2348103 Spans in use
MALLOC: 20 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
------------------------------------------------
Size class breakdown
------------------------------------------------
class 1 [ 8 bytes ] : 838 objs; 0.0 MiB; 0.0 cum MiB
class 2 [ 16 bytes ] : 2856430 objs; 43.6 MiB; 43.6 cum MiB
class 3 [ 32 bytes ] : 684331 objs; 20.9 MiB; 64.5 cum MiB
class 4 [ 48 bytes ] : 10987 objs; 0.5 MiB; 65.0 cum MiB
class 5 [ 64 bytes ] : 21645 objs; 1.3 MiB; 66.3 cum MiB
class 6 [ 80 bytes ] : 5347446 objs; 408.0 MiB; 474.3 cum MiB
class 7 [ 96 bytes ] : 905 objs; 0.1 MiB; 474.4 cum MiB
class 8 [ 112 bytes ] : 130 objs; 0.0 MiB; 474.4 cum MiB
class 9 [ 128 bytes ] : 337 objs; 0.0 MiB; 474.4 cum MiB
class 10 [ 144 bytes ] : 224 objs; 0.0 MiB; 474.4 cum MiB
class 11 [ 160 bytes ] : 260 objs; 0.0 MiB; 474.5 cum MiB
class 12 [ 176 bytes ] : 43 objs; 0.0 MiB; 474.5 cum MiB
class 13 [ 192 bytes ] : 83 objs; 0.0 MiB; 474.5 cum MiB
class 14 [ 208 bytes ] : 64 objs; 0.0 MiB; 474.5 cum MiB
class 15 [ 224 bytes ] : 70 objs; 0.0 MiB; 474.5 cum MiB
class 16 [ 240 bytes ] : 66 objs; 0.0 MiB; 474.6 cum MiB
class 17 [ 256 bytes ] : 127 objs; 0.0 MiB; 474.6 cum MiB
class 18 [ 288 bytes ] : 41 objs; 0.0 MiB; 474.6 cum MiB
class 19 [ 320 bytes ] : 76 objs; 0.0 MiB; 474.6 cum MiB
class 20 [ 352 bytes ] : 119 objs; 0.0 MiB; 474.7 cum MiB
class 21 [ 384 bytes ] : 57 objs; 0.0 MiB; 474.7 cum MiB
class 22 [ 416 bytes ] : 28 objs; 0.0 MiB; 474.7 cum MiB
class 23 [ 448 bytes ] : 52 objs; 0.0 MiB; 474.7 cum MiB
class 24 [ 480 bytes ] : 36 objs; 0.0 MiB; 474.7 cum MiB
class 25 [ 512 bytes ] : 907 objs; 0.4 MiB; 475.2 cum MiB
class 26 [ 576 bytes ] : 165 objs; 0.1 MiB; 475.3 cum MiB
class 27 [ 640 bytes ] : 18 objs; 0.0 MiB; 475.3 cum MiB
class 28 [ 704 bytes ] : 60 objs; 0.0 MiB; 475.3 cum MiB
class 29 [ 768 bytes ] : 4872097 objs; 3568.4 MiB; 4043.7 cum MiB
class 30 [ 896 bytes ] : 42 objs; 0.0 MiB; 4043.8 cum MiB
class 31 [ 1024 bytes ] : 96 objs; 0.1 MiB; 4043.9 cum MiB
class 32 [ 1152 bytes ] : 50 objs; 0.1 MiB; 4043.9 cum MiB
class 33 [ 1280 bytes ] : 38 objs; 0.0 MiB; 4044.0 cum MiB
class 34 [ 1408 bytes ] : 33 objs; 0.0 MiB; 4044.0 cum MiB
class 35 [ 1536 bytes ] : 20 objs; 0.0 MiB; 4044.0 cum MiB
class 36 [ 1792 bytes ] : 78 objs; 0.1 MiB; 4044.2 cum MiB
class 37 [ 2048 bytes ] : 122 objs; 0.2 MiB; 4044.4 cum MiB
class 38 [ 2304 bytes ] : 12 objs; 0.0 MiB; 4044.4 cum MiB
class 39 [ 2560 bytes ] : 5 objs; 0.0 MiB; 4044.5 cum MiB
class 40 [ 2816 bytes ] : 16 objs; 0.0 MiB; 4044.5 cum MiB
class 42 [ 3328 bytes ] : 52 objs; 0.2 MiB; 4044.7 cum MiB
class 43 [ 4096 bytes ] : 29 objs; 0.1 MiB; 4044.8 cum MiB
class 47 [ 6656 bytes ] : 64 objs; 0.4 MiB; 4045.2 cum MiB
class 48 [ 8192 bytes ] : 14 objs; 0.1 MiB; 4045.3 cum MiB
class 49 [ 9216 bytes ] : 27 objs; 0.2 MiB; 4045.5 cum MiB
class 53 [ 16384 bytes ] : 7 objs; 0.1 MiB; 4045.6 cum MiB
class 54 [ 20480 bytes ] : 1 objs; 0.0 MiB; 4045.7 cum MiB
class 57 [ 32768 bytes ] : 13 objs; 0.4 MiB; 4046.1 cum MiB
class 61 [ 65536 bytes ] : 1 objs; 0.1 MiB; 4046.1 cum MiB
class 85 [ 262144 bytes ] : 1 objs; 0.2 MiB; 4046.4 cum MiB
------------------------------------------------
PageHeap: 26 sizes; 251.2 MiB free; 121.0 MiB unmapped
------------------------------------------------
     1 pages * 27211 spans ~ 212.6 MiB; 212.6 MiB cum; unmapped: 5.4 MiB; 5.4 MiB cum
     2 pages * 3062 spans ~ 47.8 MiB; 260.4 MiB cum; unmapped: 12.3 MiB; 17.7 MiB cum
     3 pages * 1018 spans ~ 23.9 MiB; 284.3 MiB cum; unmapped: 18.4 MiB; 36.1 MiB cum
     4 pages * 721 spans ~ 22.5 MiB; 306.8 MiB cum; unmapped: 20.1 MiB; 56.2 MiB cum
     5 pages * 175 spans ~ 6.8 MiB; 313.7 MiB cum; unmapped: 6.8 MiB; 63.0 MiB cum
     6 pages * 113 spans ~ 5.3 MiB; 319.0 MiB cum; unmapped: 5.3 MiB; 68.3 MiB cum
     7 pages * 50 spans ~ 2.7 MiB; 321.7 MiB cum; unmapped: 2.7 MiB; 71.0 MiB cum
     8 pages * 101 spans ~ 6.3 MiB; 328.0 MiB cum; unmapped: 6.1 MiB; 77.1 MiB cum
     9 pages * 23 spans ~ 1.6 MiB; 329.6 MiB cum; unmapped: 1.6 MiB; 78.7 MiB cum
    10 pages * 18 spans ~ 1.4 MiB; 331.0 MiB cum; unmapped: 1.4 MiB; 80.1 MiB cum
    11 pages * 10 spans ~ 0.9 MiB; 331.9 MiB cum; unmapped: 0.9 MiB; 80.9 MiB cum
    12 pages * 11 spans ~ 1.0 MiB; 332.9 MiB cum; unmapped: 1.0 MiB; 82.0 MiB cum
    13 pages * 1 spans ~ 0.1 MiB; 333.0 MiB cum; unmapped: 0.1 MiB; 82.1 MiB cum
    14 pages * 2 spans ~ 0.2 MiB; 333.2 MiB cum; unmapped: 0.2 MiB; 82.3 MiB cum
    15 pages * 3 spans ~ 0.4 MiB; 333.6 MiB cum; unmapped: 0.4 MiB; 82.6 MiB cum
    16 pages * 3 spans ~ 0.4 MiB; 334.0 MiB cum; unmapped: 0.2 MiB; 82.9 MiB cum
    17 pages * 1 spans ~ 0.1 MiB; 334.1 MiB cum; unmapped: 0.0 MiB; 82.9 MiB cum
    20 pages * 1 spans ~ 0.2 MiB; 334.2 MiB cum; unmapped: 0.2 MiB; 83.0 MiB cum
    21 pages * 1 spans ~ 0.2 MiB; 334.4 MiB cum; unmapped: 0.2 MiB; 83.2 MiB cum
    29 pages * 1 spans ~ 0.2 MiB; 334.6 MiB cum; unmapped: 0.2 MiB; 83.4 MiB cum
    49 pages * 18 spans ~ 6.9 MiB; 341.5 MiB cum; unmapped: 6.9 MiB; 90.3 MiB cum
    58 pages * 1 spans ~ 0.5 MiB; 342.0 MiB cum; unmapped: 0.5 MiB; 90.8 MiB cum
    75 pages * 1 spans ~ 0.6 MiB; 342.6 MiB cum; unmapped: 0.6 MiB; 91.4 MiB cum
    98 pages * 8 spans ~ 6.1 MiB; 348.7 MiB cum; unmapped: 6.1 MiB; 97.5 MiB cum
   100 pages * 1 spans ~ 0.8 MiB; 349.5 MiB cum; unmapped: 0.8 MiB; 98.3 MiB cum
   122 pages * 1 spans ~ 1.0 MiB; 350.4 MiB cum; unmapped: 1.0 MiB; 99.2 MiB cum
>255 large * 13 spans ~ 21.8 MiB; 372.2 MiB cum; unmapped: 21.8 MiB; 121.0 MiB cum

[root@orange-11601 ~]#
[root@orange-11601 ~]# /opt/couchbase/bin/cbstats localhost:11210 -b default memory
bytes: 8468018888
ep_kv_size: 8190890236
ep_max_data_size: 13212057600
ep_mem_high_wat: 11230248960
ep_mem_low_wat: 9909043200
ep_mem_tracker_enabled: true
ep_oom_errors: 0
ep_overhead: 128884714
ep_tmp_oom_errors: 0
ep_value_size: 6628874716
mem_used: 8468018888
tcmalloc_current_thread_cache_bytes: 2581856
tcmalloc_max_thread_cache_bytes: 4194304
tcmalloc_unmapped_bytes: 126844928
total_allocated_bytes: 14068670960
total_fragmentation_bytes: 5313338896
total_free_bytes: 263061504
total_heap_bytes: 19645071360

added screenshot of the cluster

[MB-8303] [system test] Rebalance in hangs with some initial vbucket movement due to bucket state bouncing between not ready and active

Thu, 16 May 2013 18:57:57 -0500

Cluster ip is 172.23.105.23
1. create 8 nodes cluster, each node has 12G RAM, HHD
2. create 2 buckets default and saslbucket, with memory quota 6G and 4G
3. Run the KV use case for 2 days:
loading 35M items to each bucket, make resident ratio 70%~80%, access the data 15k ops/sec with 5% create, 5% delete, 5%expire, 5% update, 80 gets for several hours.
4. Then with the same work load, run some rebalance operations. When proceed to rebalance in 2 nodes, after several vbucket movement (17 vbucket for saslbucket), rebalance hangs.

When check the log, see a lot of bucket state change between not ready and active:

root@cola-s10305:/opt/couchbase/var/lib/couchbase/logs# tail -f error.1
<<"replication_building_1013_'ns_1@172.23.105.33'">>} took too long: 1196997 us
[ns_server:error,2013-05-16T15:13:17.111,ns_1@172.23.105.32:<0.11769.0>:ns_memcached:verify_report_long_call:294]call {stats,<<>>} took too long: 556521 us
[ns_server:error,2013-05-16T15:19:19.834,ns_1@172.23.105.32:<0.11769.0>:ns_memcached:verify_report_long_call:294]call {get_tap_docs_estimate,1013,
<<"replication_building_1013_'ns_1@172.23.105.33'">>} took too long: 725657 us
[ns_server:error,2013-05-16T15:22:18.618,ns_1@172.23.105.32:ns_memcached-saslbucket<0.11752.0>:ns_memcached:handle_info:671]handle_info(ensure_bucket,..) took too long: 661061 us
[ns_server:error,2013-05-16T15:23:35.127,ns_1@172.23.105.32:<0.11704.0>:ns_memcached:verify_report_long_call:294]call {stats,<<>>} took too long: 568954 us
[ns_server:error,2013-05-16T15:25:59.291,ns_1@172.23.105.32:<0.11705.0>:ns_memcached:verify_report_long_call:294]call topkeys took too long: 1217646 us
[ns_server:error,2013-05-16T15:30:17.553,ns_1@172.23.105.32:ns_doctor<0.9998.0>:ns_doctor:update_status:234]The following buckets became not ready on node 'ns_1@172.23.105.23': ["saslbucket"], those of them are active ["saslbucket"]
[ns_server:error,2013-05-16T15:30:37.540,ns_1@172.23.105.32:ns_doctor<0.9998.0>:ns_doctor:update_status:234]The following buckets became not ready on node 'ns_1@172.23.105.23': ["saslbucket"], those of them are active ["saslbucket"]

diag links:
https://s3.amazonaws.com/bugdb/jira/MB-8303/8nodes_202-800_rebalance_hang_20130516-143321.tgz

[MB-8272] there is README file from Erlang distro (on Windows) that should be removed

Thu, 16 May 2013 17:11:12 -0500

In couchbase 2.0.2-xx for mac and windows, there is a README file at root directory. I don't see this README file in rpm and deb package after install.
Is it what we intend to do for rpm and deb package or the bug?

[MB-8013] Implement detailed rebalance progress report

Dependency — Thu, 16 May 2013 16:39:27 -0500

"We need to measure progress better. When vbuckets have lots of items we'll currently display progress as 'stuck' because rebalance progress is currently measure in terms of vbucket movements.

From user perspective we should be able to track how many items/bytes needs to be moved and how many is left. This will likely need some help from ep-engine folks.

One of the goals of 1.8.1 is better rebalance progress estimation. Particularly we should present user some ETA that's not too far off.

We've discussed this with Chiyoung today. Here's what we came up as initial approach.

ns_server is aware of all vbucket movements that need to be done. And in 1.8.x (maybe later than 1.8.1) we will also build replicas during rebalance. So building of replicas will also be taken into account in same way as takeovers.

For each needed movement we will look at vbucket stats on source and destination and see if backfill is needed or not. If backfill is needed we know how many items (or bytes, hopefully) will be moved. If backfill is not needed we will look at checkpoint stats and get same information. Then for each in-flight vbucket movement we already have stat that tells us how many items are pending in particular tap cursor. We will use vbucket movement completion (how many vbucket movements are done out from total count of movements we need) and this stat (how far are currently in-flight movements from done) to get us % of completion. And having rate of % completion change we'll get ETA.

We know its not taking into account on-going mutations and not taking into account temporary oom NAKs. But hopefully that still won't be too far off. I think that because we're going to refresh our estimates periodically by looking at vbucket stats for vbuckets which movement is still pending, it will account for on-going mutations and should work well enough.

http://www.pivotaltracker.com/story/show/22972101

[MB-8280] cbrecovery should check if tap batch requests are processed correctly before moving to next batch

Thu, 16 May 2013 16:15:36 -0500

All recovered msgs are pumped into destination cluster in batch mode. If due to slow traffic or poor server performance, batch msg processing may fail.

Recovery tool should make sure every batch is processed correctly before moving to next one. Acknowledgement request for the last msg in batch may be an option.

[MB-8226] Implement ns_server side memcached API for new get_meta_batch and update_meta_batch

Thu, 16 May 2013 16:15:27 -0500

In MB-8213, ep_engine team will create two new batch operations, get_meta_batch and update_meta_batch, which will be built on new protocol. From ns_server side, we need to implement new memcached APIs correspondingly.

[MB-8034] couchbase-cli needs to add the new rest api for optimistic XDCR

Thu, 16 May 2013 16:15:09 -0500

Please see Dipti's comment in ~~MB-7701~~

Task to document at this ticket:

http://www.couchbase.com/issues/secure/RapidBoard.jspa?rapidView=22&view=planning&selectedIssue=MB-7959&epics=hidden

[MB-8298] remote_cluster_info module should return remote memcached access info

Thu, 16 May 2013 16:11:24 -0500

In 2.1, XDCR will provide an option to replicate to remote memcached instead of CAPI.

This requires XDCR core infrastructure to understand how to access memcached at remote cluster.

For example, in order to replicate remote bucket "default", vbucket 123, XDCR need to know

1) node (ip address) for this vbucket at remote cluster;
2) memcached port of the node where the vbucket lives;
3) credentials to access the memcached at that node

Module remote_cluster_info is a natural home for such information.

Today 1) has been already encoded in the vbucketmap maintained by remote_cluster_info, but 2) and 3) are not available. This task will expand remote_cluster_info module to include 2) and 3).

In particular, XDCR need an API to return 1), 2), 3) above, e.g.,

remote_cluster_info:fetch_remote_memcached_info(RemoteClusterId, Bucket, VBucket)

returns
{"10.3.114.2", 11998, "_admin", "_password"}

[MB-8139] couchbase-cli - not able to add server

Thu, 16 May 2013 15:45:42 -0500

I am trying to add a server using couchbase-cli. Command :

./couchbase-cli server-add -c caper-002.hq.northscale.net:8091 --server-add=caper-004.hq.northscale.net:8091 -u Administrator -p password

and getting the following error :

ERROR: unable to server-add caper-004.hq.northscale.net:8091 (400) Bad Request
[u'Prepare join failed. Authentication failed. Verify username and password. Got HTTP status 401 from REST call post to http://caper-004.hq.northscale.net:8091/engageCluster2. Body was: []']
ERROR: command: server-add: caper-002.hq.northscale.net:8091, 2

Tried using IP address instead of hostname :

./couchbase-cli server-add -c caper-002.hq.northscale.net:8091 --server-add=10.3.3.96:8091 -u Administrator -p password

but getting the same error:

ERROR: unable to server-add 10.3.3.96:8091 (400) Bad Request
[u'Prepare join failed. Authentication failed. Verify username and password. Got HTTP status 401 from REST call post to http://10.3.3.96:8091/engageCluster2. Body was: []']
ERROR: command: server-add: 10.3.3.100:8091, 2

Build used : 2.0.2-769-rel

[MB-8229] [Doc'd] Couchbase UI shows a node down when Rest API is used to rename a node in a multiple node cluster

Thu, 16 May 2013 15:28:58 -0500

When we reference a node from hostname to IP in a multiple node cluster, using REST API, then UI of another node shows that first node is down.

In Detail:

Steps followed:

1. Take a 2-node cluster referred with hostame (lets say node A and node B).
2. Rename node A from hostname to its corresponding IP, using the rest API - node/controller/rename.

Result: Node A gets renamed to its corresponding IP but the UI of other node(i.e. node B) shows that node A is down.

Attaching cbcollectinfo from both the nodes. Also uploading the screen-shot of UI. Please see.

[MB-7996] [system test] rebalance hang when add a node to cluster

Thu, 16 May 2013 13:45:25 -0500

Install couchbase server 2.0.1-185 on 4 physical servers with 2 separated disks
Create a cluster with 3 nodes
10.2.1.61
10.2.1.62
10.2.1.63
Create 2 buckets: default (14GB) and sasl (10GB)
No view or xdcr created
Load 20+ million items to both bucket until resident ratio on both bucket around 90%
Access cluster in 3 hours with spec in this page http://hub.internal.couchbase.com/confluence/pages/viewpage.action?pageId=6785119
Add node 10.2.1.64 to cluster and rebalance.
Rebalance failed. Filed bug ~~MB-7995~~

Start rebalance again. Rebalance hang

Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201304/4phy-win-201_185-reb-not-moving_node-61-erl-frozen-20130401-122033.tgz

Link to manifest file of the build http://builds.hq.northscale.net/latestbuilds/couchbase-server-community_x86_64_2.0.1-185-rel.setup.exe.manifest.xml

[MB-8293] Alert UI annoyingly flashes as metadata is used

Duplicate — Thu, 16 May 2013 03:29:06 -0500

As we go beyond 50% metadata, the UI will show alerts in the console. Problem is, the alerts display non-stop, with no way to disable them, making use of the UI very hard. One has to constantly close the alert box.

Scenario:
Load system with many items, such that metadata is consumed.

Expected behavior:
Alert can be acknowledged and won't be displayed again as long as we're in the alert area. Obviously, if we exit the alert range and hit it again, we'd expect to see the alert again.

Observed behavior:
While workload is running, the alert box is always popping up. Incessantly. It cannot be stopped. Yes, I know I'm above 50% metadata, go away. No, really, please go away. GO!