Couchbase

[MB-8083] [2.0.2] memcachetest+moxi/mcsoda: downstream timeout( with automation tests)

Tue, 23 Apr 2013 15:35:47 -0500

know there are already a few bugs with the problem(CBSE-475/498...), but will create a new one, as no separate ticket for 2.0.2 and now there are automation tests to reproduce it

http://qa.hq.northscale.net/view/2.0.1/job/ubuntu-64-2.0-connections-tests
./testrunner -i /tmp/connections-tests.ini get-logs=True,wait_timeout=180,get-cbcollect-info=True -t connectionstests.ConnectionTests.multiple_connections_using_memcachetest

steps in test with 1 node cluster:
1. run moxi on node
nohup /opt/couchbase/bin/moxi -u root -Z usr=Administrator,pwd=password,port_listen=51500,concurrency=1024,wait_queue_timeout=200,connect_timeout=400,connect_max_errors=3,connect_retry_interval=30000,auth_timeout=100,downstream_conn_max=16,downstream_timeout=5000,cycle=200,default_bucket_name=default http://10.5.2.13:8091/pools/default/bucketsStreaming/default -d
2. run mcsoda through moxi
nohup python lib/perf_engines/mcsoda.py 10.5.2.13:51500 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=0.9 ratio-sets=0.9 min-value-size=1024 max-items=1000000 ratio-deletes=0.1 exit-after-creates=0 threads=4 prefix=default &
3.run memcachetest on the node
/tmp/memcachetest/memcachetest -h localhost:11211 -i 100000 -W 320 -t 320 -c 0 -M 2

result:

2013-04-12 08:33:37 | INFO | MainProcess | MainThread | [remote_util.log_command_output] ASCII set failure: SERVER_ERROR proxy downstream timeout
2013-04-12 08:33:37 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Failed to set [29735]: during populate.

the same behaviour is for the test with 3 node. Tests can be expanded through parameters + can also be added additional scenarios as mentioned in http://hub.internal.couchbase.com/confluence/display/QA/QE+Test+Enhancements+for+2.0.2

[MB-8002] Memcached ops sometimes take an extraordinarily long amount of time

Mon, 1 Apr 2013 19:30:51 -0500

We have noticed that memcached ops sometimes take a very long time to complete and can cause ns_server to think that memcached has become unresponsive. We usually see this when a system starts to swap, but we have also occasionally seen this issue even when we're not in swap.

[MB-7887] [RN 2.0.2] Appends can cause large amounts of memory fragmentation in tcmalloc

Duplicate — Tue, 23 Apr 2013 15:13:21 -0500

[RN 2.0.2] Appends can cause large amounts of memory fragmentation in tcmalloc

[MB-7657] [RN 2.0.1] XDCR docs to replicate for bucket queue is draining unevenly.

Thu, 11 Apr 2013 17:38:40 -0500

linux cluster
22-node-ec2-10GB-Linux

source Cluster ( 15 Nodes)
create default bucket -3.5 GB per node
sasl bucket - 2.0 GB per node

destination Cluster ( 7 Nodes)
create default bucket - 3.5 GB per node
sasl bucket - 2.0 GB per node

**Loading phase
Source cluster:
Default bucket
define a workload that loads 40 M json items , key size 128-512 bytes to default bucket.
The workload should push the system into a light dgm ( active resident ratio at 90 percent)

**Access Phase:
Source cluster:
Default bucket
create:5%, uupdate:10%, get:80%, delete:5% with cache miss:5%, opsPerSec:30000, running for 8 hours

Then start replication:
set a bi-directional XDCR on default bucket
Source cluster:
default bucket
create:5%, uupdate:10%, get:80%, delete:5% with cache miss:5%, opsPerSec:30000, running for 2 hours

Destination cluster:
default bucket
define a workload that loads 30 M json items(non-conflicting key-sets) , key size 128-512 bytes to default bucket.

After all the loads on both clusters stop, wait for "XDCR docs to replicate for bucket" drops. But really slow.
Saw XDCR docs to replicate for bucket queue is draining unevenly on default bucket on source
The screen shot is from source cluster.

[MB-6726] Rebalance is slow when indexing/compaction and query load are going on in parallel

Dependency — Thu, 25 Apr 2013 17:40:01 -0500

Cluster information:
- 8 centos 6.2 64bit server with 4 cores CPU
- Each server has 32 GB RAM and 400 GB SSD disk.
- SSD disk format ext4 on /data
- Each server has its own drive, no disk sharing with other server.
- Cluster has 2 buckets, default (11GB) and saslbucket (11GB) with consistent view enable. For 2 buckets, we use only 68% total RAM of system.
- Load 12 million items to saslbucket and 45 million items to default bucket. Each key has size from 512 bytes to 1500 bytes
- Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)

* Create cluster with 4 nodes installed couchbase server 2.0.0-1746

10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40

* Data path /data
* View path /data

* Let load running at 6K ops, query 400 to 500 per second
* Add 4 nodes to cluster and rebalance
10.6.2.42
10.6.2.43
10.6.2.44
10.6.2.45

* rebalance hang. Filed bug ~~MB-6706~~
* Try rebalance again. Cluster rebalances saslbucket first and took 9 hours to complete rebalance saslbucket
* default bucket took 32 hours to complete

Link to collect_info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1746-reb-slow-constn-enable-20120925-141314.tgz

Each atop file size from 1.5 GB to 2.5 GB. If atop file needed, I will upload them as request

[MB-5598] Couchbase Server started too early in boot sequence...IP address wasn't yet ready

Mon, 8 Oct 2012 15:59:35 -0500

Logs attached. The reported problem was that after a power failure, one node of a 2-node Couchbase cluster returned with a reset configuration.

In the Couchbase logs, right after the restart, we are unable to listen on the IP address we think we should be listening on:
ERROR REPORT <0.57.0> 2012-06-18 09:12:01
===============================================================================

Got error:eaddrnotavail. Cannot listen on configured address:192.168.1.8

I see in the /var/log/messages that the DHCP client got the address 3 seconds after we tried to listen on it:
Jun 18 09:12:03 cheetah dhclient: bound to 192.168.1.8 -- renewal in 807476524 seconds.

And then because we don't know who we are:

INFO REPORT <6044.171.0> 2012-06-18 09:12:02
===============================================================================

ns_1@127.0.0.1:<6044.171.0>:ns_node_disco:189: We've been shunned (nodes_wanted = ['ns_1@192.168.1.71',
'ns_1@192.168.1.8']). Leaving cluster.

INFO REPORT <6044.66.0> 2012-06-18 09:12:02
===============================================================================

ns_log: logging ns_cluster:1:Node 'ns_1@127.0.0.1' is leaving cluster.

Then we spiral around a bunch, seemingly resetting the configuration a number of times (not sure what that's all about, seems like spamming the logs for a few minutes). We settle into a single node cluster, then magically reboot:

INFO REPORT <0.54.0> 2012-06-18 09:19:27
===============================================================================

nonode@nohost:<0.54.0>:log_os_info:25: OS type: {unix,linux} Version: {2,6,32}
Runtime info: [{otp_release,"R14B03"},

Try to listen on the correct address again:

INFO REPORT <0.57.0> 2012-06-18 09:19:27
===============================================================================

nonode@nohost:<0.57.0>:dist_manager:105: Attempting to bring up net_kernel with name 'ns_1@192.168.1.8'

But it's too late, we've already been kicked out of the cluster and reset the config.
--------------------------------------------------------------------------------------------------------------------------------

Adding the ns_server component since I think it could definitely handle this case much better, and probably retry to listen on the correct IP address a few times (more than 1) before wiping the config

Adding the linux_installer component since it would probably be a best practice to configure Couchbase as one of the very last services that starts up to ensure the rest of the system is ready when we come up.

[MB-7282] erlang's global naming facility apparently drops globally registered service with actual service still alive (was: impossible to change settings/autoFailover after rebalance)

Thu, 2 May 2013 17:31:37 -0500

build
./testrunner -i resources/jenkins/centos-64-5node-failover.ini get-logs=True -t autofailovertests.AutoFailoverTests.test_enable,replicas=2,keys-count=1000000,num-buckets=2
http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/480/consoleFull

test changes settings/autoFailover after rebalance

test logs:

2012-11-27 22:29:42,105 - INFO - MainProcess:MainThread - rest_client:_rebalance_progress - rebalance percentage : 99.5678770149 %
2012-11-27 22:29:44,114 - INFO - MainProcess:MainThread - rest_client:_rebalance_progress - rebalance percentage : 99.6821896236 %
2012-11-27 22:29:48,134 - INFO - MainProcess:MainThread - rest_client:monitorRebalance - rebalance progress took 1111.69762397 seconds
2012-11-27 22:29:48,135 - INFO - MainProcess:MainThread - rest_client:monitorRebalance - sleep for 10 seconds after rebalance...
2012-11-27 22:29:58,133 - INFO - MainProcess:MainThread - rest_client:update_autofailover_settings - settings/autoFailover params : enabled=true&timeout=30
2012-11-27 22:29:58,140 - ERROR - MainProcess:MainThread - rest_client:_http_request - http://10.1.3.114:8091/settings/autoFailover error 500 reason: unknown ["Unexpected server error, request logged."]

server logs:

[menelaus:warn,2012-11-27T22:53:28.404,ns_1@10.1.3.114:<0.21541.23>:menelaus_web:loop:430]Server error during processing: ["web request failed",
                                 {path,"/settings/autoFailover"},
                                 {type,exit},
                                 {what,
                                  {noproc,
                                   {gen_server,call,
                                    [{global,auto_failover},
                                     {enable_auto_failover,30,1}]}}},
                                 {trace,
                                  [{gen_server,call,2},
                                   {menelaus_web,
                                    handle_settings_auto_failover_post,1},
                                   {menelaus_web,loop,3},
                                   {mochiweb_http,headers,5},
                                   {proc_lib,init_p_do_apply,3}]}]

Alk, if we have to handle the error and retry again, please assign the ticket back to me

[MB-7941] [windows] Rebalance exited with reason {unexpected_exit,{badmatch,{error,etimedout}

Fri, 3 May 2013 02:29:48 -0500

Test to reproduce:

./testrunner -i vm-list.ini -t swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=3,num-swap=1,swap-orchestrator=True

This looks related to timeout. Filing this as separate as the reason/crash reports are different than before.

[user:info,2013-03-19T13:02:32.089,ns_1@10.142.174.97:<0.30244.1190>:ns_orchestrator:handle_info:319]Rebalance exited with reason {unexpected_exit,
                              {'EXIT',<0.27519.1205>,
                               {{badmatch,
                                 [{'EXIT',
                                   {{badmatch,{error,etimedout}},
                                    {gen_server,call,



=========================CRASH REPORT=========================
  crasher:
    initial call: ebucketmigrator_srv:init/1
    pid: <15682.11394.469>
    registered_name: []
    exception error: no match of right hand side value {error,etimedout}
      in function ebucketmigrator_srv:connect/4
      in call from ebucketmigrator_srv:init/1
    ancestors: [<0.27512.1205>,<0.27480.1205>,<0.23778.1205>,<0.23697.1205>]
    messages: [{'$gen_call',{<0.27521.1205>,#Ref<0.0.5074.212183>},
                               had_backfill}]

[error_logger:error,2013-03-19T13:02:31.075,ns_1@10.142.174.97:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
  crasher:
    initial call: new_ns_replicas_builder:init/1
    pid: <0.27512.1205>
    registered_name: []
    exception exit: {{badmatch,[{<15682.11394.469>,noproc}]},
                     [{misc,sync_shutdown_many_i_am_trapping_exits,1},
                      {misc,try_with_maybe_ignorant_after,2},
                      {gen_server,terminate,6},
                      {proc_lib,init_p_do_apply,3}]}
      in function gen_server:terminate/6
    ancestors: [<0.27480.1205>,<0.23778.1205>,<0.23697.1205>]
    messages: [{'EXIT',<0.27480.1205>,shutdown}]

[error_logger:error,2013-03-19T13:02:31.075,ns_1@10.142.174.97:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
  crasher:
    initial call: ns_single_vbucket_mover:mover/6
    pid: <0.27480.1205>
    registered_name: []
    exception exit: {unexpected_exit,
                     {'EXIT',<0.27519.1205>,
                      {{badmatch,
                        [{'EXIT',
                          {{badmatch,{error,etimedout}},
                           {gen_server,call,

Attaching the collect_info.zip and testrunner logs.

[MB-7919] [windows] Rebalance exited with reason {bulk_set_vbucket_state_failed, {timeout, when data mutation is in progress

Fri, 3 May 2013 02:30:05 -0500

Rebalance exited with below reason when data mutation is in progress with rebalance

{{bulk_set_vbucket_state_failed,
                               [{'ns_1@10.142.85.118',
                                 {'EXIT',
                                  {{{timeout,

Test to repro:
nohup ./testrunner -i win-aws.ini -t rebalance.rebalancein.RebalanceInTests.incremental_rebalance_in_with_ops,replicas=2,items=100000,doc_ops=update &

The test failed at 2013-03-15 20:53:08 and the info.18 log file at the orchestrator nodes has the logs. Below crash reports can be seen.

=========================CRASH REPORT=========================
  crasher:
    initial call: ns_single_vbucket_mover:mover/6
    pid: <0.32365.663>
    registered_name: []
    exception exit: {{{timeout,
                          {gen_server,call,
                              [<21714.30742.56>,
                               {start_vbucket_filter_change,


=========================CRASH REPORT=========================
  crasher:
    initial call: ns_single_vbucket_mover:mover/6
    pid: <0.32167.663>
    registered_name: []
    exception exit: {{{timeout,
                          {gen_server,call,
                              [<21714.30742.56>,
                               {start_vbucket_filter_change,

=========================CRASH REPORT=========================
  crasher:
    initial call: ns_single_vbucket_mover:mover/6
    pid: <0.31877.663>
    registered_name: []
    exception exit: {unexpected_exit,
                     {'EXIT',<0.48.664>,
                      {{wait_checkpoint_persisted_failed,"default",853,4,
                        [{'ns_1@10.142.85.118',
                          {'EXIT',
                           {{{timeout,
                              {gen_server,call,
                               [<21714.30742.56>,
                                {start_vbucket_filter_change,


Attaching the testrunner logs and collect_info.zip.

I executed the same test again and it passed so this is not consistently reproducible.

[MB-6232] ep-engine needs 1.5 minutes to create 1k vbuckets. Seems too slow (but gets fast with barrier=0)

Duplicate — Tue, 7 May 2013 00:54:57 -0500

SUBJ.

Just try single node dev cluster and try to load beers sample (it'll fail for perhaps different reason).

You can observe in logs:

We're doing set_vbucket_state requests from janitor (ns_memcached messages) and this is the first messages from mccouch about just created vbucket (capi_set_view_manager messages and mc_connection messages). Note: I've added extra log messages at the beginning of sync_notify and end to see if it's couchdb's or ns_servers' fault. From timestamps it's clear that it's not.

[ns_server:debug,2012-08-15T9:47:40.309,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:106]sending out sync_notify
[ns_server:info,2012-08-15T9:47:40.309,n_0@127.0.0.1:<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 740 state to active
[views:debug,2012-08-15T9:47:40.309,n_0@127.0.0.1:mc_couch_events:capi_set_view_manager:handle_mc_couch_event:418]Got set_vbucket event for beer-sample/1023. Updated state: active
[ns_server:info,2012-08-15T9:47:40.309,n_0@127.0.0.1:<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 739 state to active
[ns_server:debug,2012-08-15T9:47:40.309,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:113]done
[ns_server:info,2012-08-15T9:47:40.310,n_0@127.0.0.1:<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 738 state to active
[ns_server:info,2012-08-15T9:47:40.310,n_0@127.0.0.1:<0.870.0>:ns_memcached:do_handle_call:485]Changed vbucket 737 state to active

And last message is at:

[ns_server:debug,2012-08-15T9:49:14.410,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:106]sending out sync_notify
[views:debug,2012-08-15T9:49:14.410,n_0@127.0.0.1:mc_couch_events:capi_set_view_manager:handle_mc_couch_event:418]Got set_vbucket event for beer-sample/1. Updated state: active
[ns_server:debug,2012-08-15T9:49:14.410,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:113]done
[ns_server:debug,2012-08-15T9:49:14.501,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:106]sending out sync_notify
[views:debug,2012-08-15T9:49:14.502,n_0@127.0.0.1:mc_couch_events:capi_set_view_manager:handle_mc_couch_event:418]Got set_vbucket event for beer-sample/0. Updated state: active
[ns_server:debug,2012-08-15T9:49:14.502,n_0@127.0.0.1:<0.868.0>:mc_connection:do_notify_vbucket_update:113]done

I've just retried with remounting with barrier=0 and it was blazing fast. What this mount option changes is short fsync are nearly instant because they merely send data to disk's buffer without waiting until data actually hits platter.

So that's evidence of some excessive use of fsyncs somewhere during vbucket db files creation and that's clearly on ep-engine/couchstore side.

Set this to blocker as it'll affect people trying to create buckets on real hardware, hard disks and any modern linux distro (where barriers are on by default).

[MB-8210] Update the incorrect information on the Automated Index Updates

Tue, 7 May 2013 18:20:49 -0500

We need to fix this page http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-views-operation-autoupdate.html ?

Relevant information about how it actually works.

"Every updateInterval milliseconds it checks if index file is more than updateMinChanges behind .couch files (which is itself behind in-memory source of truth, potentially for tens of seconds). And true, it triggers view update."

[MB-7168] [Done- RN 2.0.2] failover of node that's completely down is still not quick (was: Rebalance exited with reason {not_all_nodes_are_ready_yet after failover node)

Sun, 12 May 2013 05:18:59 -0500

version=2.0.0-1947-rel
http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/448/consoleFull
/testrunner -i resources/jenkins/centos-64-5node-failover.ini get-logs=True,GROUP=BAT -t failovertests.FailoverTests.test_failover_firewall,replica=1,keys_count=100000,dgm_run=True,GROUP=BAT

steps:
1.3 nodes in cluster:10.1.3.114, 10.1.3.118,10.1.3.116
2.enabled firewall on ip:10.1.3.118 : /sbin/iptables -A INPUT -p tcp -i eth0 --dport 1000:60000 -j REJECT
3
[2012-11-12 11:28:37,102] - [failovertests:260] INFO - node 10.1.3.118:8091 is 'unhealthy' as expected
[2012-11-12 11:29:07,544] - [rest_client:849] INFO - fail_over successful
[2012-11-12 11:29:07,545] - [failovertests:278] INFO - failed over node : ns_1@10.1.3.118
[2012-11-12 11:29:07,545] - [failovertests:292] INFO - 10 seconds sleep after failover before invoking rebalance...
4.[2012-11-12 11:29:17,545] - [rest_client:883] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.1.3.118&user=Administrator&knownNodes=ns_1%4010.1.3.114%2Cns_1%4010.1.3.118%2Cns_1%4010.1.3.116

result:
[2012-11-12 11:29:17,562] - [rest_client:890] INFO - rebalance operation started
[2012-11-12 11:29:17,569] - [rest_client:986] INFO - rebalance percentage : 0 %
[2012-11-12 11:29:19,573] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:21,577] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:23,584] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:25,589] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:27,593] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:29,599] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:31,603] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:33,607] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:35,612] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:37,616] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:39,621] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:41,626] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:43,630] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:45,635] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:47,640] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:49,644] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:51,648] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:53,652] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:55,657] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:57,662] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:29:59,667] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:30:01,671] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:30:03,676] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:30:05,682] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:30:07,698] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:30:09,703] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:30:11,708] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:30:13,712] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:30:15,716] - [rest_client:986] INFO - rebalance percentage : 0.0 %
[2012-11-12 11:30:17,721] - [rest_client:971] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed

[rebalance:info,2012-11-12T11:49:03.581,ns_1@10.1.3.114:<0.20240.1>:ns_rebalancer:rebalance:258]Waiting for bucket "default" to be ready on ['ns_1@10.1.3.114',
                                             'ns_1@10.1.3.116']
[ns_server:info,2012-11-12T11:49:09.386,ns_1@10.1.3.114:<0.769.0>:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,<0.20215.1>,
                                        {dict,3,16,16,8,80,48,
                                         {[],[],[],[],[],[],[],[],[],[],[],[],
                                          [],[],[],[]},
                                         {{[],[],[],[],
                                           [['ns_1@10.1.3.114'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.116'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.118'|0.0]],
                                           [],[],[],[],[],[],[]}}},
                                        ['ns_1@10.1.3.114','ns_1@10.1.3.116'],
                                        [],
                                        ['ns_1@10.1.3.118']}
[ns_server:info,2012-11-12T11:49:19.386,ns_1@10.1.3.114:<0.769.0>:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,<0.20215.1>,
                                        {dict,3,16,16,8,80,48,
                                         {[],[],[],[],[],[],[],[],[],[],[],[],
                                          [],[],[],[]},
                                         {{[],[],[],[],
                                           [['ns_1@10.1.3.114'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.116'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.118'|0.0]],
                                           [],[],[],[],[],[],[]}}},
                                        ['ns_1@10.1.3.114','ns_1@10.1.3.116'],
                                        [],
                                        ['ns_1@10.1.3.118']}
[ns_server:info,2012-11-12T11:49:23.120,ns_1@10.1.3.114:<0.20319.1>:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `default`
[ns_server:info,2012-11-12T11:49:23.121,ns_1@10.1.3.114:<0.20319.1>:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket default with config:
[{database_fragmentation_threshold,{30,undefined}},
{view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2012-11-12T11:49:24.348,ns_1@10.1.3.114:ns_config_rep<0.358.0>:ns_config_rep:do_pull:341]Pulling config from: 'ns_1@10.1.3.116'

[ns_server:info,2012-11-12T11:49:29.386,ns_1@10.1.3.114:<0.769.0>:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,<0.20215.1>,
                                        {dict,3,16,16,8,80,48,
                                         {[],[],[],[],[],[],[],[],[],[],[],[],
                                          [],[],[],[]},
                                         {{[],[],[],[],
                                           [['ns_1@10.1.3.114'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.116'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.118'|0.0]],
                                           [],[],[],[],[],[],[]}}},
                                        ['ns_1@10.1.3.114','ns_1@10.1.3.116'],
                                        [],
                                        ['ns_1@10.1.3.118']}
[ns_server:warn,2012-11-12T11:49:29.546,ns_1@10.1.3.114:capi_set_view_manager-default<0.15508.1>:capi_set_view_manager:handle_info:345]Remote server node {'capi_ddoc_replication_srv-default','ns_1@10.1.3.118'} process down: noconnection
[ns_server:warn,2012-11-12T11:49:29.546,ns_1@10.1.3.114:xdc_rdoc_replication_srv<0.470.0>:xdc_rdoc_replication_srv:handle_info:128]Remote server node {xdc_rdoc_replication_srv,'ns_1@10.1.3.118'} process down: noconnection
[user:warn,2012-11-12T11:49:29.546,ns_1@10.1.3.114:ns_node_disco<0.351.0>:ns_node_disco:handle_info:168]Node 'ns_1@10.1.3.114' saw that node 'ns_1@10.1.3.118' went down.
[error_logger:error,2012-11-12T11:49:29.546,ns_1@10.1.3.114:error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]** Node 'ns_1@10.1.3.118' not responding **
** Removing (timedout) connection **

[ns_server:info,2012-11-12T11:49:30.536,ns_1@10.1.3.114:ns_config_rep<0.358.0>:ns_config_rep:do_pull:341]Pulling config from: 'ns_1@10.1.3.116'

[ns_server:info,2012-11-12T11:49:39.386,ns_1@10.1.3.114:<0.769.0>:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,<0.20215.1>,
                                        {dict,3,16,16,8,80,48,
                                         {[],[],[],[],[],[],[],[],[],[],[],[],
                                          [],[],[],[]},
                                         {{[],[],[],[],
                                           [['ns_1@10.1.3.114'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.116'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.118'|0.0]],
                                           [],[],[],[],[],[],[]}}},
                                        ['ns_1@10.1.3.114','ns_1@10.1.3.116'],
                                        [],
                                        ['ns_1@10.1.3.118']}
[ns_server:info,2012-11-12T11:49:49.386,ns_1@10.1.3.114:<0.769.0>:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,<0.20215.1>,
                                        {dict,3,16,16,8,80,48,
                                         {[],[],[],[],[],[],[],[],[],[],[],[],
                                          [],[],[],[]},
                                         {{[],[],[],[],
                                           [['ns_1@10.1.3.114'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.116'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.118'|0.0]],
                                           [],[],[],[],[],[],[]}}},
                                        ['ns_1@10.1.3.114','ns_1@10.1.3.116'],
                                        [],
                                        ['ns_1@10.1.3.118']}
[ns_server:info,2012-11-12T11:49:53.135,ns_1@10.1.3.114:<0.20445.1>:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `default`
[ns_server:info,2012-11-12T11:49:53.136,ns_1@10.1.3.114:<0.20445.1>:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket default with config:
[{database_fragmentation_threshold,{30,undefined}},
{view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2012-11-12T11:49:59.386,ns_1@10.1.3.114:<0.769.0>:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,<0.20215.1>,
                                        {dict,3,16,16,8,80,48,
                                         {[],[],[],[],[],[],[],[],[],[],[],[],
                                          [],[],[],[]},
                                         {{[],[],[],[],
                                           [['ns_1@10.1.3.114'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.116'|0.0]],
                                           [],
                                           [['ns_1@10.1.3.118'|0.0]],
                                           [],[],[],[],[],[],[]}}},
                                        ['ns_1@10.1.3.114','ns_1@10.1.3.116'],
                                        [],
                                        ['ns_1@10.1.3.118']}
[user:info,2012-11-12T11:50:03.582,ns_1@10.1.3.114:<0.769.0>:ns_orchestrator:handle_info:319]Rebalance exited with reason {not_all_nodes_are_ready_yet,
                                 ['ns_1@10.1.3.114','ns_1@10.1.3.116']}

[MB-7074] xdcr: data compaction doesn't catch up

Dependency — Mon, 13 May 2013 00:53:43 -0500

While it works fine in bidirectional case, there is obvious issue with 2 buckets and double unidirectional replication - even after 5 hours of access phase server didn't manage to compact data.

I believe it may be related to general issues with capacity planning (as MB-6172) but apparently we should not ignore that.

[MB-8258] Complete Metadata ejection when item becomes non-resident

Mon, 13 May 2013 19:19:57 -0500

Memory overhead from an item's metadata would be huge in case of a very large number of items. Moving the metadata to the value blob allows us to evict the metadata with the value together and fetch it from disk on demand.

[MB-8145] 1 read-only user for UI and REST API

Mon, 13 May 2013 19:26:41 -0500

http://www.pivotaltracker.com/story/show/25735995

[MB-8012] Need a purging mechanism for Couchstore to remove deleted / expired items

Dependency — Mon, 13 May 2013 20:58:35 -0500

let's discuss more in person. customers are hitting this already

discussion: http://hub.internal.couchbase.com/confluence/display/cbeng/Deletion+Purging

[MB-7902] [windows] Rebalance exited with reason {detected_nodes_change, {ns_node_disco_events,

Dependency — Tue, 14 May 2013 16:06:44 -0500

The below test is failing while doing rebalance:
./testrunner -i win-aws.ini -t rebalancetests.IncrementalRebalanceInTests.test_load,replica=2,delete-ratio=0.6,expiry-ratio=0.2,do-stop=True

Logs show below error message:

<0.6654.65>:ns_orchestrator:handle_info:319]Rebalance exited with reason {detected_nodes_change,
                                 {ns_node_disco_events,
                                     ['ns_1@10.130.71.207',
                                      'ns_1@10.131.33.89',
                                      'ns_1@10.131.35.139',
                                      'ns_1@10.142.174.97'],

The run log of testrunner is attached. Will be attaching cbcollect_info shortly.

[MB-8299] remote_cluster_info module should return remote memcached access info

Thu, 16 May 2013 16:10:42 -0500

In 2.1, XDCR will provide an option to replicate to remote memcached instead of CAPI.

This requires XDCR core infrastructure to understand how to access memcached at remote cluster.

For example, in order to replicate remote bucket "default", vbucket 123, XDCR need to know

1) node (ip address) for this vbucket at remote cluster;
2) memcached port of the node where the vbucket lives;
3) credentials to access the memcached at that node

Module remote_cluster_info is a natural home for such information.

Today 1) has been already encoded in the vbucketmap maintained by remote_cluster_info, but 2) and 3) are not available. This task will expand remote_cluster_info module to include 2) and 3).

In particular, XDCR need an API to return 1), 2), 3) above, e.g.,

remote_cluster_info:fetch_remote_memcached_info(RemoteClusterId, Bucket, VBucket)

returns
{"10.3.114.2", 11998, "_admin", "_password"}

[MB-8213] Implement batch getMeta and setMeta/delMeta operations in ep_engine

Thu, 16 May 2013 16:19:04 -0500

In 2.1, XDCR will be able to replicate documents to remote memcached directly.

In order to allow batch replication, we need at least two new memcached operations, get_meta_batch, and update_meta_batch.

1. get_meta_batch: by this new command ep_engine should receive a list of keys and return a list of metadata for these keys.

2. update_meta_batch: this new command will take a list of (ops, key, metadata) as input and execute them in the same order as the keys ordered in the list. The ops can be either "deleteWithMeta" or "setWithMeta".

For these two new operations, we need to design new protocol (request and response) for ep_engine.

On the ns_server side, we also need to create memcached API for these two new operations. This will be tracked separately.

The first step is to determine the protocol so xdcr and ep_engine team can work in parallel.

[MB-8300] Implement ns_server side memcached API for new get_meta_batch and update_meta_batch

Thu, 16 May 2013 16:16:42 -0500

In MB-8213, ep_engine team will create two new batch operations, get_meta_batch and update_meta_batch, which will be built on new protocol. From ns_server side, we need to implement new memcached APIs correspondingly.

[MB-8301] XDCR core infrastructure change to enable replicate to remote memcached

Thu, 16 May 2013 16:24:57 -0500

XDCR core infrastructure change to enable replicate to remote memcached

[MB-8221] Updates to Upgrade Process for 2.0.2

Thu, 16 May 2013 17:21:57 -0500

Review session on 2.0.2 upgrade process: see what overhaul required to existing chapter.

[MB-7250] Mac OS X App should be signed by a valid developer key

Thu, 16 May 2013 18:12:06 -0500

Currently launching the Mac OS X version tells you it's from an unidentified developer. You have to right click to launch the app. We can fix this.

[MB-7852] [2.0.2 RN + Doc] Complete productization of Replica Read API

Thu, 16 May 2013 18:10:02 -0500

Replica Read API implementation is ready but testing and related documentation have not completed up to the product-feature ready state.

[MB-7960] Replace XDCR CAPI protocol on the destination with memcached protocol

Thu, 16 May 2013 16:22:22 -0500

Replace XDCR CAPI protocol on the destination with memcached protocol

[MB-8259] Disk not flush on xdcr dest node

Fri, 17 May 2013 13:39:20 -0500

On the ec2 cluster I used for cbrecovery, the disk write queue didn't flush for a very long time. (this symptom was observed twice)

It's an xdcr dest cluster, no other ops rather than xdcr incoming traffic.

Fired as blocker, since it blocks the cbrecovery testing. The clusters are left intact as they are for developer to diagnose, please email back to me when the problem got resolved then I could continue the test. Thanks!

Ronnie

Destination:
http://ec2-184-169-190-197.us-west-1.compute.amazonaws.com:8091

And the source:
http://ec2-23-21-15-103.compute-1.amazonaws.com:8091

User: Administrator
Passwd: password

SSH:
user: root
passwd: couchbase

Now attach the original email thread with diagnosis from Jin and Junyi:

This should be a bug. There are a lot of crash in ns_server/baby_sitter and couchdb/writer process. Couchdb is unable to spawn writer process for some reasons.

I am not 100% sure which caused which but It should be unrelated to XDCR.

Please file a 2.0.2 bug and assign to Filipe to take first look. Thanks.

Junyi

On May 13, 2013, at 10:56 AM, Jin Lim wrote:

Logs from the ec2-184-169-190-197.us-west-1.compute.amazonaws.com. I see following error. After the error I see couch_db/updater kept crashing. I will wait until Junyi's quick looking at it from the xdcr side then figure out what is next.

Thanks,
Jin

[ns_server:info,2013-05-10T18:00:22.665,ns_1@ec2-184-169-190-197.us-west-1.compute.amazonaws.com:ns_config_log<0.562.0>:ns_config_log:handle_info:57]config change: rest_creds -> ********
[user:info,2013-05-10T18:00:22.718,ns_1@ec2-184-169-190-197.us-west-1.compute.amazonaws.com:<0.570.0>:ns_log:crash_consumption_loop:64]Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: WARNING: curl error: transfer closed with outstanding read data remaining from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
EOL on stdin. Exiting
[user:info,2013-05-10T18:00:32.169,ns_1@ec2-184-169-190-197.us-west-1.compute.amazonaws.com:<0.570.0>:ns_log:crash_consumption_loop:64]Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: 2013-05-10 18:00:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2013-05-10 18:00:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)

On May 13, 2013, at 10:41 AM, Ronnie Sun wrote:

Hi Jin, Junyi

On the ec2 cluster I used for cbrecovery, the disk write queue didn't flush for a very long time. (this symptom was observed twice)

It's an xdcr dest cluster, no other ops rather than xdcr incoming traffic.

Everything looks normal, not sure if it's related to the recent reb regression. Would you please take a look at it, thanks!

Ronnie

Destination:
http://ec2-184-169-190-197.us-west-1.compute.amazonaws.com:8091

And the source:
http://ec2-23-21-15-103.compute-1.amazonaws.com:8091

User: Administrator
Passwd: password

SSH:
user: root
passwd: couchbase

[MB-8009] Couchbase Logo update on UI

Fri, 17 May 2013 13:04:18 -0500

Couchbase Logo update on UI

[MB-7972] Sever performance dip during rebalancing observed in demo environment

Fri, 17 May 2013 16:19:28 -0500

Environment:
-memcache-test running through client-side Moxi.
-memcache-test workload: /root/mctest/Users/ingenthr/tmp/memcachetest-centos_x86 -h localhost -i 1000 -t 4 -K test -M 10 -F -S -l

-Moxi config: CLI: /opt/moxi/bin/moxi -u nobody -Z usr=Administrator,pwd=password http://10.197.1.147:8091/pools/default/bucketsStreaming/beer-sample -vv -O /var/log/moxi,
config:
port_listen=11211,
default_bucket_name=default,
downstream_max=1024,
downstream_conn_max=4,
downstream_conn_queue_timeout=200,
downstream_timeout=5000,
wait_queue_timeout=200,
connect_max_errors=5,
connect_retry_interval=30000,
connect_timeout=400,
auth_timeout=100,
cycle=200

-4 nodes of Couchbase Server 2.0.1. c1.medium EC2 instances, beer-sample database bucket
-2 nodes of same config added to cluster and rebalanced

Observation:
-Workload very steady around 8k ops/sec before rebalance
-Workload drops to around 4k ops/sec during rebalance. Both gets and sets affected, compaction not running when load initially drops (kicks in eventually). Workload very choppy during rebalance.
-Workload returns to 8k ops/sec after rebalance

Client memcachetest and moxi logs:
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/client1.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/client2.zip

Cluster logs:
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node1.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node2.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node3.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node4.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node5.zip (aded to first 4)
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node6.zip (added to first 4)

[MB-7735] Memcached crash on a node on the destination cluster

Fri, 17 May 2013 17:17:34 -0500

Live clusters:
Source: http://10.6.2.37:8091/
Destination: http://10.6.2.89:8091/

On Source:
default: ~>70% Resident ratio
saslbucket: ~>40% Resident ratio

System uptime: >36 hours

On Destination:
2 nodes went down: 10.6.2.68, 10.6.2.69

- "ip seems to have changed" seen for both the nodes
- ns_1@10.6.2.68
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_node_disco,nodes_wanted]}}},
{trace,
[{gen_server,call,2},
{ns_orchestrator,needs_rebalance,0},
{ns_cluster_membership,is_balanced,0},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info,2},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] (repeated 15 times)

- ns_1@10.6.2.69
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{{timeout,
{gen_server,call,
[{'stats_reader-default',
'ns_1@10.6.2.69'},
{latest,minute,1}]}},
{gen_server,call,
[menelaus_web_alerts_srv,fetch_alert]}}},
{trace,
[{gen_server,call,2},
{diag_handler,diagnosing_timeouts,1},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info,2},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}]

- Memcached core on 10.6.2.68

The Back trace:

[root@pine-11803 data]# gdb /opt/couchbase/bin/memcached core.memcached.17636
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 17656]
[New Thread 17645]
[New Thread 17654]
[New Thread 17646]
[New Thread 17652]
[New Thread 17653]
[New Thread 17655]
[New Thread 17636]
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x00007fc8afec28a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.1-153.x86_64
(gdb) t a a bt

Thread 8 (Thread 0x7fc8b1376720 (LWP 17636)):
#0 0x00007fc8aff6a43d in write () from /lib64/libc.so.6
#1 0x00007fc8aff01033 in _IO_new_file_write () from /lib64/libc.so.6
#2 0x00007fc8aff00efa in _IO_new_file_xsputn () from /lib64/libc.so.6
#3 0x00007fc8afef692c in fputs () from /lib64/libc.so.6
#4 0x00007fc8aeb61143 in logger_log (severity=EXTENSION_LOG_WARNING, client_cookie=<value optimized out>, fmt=0x7fc8aab1bebb "Schedule cleanup of \"%s\"") at extensions/loggers/file_logger.c:275
#5 0x00007fc8aaad06a4 in TapConnMap::shutdownAllTapConnections (this=0x636e240) at src/tapconnmap.cc:366
#6 0x00007fc8aaa989d1 in EventuallyPersistentEngine::destroy (this=0x6374000, force=<value optimized out>) at src/ep_engine.cc:1401
#7 0x00007fc8aaa98ace in EvpDestroy (handle=<value optimized out>, force=false) at src/ep_engine.cc:130
#8 0x00007fc8adf52bb5 in bucket_shutdown_engine (key=<value optimized out>, nkey=<value optimized out>, val=0x63262a0, nval=<value optimized out>, args=<value optimized out>) at bucket_engine.c:1290
#9 0x00007fc8adf5966c in genhash_iter (h=0x632a000, iterfunc=0x7fc8adf52b80 <bucket_shutdown_engine>, arg=0x0) at genhash.c:275
#10 0x00007fc8adf53f46 in bucket_destroy (handle=0x7fc8ae15c640, force=<value optimized out>) at bucket_engine.c:1327
#11 0x0000000000409777 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7927

Thread 7 (Thread 0x7fc8a8627700 (LWP 17655)):
#0 0x00007fc8b022d7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fc8aaa76f28 in wait (this=0x63a22d0, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x63a22d0, d=...) at src/dispatcher.cc:336
#3 0x00007fc8aaa795ea in Dispatcher::run (this=0x636b880) at src/dispatcher.cc:173
#4 0x00007fc8aaa79eeb in launch_dispatcher_thread (arg=0x636b880) at src/dispatcher.cc:28
#5 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fc8a9a29700 (LWP 17653)):
#0 0x00007fc8b022d7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fc8aaa76f28 in wait (this=0x63a2120, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x63a2120, d=...) at src/dispatcher.cc:336
#3 0x00007fc8aaa795ea in Dispatcher::run (this=0x636ac40) at src/dispatcher.cc:173
#4 0x00007fc8aaa79eeb in launch_dispatcher_thread (arg=0x636ac40) at src/dispatcher.cc:28
#5 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fc8aa638700 (LWP 17652)):
#0 0x00007fc8aff3b97d in nanosleep () from /lib64/libc.so.6
#1 0x00007fc8aff70b34 in usleep () from /lib64/libc.so.6
#2 0x00007fc8aaab67f5 in updateStatsThread (arg=0x1ab44c0) at src/memory_tracker.cc:31
#3 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fc8aeb5d700 (LWP 17646)):
#0 0x00007fc8b022d7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fc8aeb614d6 in logger_thead_main (arg=0x1ab4040) at extensions/loggers/file_logger.c:368
#2 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fc8a9028700 (LWP 17654)):
#0 0x00007fc8b022d7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fc8aaa76f28 in wait (this=0x63a2090, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x63a2090, d=...) at src/dispatcher.cc:336
#3 0x00007fc8aaa795ea in Dispatcher::run (this=0x636aa80) at src/dispatcher.cc:173
#4 0x00007fc8aaa79eeb in launch_dispatcher_thread (arg=0x636aa80) at src/dispatcher.cc:28
#5 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fc8af772700 (LWP 17645)):
#0 0x00007fc8aff6a3dd in read () from /lib64/libc.so.6
#1 0x00007fc8aff01248 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007fc8aff02d4e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007fc8afef753a in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007fc8afef6399 in fgets () from /lib64/libc.so.6
#5 0x00007fc8af773939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#6 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fc8aff776dd in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 1 (Thread 0x7fc8a7c26700 (LWP 17656)):
#0 0x00007fc8afec28a5 in raise () from /lib64/libc.so.6
#1 0x00007fc8afec4085 in abort () from /lib64/libc.so.6
#2 0x0000000000404315 in release_cookie (cookie=<value optimized out>) at daemon/memcached.c:6707
#3 0x00007fc8adf55009 in bucket_engine_release_cookie (cookie=0x62c3b80) at bucket_engine.c:2565
#4 0x00007fc8aaa9461a in EventuallyPersistentEngine::releaseCookie (this=0x6374000, cookie=0x62c3b80) at src/ep_engine.cc:1230
#5 0x00007fc8aaabf376 in TapConnection::releaseReference (this=0x638a000, force=<value optimized out>) at src/tapconnection.cc:110
#6 0x00007fc8aaad2aeb in TapConnectionReaperCallback::callback(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /opt/couchbase/lib/memcached/ep.so
#7 0x00007fc8aaa795ea in Dispatcher::run (this=0x636b6c0) at src/dispatcher.cc:173
#8 0x00007fc8aaa79eeb in launch_dispatcher_thread (arg=0x636b6c0) at src/dispatcher.cc:28
#9 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

[MB-8238] NO menu bar to access menu commands after installation and click the Couchbase icon on MAC

Fri, 17 May 2013 18:03:39 -0500

1. Intall Couchbase server on MAC Lion OS
2. Click Couchbase Icon no access menu poped out

For previous build like 755, we have this access menu. The screen shot is from 755

[MB-8266] erlang crashed with error: "no match of right hand side value {error, closed}...

Fri, 17 May 2013 19:39:43 -0500

Environment:
  4 windows vm with 4 core, 4GB RAM each

Couchbase server version 2.0.2-798

Run sanity test with rebalance is part of the test.
See erlang crash in log
Check to see if there is an erlang crash dump in cluster => no core dump.

Error in diags

=========================CRASH REPORT=========================
  crasher:
    initial call: erlang:apply/2
    pid: <0.13944.1>
    registered_name: []
    exception error: no match of right hand side value {error,closed}
      in function mc_binary:quick_stats_recv/3
      in call from mc_binary:quick_stats_loop/5
      in call from mc_binary:quick_stats/5
      in call from ns_memcached:do_handle_call/3
      in call from ns_memcached:worker_loop/3
    ancestors: ['ns_memcached-default','single_bucket_sup-default',
                  <0.13913.1>]
    messages: []
    links: [<0.13929.1>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 17711
    stack_size: 24
    reductions: 8661
  neighbours:

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-798-rel.setup.exe.manifest.xml

[MB-7149] cbbackup loops infinitely

Sat, 18 May 2013 05:59:48 -0500

I'm trying to backup my cluster via cbbackup. I start the backup via
/opt/couchbase/bin/cbbackup couchbase://Administrator:password@mymachine:8091 /backups/couchbase_backup_test
The backup appears to work fine and a progress bar appears, but then exceeds 100% progress and never stops! Example:
^Cinterrupted.###############################] 210.8% (26141950/12403784 msgs)
(I ctrl-C'd to kill it after 200% because it seems that this can't possibly work. Note that both the percent and the number of messages are off). I only have ~12 million items in the bucket, but it went right past that limit when backing up.
Help?

[MB-8309] online_upgrade_swap_rebalance 1.8.1->2.0.2: Rebalance exited with reason {{change_filter_failed,{'EXIT',{{unexpected_reason,killed}

Fri, 17 May 2013 17:30:13 -0500

http://qa.hq.northscale.net/job/centos-32-2.0-upgrade-P1/4/consoleFull

./testrunner -i /tmp/centos-32-2.0-upgrade.ini upgrade_version=2.0.2-805-rel,get-cbcollect-info=True,GROUP=P1 -t newupgradetests.MultiNodesUpgradeTests.online_upgrade_swap_rebalance,initial_version=1.8.1-942-rel,standard_buckets=1,items=500000,max_verify=1000,GROUP=1_8;ONLINE;WINDOWS;P1

steps:
1)10.3.3.151, 10.3.3.152 with 1.8.1 build
10.3.3.153 with 2.0.2-805-rel

2)[2013-05-17 01:11:42,198] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.151&user=Administrator&knownNodes=ns_1%4010.3.3.152%2Cns_1%4010.3.3.151%2Cns_1%4010.3.3.153
...
[2013-05-17 01:26:15,908] - [task:340] INFO - rebalancing was completed with progress: 100.0% in 873.699863911 sec

3)[2013-05-17 01:26:37,009] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.152&user=Administrator&knownNodes=ns_1%4010.3.3.152%2Cns_1%4010.3.3.153
...

[2013-05-17 01:31:07,549] - [rest_client:1014] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2013-05-17 01:31:07,550] - [rest_client:1015] INFO - Latest logs from UI:
[2013-05-17 01:31:07,601] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 2, u'text': u"Rebalance exited with reason {{change_filter_failed,\n {'EXIT',\n {{unexpected_reason,killed},\n [{misc,executing_on_new_process,1},\n {ns_vbm_sup,local_change_vbucket_filter,4},\n {rpc,local_call,3},\n {ns_vbm_sup,change_vbucket_filter,4},\n {ns_vbm_sup,'-set_replicas/3-fun-2-',5},\n {lists,foldl,3},\n {ns_vbm_sup,set_replicas,3},\n {ns_vbm_sup,\n '-set_replicas_on_nodes/3-fun-1-',3}]}}},\n [{ns_vbm_sup,change_vbucket_filter,4},\n {ns_vbm_sup,'-set_replicas/3-fun-2-',5},\n {lists,foldl,3},\n {ns_vbm_sup,set_replicas,3},\n {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',\n 3},\n {lists,foreach,2},\n {janitor_agent,\n do_bulk_set_vbucket_state_old_style,4},\n {ns_vbucket_mover,\n update_replication_post_move,3}]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368779444497.0, u'type': u'info'}
[2013-05-17 01:31:07,602] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 43 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779418787.0, u'type': u'critical'}
[2013-05-17 01:31:07,603] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779358861.0, u'type': u'critical'}
[2013-05-17 01:31:07,603] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 179 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779358787.0, u'type': u'critical'}
[2013-05-17 01:31:07,604] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779299138.0, u'type': u'critical'}
[2013-05-17 01:31:07,604] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 154 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779298786.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779238798.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 132 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779238787.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779184396.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Bucket "standard_bucket0" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368779184144.0, u'type': u'info'}

[MB-8214] bgfetcher performance regression

Sun, 19 May 2013 00:10:40 -0500

* bgfetcher condition variable for woken task is not getting correctly (hasWokenTask).
* also need to immediately move any woken task to readyQueue instead of re-pushing to futureQueue

[MB-7943] Memcached drops tap connections without warning

Tue, 23 Apr 2013 14:14:51 -0500

Servers M1 and M2 are each running couchbase (C1 and C2 respectively, both added to the same cluster via the web interface) and an application (A1 and A2 respectively). A1 is configured to primarily access C2 and to fail over to C1 if C2 becomes unavailable. A2 is similarly configured, only accessing C1 primarily and failing over to C2.

When accessing A1, Couchbase reports that the value is stored on C2 and has replicated to C1, howerver while accessing A2, Couchbase reports that the value is stored on C1 and has not replicated to C2. Additionally, when attempting to manually rebalance the cluster, the rebalance fails (the log output is the attached image). After consulting the IRC channel, user ingenthr suggested I file this as an issue.

I've also attached the output of cbcollect_info in the hopes that it might contain something useful. If you need more information, do not hesitate to ask.

[MB-7897] [Done- RN 2.0.2] Modify a bucket resets unspecified values to the default value

Tue, 23 Apr 2013 16:01:38 -0500

All values omitted in a modify request for a bucket is reset to their default value. Most users won't find it natural to first go get the bucket properties before toggling their value before setting it (if they want to lets say just enable flush for a bucket).

Eg:

As a user I would expect to be able to do something like this (by using the php api):

$cb = new CouchbaseClusterManager("192.168.0.97", "Administrator", "asdasd");
$cb->modifyBucket("default", array("enable flush" => 1));

Instead we force the user to grab the properties first, and then send the entire set back with their modification.

In theory we could do this "under the cover" in the API's, but that will add complexity in the clients (if they're running in an async environment etc). Sending the full config back and forth is just unneeded overhead.

[MB-7725] disk channels (threads) are configurable, weighable for different scenarios / workloads

Fri, 15 Feb 2013 14:57:38 -0600

Multiple Reader Writer (MRW) design document:
http://hub.internal.couchbase.com/confluence/display/cbeng/Multiple+Readers+and+Writers+Per+Bucket?focusedCommentId=6784990&#comment-6784990

MRW development plan:
http://hub.internal.couchbase.com/confluence/display/cbeng/Multiple+Reader+and+Writer+Development+Plan

[MB-7691] couchstore fsyncs and file per vbucket create very fragmented files (was: couchstore compactor is very poor on reading data from src couch file)

Tue, 23 Apr 2013 13:22:30 -0500

(Assigned to Damien for initial prioritization)

I have 120 megs file that takes about 15 seconds to compact on cold page cache. On warm page cache it's less than 3 seconds.

When I modified dbdump to read all data (not just btree nodes, but also doc bodies) I've found it needs about 3 seconds to process entire file on cold page cache. And if I ran compactor afterwards I get warm page cache compactor timing. Which confirms that modified dbdump warms all pages. And it clearly does that about 4 times more efficiently.

So I conclude that even without advanced prefetch our data is natually linear and simply loading it depth first (which is what happens during compaction) is fast enough. But there's something bad happening in couchstore compactor that either triggers wrong behavior of kernel's prefetch logic or maybe it does some clearly excessive and non-linearly aligned work.

I'm afraid I have to move to other topic and have to stop spending time on this. So leaving this to you folks.

I started this investigation when I saw 'mere' 90 gigs of data requiring 4 hours to compact in perf runs. That's a bit too slow IMHO.

I'll attach my dbdump patches.

[MB-7690] couchstore compactor interleaves byseq btree nodes and doc bodies

Tue, 5 Feb 2013 17:58:34 -0600

(Assigning on Damien for initial prioritization)

Looks like my old finding is still valid. Number of page cache pages required to keep byseq index in ram is _actually increasing_ after compactor.

That's because it saves by seq leaf node after it's data items.

I have tools and data that confirm my finding (real vbucket files from perf run).

When I told about this problem to Aaron prior to 2.0 he said it's going to be easy on afaik indexer branch. Looks like this branch is merged at last and it's time to reconsider.

[MB-8022] Fsync optimizations (remove double fsyncs)

Tue, 23 Apr 2013 13:24:43 -0500

Fsync optimizations (remove double fsyncs)

[MB-7667] System Test : Higher ep_commit_time, beam size on source nodes during xdcr.

Fri, 22 Mar 2013 12:57:59 -0500

Seeing unusual stats for

( during xdcr_start phase)
1.ep_commit_time
2. rsize_beam : > 2.8G on some points
3. Uneven compaction growth
4. Uneven CPU across the cluster.

The source node cluster stats are here

https://github.com/couchbaselabs/couchbase-qe-docs/tree/master/system-tests/plum-cluster/01_30

[MB-7661] View query on Rebalance.Out fails with Reason: A view spec can not consist of merges exclusively.

Dependency — Mon, 8 Apr 2013 20:07:55 -0500

I've been running SDKQE stester runs for the Java SDK, specifically testing the following scenario:

/stester -C 127.0.0.1:8050 -i 20devcluster.ini -c rebalance.Once --vdsw_dvname ddoc/vquery --rbcount 2 --dsw_timeres 1 --hdsw_mc_threads 10 --workload dsw.Hybrid --mode out --hdsw_http_threads 5 --hdsw_cb_threads 10 --rebound 90 -d -o rebalance-once.log

My cluster is a 4 node cluster, and this scenario rebalances 2 nodes out of the cluster. During the rebalance, view queries are issued.

Now it happens that during this rebalance (I assume this happens at the end of the rebalance) I get the following Exception:

java.lang.RuntimeException: Failed to access the view
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:838)
at com.couchbase.sdkd.cbclient.ViewQueryCommandContext.execIter(ViewQueryCommandContext.java:252)
at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311)
at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135)
at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156)
at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212)
Caused by: java.util.concurrent.ExecutionException: OperationException: SERVER: error Reason: A view spec can not consist of merges exclusively.
at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:89)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:73)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:63)
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:834)
... 5 more
Caused by: OperationException: SERVER: error Reason: A view spec can not consist of merges exclusively.
at com.couchbase.client.protocol.views.NoDocsOperationImpl.parseError(NoDocsOperationImpl.java:106)
at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:68)
at com.couchbase.client.ViewNode$MyHttpRequestExecutionHandler.handleResponse(ViewNode.java:199)
at org.apache.http.nio.protocol.AsyncNHttpClientHandler.processResponse(AsyncNHttpClientHandler.java:417)
at org.apache.http.nio.protocol.AsyncNHttpClientHandler.inputReady(AsyncNHttpClientHandler.java:242)
at com.couchbase.client.http.AsyncConnectionManager$ManagedClientHandler.inputReady(AsyncConnectionManager.java:244)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:172)
at org.apache.http.impl.nio.DefaultClientIOEventDispatch.inputReady(DefaultClientIOEventDispatch.java:155)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:161)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:335)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:275)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:542)
at java.lang.Thread.run(Thread.java:680)
Feb 1, 2013 1:04:30 PM com.couchbase.sdkd.cbclient.CommandResult warnAbout
WARNING: Unknown exception encountered (for operation) future warnings will be suppressed
java.lang.RuntimeException: Failed to access the view
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:838)
at com.couchbase.sdkd.cbclient.ViewQueryCommandContext.execIter(ViewQueryCommandContext.java:252)
at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311)
at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135)
at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156)
at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212)
Caused by: java.util.concurrent.ExecutionException: OperationException: SERVER: no_active_vbuckets Reason: Cannot execute view query since the node has no active vbuckets
at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:89)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:73)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:63)
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:834)
... 5 more
Caused by: OperationException: SERVER: no_active_vbuckets Reason: Cannot execute view query since the node has no active vbuckets
at com.couchbase.client.protocol.views.NoDocsOperationImpl.parseError(NoDocsOperationImpl.java:106)
at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:68)
at com.couchbase.client.ViewNode$MyHttpRequestExecutionHandler.handleResponse(ViewNode.java:199)
at org.apache.http.nio.protocol.AsyncNHttpClientHandler.processResponse(AsyncNHttpClientHandler.java:417)
at org.apache.http.nio.protocol.AsyncNHttpClientHandler.inputReady(AsyncNHttpClientHandler.java:242)
at com.couchbase.client.http.AsyncConnectionManager$ManagedClientHandler.inputReady(AsyncConnectionManager.java:244)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:172)
at org.apache.http.impl.nio.DefaultClientIOEventDispatch.inputReady(DefaultClientIOEventDispatch.java:155)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:161)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:335)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:275)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:542)

Note that the Java stuff is not of particular interest for this ticket, but the Error messages are. First, I get

error Reason: A view spec can not consist of merges exclusively.
and then
no_active_vbuckets Reason: Cannot execute view query since the node has no active vbuckets

The way the Java client currently implements is that it will remove the ViewNode during rebalance when it vanishes from the couchNodes list. I'll attach a rebalance log from one of the nodes that is getting rebalanced out, but as expected this happens at the very last step, its still in the list when it has no vbuckets anymore.

Should the server not be able to handle view requests, even when it has no vbuckets attached?

Thanks,
Michael

[MB-7626] [RN 2.0.2] Couchbase continually crashes and restart when Couch bucket is created in OpenVZ environment

Duplicate — Fri, 19 Apr 2013 13:17:00 -0500

I have done some QA to help narrow down an issue I have discovered.

1. Setup Couchbase from 64bit Ubunut 12 package.
2. Next select a CouchDB bucket.
3. Next complete the setup.

Expected: Couchbase server and bucket functioning.

Actual: The Couchbase server appears to just keep crashing / i.e. restarting every few seconds.

If I remove the couchDB bucket no crashes / restart occur.
If I create a memcached bucket only the server is functioning fine and the bucket functions as expected.

This is on a OpenVZ VPS container running Ubuntu 12.04 with a CentOS 6 OS as the host on 2.6.32 kernel.

I went ahead and setup couchbase on another Ubuntu 12.04 dedicated server (exact replica of the above server without the virtualization) and no issues.

What debug output information from the logs would you like to help identify this issue?

[MB-7432] XDCR Stats enhancements

Mon, 25 Mar 2013 13:34:10 -0500

After seeing XDCR in action, would like to propose a few enhancements:

-Put certain statistics in the XDCR screen as well as on the graph page:
    -Percentage complete/caught up. While backfilling replication this would describe the number of items already sent to the remote side out of the total in the bucket. Once running, it would show whether there is a significant amount of backup in the queue
    -Items per second to see speed of each stream and in total
    -Bandwidth in use. As per a customer, the most important thing with XDCR is going to be the possibly cross-country internet bandwidth and will need to monitor that for each replication stream and in total

-On the graph page of outgoing, I would recommend removing "mutations checked", "mutations replicated", "data replication", "active vb reps", "waiting vb reps", "secs in replicating", "secs in checkpointing", "checkpoints issued" and "checkpoints failed". These stats really aren't useful from the perspective of someone trying to monitor or troubleshoot the current state of their cluster.
-On the graph page of outbound, there's a bit of confusion over the difference between "mutations to replicate", "mutations in queue" and "queue size". Unless they are showing significantly (and usefully) different metrics, recommend to remove all but one
-On the graph page of incoming, recommend to put "total ops/sec" on the far left to line up with the "ops/sec" in the summary section
-"XDCR dest ops per sec" is confusing because this cluster is the "destination" yet the stat implies the other way around. Recommend "Incoming XDCR ops per sec"
-"XDCR docs to replicate" is a little confusing because it doesn't match the same stat in the "outbound". Recommend to change "mutations to replicate" to "XDCR docs to replicate"
-Would also be good to see outbound ops/sec in the summary section alongside the number remaining to replicate

[MB-7286] UI logs showing multiple errors, (XDCR bi and uni, at the start of a rebalance-in operation): Server error during processing: ["web request failed" ...

Mon, 18 Feb 2013 11:31:48 -0600

+ 5 nodes rebalance in on each cluster
Cluster setup: c1:c2::10:10
biXDCR_bucket: c1 <---> c2
uniXDCR_src: c1 ---> c2 :uniXDCR_dest
Front end loads on c1 and c2 for biXDCR_bucket, and on c1 for uniXDCR_src.
c1: http://ec2-177-71-230-72.sa-east-1.compute.amazonaws.com:8091/
c2: http://ec2-175-41-186-167.ap-southeast-1.compute.amazonaws.com:8091/

UI reports whole bunch of these errors at the start of rebalancing operation and with the front end loads:
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_doctor,get_tasks_version]}}},
{trace,
[{gen_server,call,2},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info_wait,5},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] (repeated 1 times)

Will attach grabbed diags from the particular server in a bit.

[MB-7287] XDCR: replication lag on ec2 periodically reaches 1 minute and more

Mon, 8 Apr 2013 05:51:33 -0500

While mutation rate is about 6K ops/sec replication queue reaches 400K items from time to time (see page 37, east coast (source) is worse for some reason).

I doesn't seem to be an issue, but I never saw requirements for delays. So wondering.

This is ec2 specific issue (taking into account laws of physics).

[MB-7230] [system test] warmup access log corrupt during warmup in windows

Wed, 10 Apr 2013 15:06:59 -0500

Environment:
8 windows server 2008 R2 64 bit in ec2. Each server has 7.5 GB RAM and 45GB EBS disk to store data.
Create a 6 nodes windows cluster in ec2

Loading phase:
* Create one default bucket and load 24 million items to it. Each item has size from 128 to 512 bytes.

Access phase:
* Mutate items with ratio creates/updates/gets/deletes/expirations = 10/60/20/5/5

* Doing rebalance add node, remove node, failover, failover and add back.
* The last test was reboot all 6 nodes in reboot warmup test. During warmup, I see error when collect raw warmup stats
ep_warmup_access_log: corrupt

root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.96.107.176:11210 raw warmup
ep_warmup: enabled
ep_warmup_dups: 0
ep_warmup_estimate_time: 20704
ep_warmup_estimated_key_count: 14538479
ep_warmup_estimated_value_count: 14538479
ep_warmup_key_count: 14538479
ep_warmup_keys_time: 745292809
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 941489106
ep_warmup_value_count: 2285132
root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.110.222.56:11210 raw warmup
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 22194
ep_warmup_estimated_key_count: 23969171
ep_warmup_estimated_value_count: 23969171
ep_warmup_key_count: 23969171
ep_warmup_keys_time: 2466454551
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 2676578146
ep_warmup_value_count: 976671
root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.214.194.95:11210 raw warmup
ep_warmup: enabled
ep_warmup_dups: 0
ep_warmup_estimate_time: 22131
ep_warmup_estimated_key_count: 14540213
ep_warmup_estimated_value_count: 14540213
ep_warmup_key_count: 14540213
ep_warmup_keys_time: 662356019
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 932353521
ep_warmup_value_count: 2283472
root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.46.181.58:11210 raw warmup
Stats 'warmup' are not available from the requested engine.
root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.110.218.124:11210 raw warmup
Stats 'warmup' are not available from the requested engine.
root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.111.66.215:11210 raw warmup
ep_warmup: enabled
ep_warmup_dups: 0
ep_warmup_estimate_time: 21788
ep_warmup_estimated_key_count: 14679821
ep_warmup_estimated_value_count: 14679821
ep_warmup_key_count: 14679821
ep_warmup_keys_time: 638006011
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 785227909
ep_warmup_value_count: 2264710
root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.2.63.249:11210 raw warmup
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 26696
ep_warmup_estimated_key_count: 24113568
ep_warmup_estimated_value_count: 24113568
ep_warmup_key_count: 24113568
ep_warmup_keys_time: 1970642317
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 2221105121
ep_warmup_value_count: 956974
root@ip-10-116-233-143:~/testrunner# /opt/couchbase/bin/cbstats 10.110.206.19:11210 raw warmup
ep_warmup: enabled
ep_warmup_dups: 0
ep_warmup_estimate_time: 20255
ep_warmup_estimated_key_count: 24110614
ep_warmup_estimated_value_count: unknown
ep_warmup_key_count: 21603099
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: loading keys
ep_warmup_thread: running
ep_warmup_value_count: 0

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1952-rel.setup.exe.manifest.xml

[MB-7091] race condition while loading sample data via setup wizard

Thu, 11 Apr 2013 13:49:57 -0500

/Users/farshid/Downloads/ns-diag-20121104183310.txt:38769:[ns_server:debug,2012-11-04T18:32:32.847,ns_1@127.0.0.1:<0.1038.0>:samples_loader_tasks:wait_for_exit:101]output from gamesim-sample: "{'username': None, 'node': '127.0.0.1:8091', 'password': None, 'bucket': 'gamesim-sample', 'ram_quota': 100} ['/Users/farshid/Downloads/couchbase-server-community_x86_64_2.0.0-1939-rel/Couchbase Server.app/Contents/Resources/couchbase-core/bin/../samples/gamesim-sample.zip']\nTraceback (most recent call last):\n File \"/Users/farshid/Downloads/couchbase-server-community_x86_64_2.0.0-1939-rel/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 237, in <module>\n main()\n File \"/Users/farshid/Downloads/couchbase-server-community_x86_64_2.0.0-1939-rel/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 226, in main\n docloader.init_bucket()\n File \"/Users/farshid/Downloads/couchbase-server-community_x86_64_2.0.0-1939-rel/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 121, in init_bucket\n "
/Users/farshid/Downloads/ns-diag-20121104183310.txt:38770:[ns_server:debug,2012-11-04T18:32:32.848,ns_1@127.0.0.1:<0.1038.0>:samples_loader_tasks:wait_for_exit:101]output from gamesim-sample: "authType='sasl')\n File \"/Users/farshid/Downloads/couchbase-server-community_x86_64_2.0.0-1939-rel/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/couchbase/rest_client.py\", line 799, in create_bucket\n"
/Users/farshid/Downloads/ns-diag-20121104183310.txt:38776:[ns_server:debug,2012-11-04T18:32:32.892,ns_1@127.0.0.1:<0.1038.0>:samples_loader_tasks:wait_for_exit:101]output from gamesim-sample: " 'proxyPort': self.get_nodes_self().moxi,\n File \"/Users/farshid/Downloads/couchbase-server-community_x86_64_2.0.0-1939-rel/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/couchbase/rest_client.py\", line 591, in get_nodes_self\n"
/Users/farshid/Downloads/ns-diag-20121104183310.txt:38777:[ns_server:debug,2012-11-04T18:32:32.893,ns_1@127.0.0.1:<0.1038.0>:samples_loader_tasks:wait_for_exit:101]output from gamesim-sample: " json_parsed = json.loads(content)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py\", line 326, in loads\n"
/Users/farshid/Downloads/ns-diag-20121104183310.txt:38778:[ns_server:debug,2012-11-04T18:32:32.893,ns_1@127.0.0.1:<0.1038.0>:samples_loader_tasks:wait_for_exit:101]output from gamesim-sample: " return _default_decoder.decode(s)\n"
/Users/farshid/Downloads/ns-diag-20121104183310.txt:38779:[ns_server:debug,2012-11-04T18:32:32.893,ns_1@127.0.0.1:<0.1038.0>:samples_loader_tasks:wait_for_exit:101]output from gamesim-sample: " File \"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py\", line 366, in decode\n"
/Users/farshid/Downloads/ns-diag-20121104183310.txt:38780:[ns_server:debug,2012-11-04T18:32:32.894,ns_1@127.0.0.1:<0.1038.0>:samples_loader_tasks:wait_for_exit:101]output from gamesim-sample: " obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File \"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py\", line 384, in raw_decode\n"
/Users/farshid/Downloads/ns-diag-20121104183310.txt:38781:[ns_server:debug,2012-11-04T18:32:32.894,ns_1@127.0.0.1:<0.1038.0>:samples_loader_tasks:wait_for_exit:101]output from gamesim-sample: " raise ValueError(\"No JSON object could be decoded\")\nValueError: No JSON object could be decoded\n"
/Users/farshid/Downloads/ns-diag-20121104183310.txt:38782:[ns_server:debug,2012-11-04T18:32:32.904,ns_1@127.0.0.1:samples_loader_tasks<0.1025.0>:samples_loader_tasks:handle_info:58]Consumed exit signal from samples loading task gamesim-sample: {'EXIT',
/Users/farshid/Downloads/ns-diag-20121104183310.txt:38786:[user:error,2012-11-04T18:32:32.905,ns_1@127.0.0.1:samples_loader_tasks<0.1025.0>:samples_loader_tasks:handle_info:64]Loading sample bucket gamesim-sample failed: {failed_to_load_samples_with_status,
/Users/farshid/Downloads/ns-diag-20121104183310.txt:38788:[ns_server:debug,2012-11-04T18:32:32.906,ns_1@127.0.0.1:samples_loader_tasks<0.1025.0>:samples_loader_tasks:handle_info:68]Token holder died

[MB-7177] lack of fsyncs in view engine may lead to silent index corruption

Fri, 4 Jan 2013 14:21:29 -0600

SUBJ. Found out about this in discussion with Filipe about how views work.

If I understood correctly it doesn't fsync at all silently assuming that if there's valid header, then preceding data is valid as well. Which is clearly not true.

IMHO that's a massive blocker that needs to be fixed sooner rather than later.

[MB-6746] separate disk path for replica index (and individual design doc) disk path

Thu, 25 Oct 2012 14:44:50 -0500

On the UI and REST API to create a separate disk path for replica indexes (right under the replica index check box in the setup wizard)

This will allow users to have better disk solution is replica index is used.

In addition, add new rest APIs to enable a separate disk path for each design document (Not in the UI, only in REST)

[MB-6737] Complete and integrate file I/O NIF to improve couchdb-side latency and throughput

Mon, 8 Apr 2013 18:59:35 -0500

Complete and integrate file I/O NIF to improve couchdb-side latency and throughput

[MB-6635] Crash in memcached's release_cookie

Tue, 30 Oct 2012 10:23:55 -0500

(I was having some crashes regularly for months already, but figured it wasn't worth filing as QEs most likely found some of them as well. Pre-beta things are quite different)

Just found core dump. Doing nothing. 16 cluster_run nodes. And 5 empty buckets.

Here's what backtrace shows me:

Thread 26 (Thread 0xeccf9b70 (LWP 25048)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xeccf9b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 25 (Thread 0xeb4f6b70 (LWP 25164)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xeb4f6b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 24 (Thread 0xefcffb70 (LWP 24864)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xefcffb70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 23 (Thread 0xe8cf1b70 (LWP 25289)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xe8cf1b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 22 (Thread 0xf36fab70 (LWP 24868)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e3687f in EventuallyPersistentEngine::notifyPendingConnections() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e369a2 in EvpNotifyPendingConns () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf7644c39 in start_thread (arg=0xf36fab70) at pthread_create.c:304
#5 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 21 (Thread 0xe9cf3b70 (LWP 25167)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xe9cf3b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 20 (Thread 0xe84f0b70 (LWP 25290)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xe84f0b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 19 (Thread 0xf1500b70 (LWP 24950)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xf1500b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 18 (Thread 0xed4fab70 (LWP 25047)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xed4fab70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 17 (Thread 0xebcf7b70 (LWP 25050)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e3687f in EventuallyPersistentEngine::notifyPendingConnections() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e369a2 in EvpNotifyPendingConns () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf7644c39 in start_thread (arg=0xebcf7b70) at pthread_create.c:304
#5 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 16 (Thread 0xee4fcb70 (LWP 24954)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e3687f in EventuallyPersistentEngine::notifyPendingConnections() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e369a2 in EvpNotifyPendingConns () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf7644c39 in start_thread (arg=0xee4fcb70) at pthread_create.c:304
#5 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 15 (Thread 0xf2502b70 (LWP 24866)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xf2502b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 14 (Thread 0xe94f2b70 (LWP 25168)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e3687f in EventuallyPersistentEngine::notifyPendingConnections() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e369a2 in EvpNotifyPendingConns () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf7644c39 in start_thread (arg=0xe94f2b70) at pthread_create.c:304
#5 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 13 (Thread 0xe7cefb70 (LWP 25291)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xe7cefb70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 12 (Thread 0xeacf5b70 (LWP 25165)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xeacf5b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 11 (Thread 0xedcfbb70 (LWP 25046)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xedcfbb70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 10 (Thread 0xea4f4b70 (LWP 25166)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xea4f4b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 9 (Thread 0xeecfdb70 (LWP 24953)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xeecfdb70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 8 (Thread 0xf74e06c0 (LWP 23970)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf75a2fbb in write () at ../sysdeps/unix/syscall-template.S:82
#2 0xf754bff4 in _IO_new_file_write (f=0xf763b580, data=0xff82fad0, n=90) at fileops.c:1276
#3 0xf754bc7f in new_do_write (fp=0xf763b580, data=0xff82fad0 "Wed Sep 12 23:21:10.772212 3: Schedule cleanup of \"eq_tapq:replication_n_14@10.17.22.233\"\n", to_do=90) at fileops.c:530
#4 0xf754bf46 in _IO_new_file_xsputn (f=0xf763b580, data=0xff82fad0, n=90) at fileops.c:1370
#5 0xf75415b0 in *__GI__IO_fputs (str=str@entry=0xff82fad0 "Wed Sep 12 23:21:10.772212 3: Schedule cleanup of \"eq_tapq:replication_n_14@10.17.22.233\"\n", fp=0xf763b580) at iofputs.c:42
#6 0xf76f5521 in logger_log (severity=EXTENSION_LOG_WARNING, client_cookie=0x0, fmt=0xf2eca190 "Schedule cleanup of \"%s\"") at extensions/loggers/file_logger.c:268
#7 0xf2e7388e in TapConnMap::shutdownAllTapConnections() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#8 0xf2e3a1be in EvpDestroy () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#9 0xf609d5fa in bucket_shutdown_engine (key=0xf0000de8, nkey=4, val=0xf0000b78, nval=0, args=0x0) at bucket_engine.c:1290
#10 0xf60a30b9 in genhash_iter (h=0xb9d5d80, iterfunc=iterfunc@entry=0xf609d5a0 <bucket_shutdown_engine>, arg=arg@entry=0x0) at genhash.c:275
#11 0xf60a1f42 in bucket_destroy (handle=0xf60a6960, force=<optimized out>) at bucket_engine.c:1327
#12 bucket_destroy (handle=0xf60a6960, force=false) at bucket_engine.c:1307
#13 0x0804be30 in main (argc=21, argv=0xff831b34) at daemon/memcached.c:7923

Thread 7 (Thread 0xec4f8b70 (LWP 25049)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xec4f8b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 6 (Thread 0xf1d01b70 (LWP 24867)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xf1d01b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 5 (Thread 0xf2d03b70 (LWP 24865)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xf2d03b70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 4 (Thread 0xef4feb70 (LWP 24952)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xef4feb70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 3 (Thread 0xf0cffb70 (LWP 24951)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf2e16e8c in IdleTask::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0xf7644c39 in start_thread (arg=0xf0cffb70) at pthread_create.c:304
#6 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 2 (Thread 0xf68dcb70 (LWP 23985)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf7649703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236
#2 0xf76f5864 in logger_thead_main (arg=0x8ef4e40) at extensions/loggers/file_logger.c:361
#3 0xf7644c39 in start_thread (arg=0xf68dcb70) at pthread_create.c:304
#4 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 1 (Thread 0xe74eeb70 (LWP 25292)):
#0 0xf7700430 in __kernel_vsyscall ()
#1 0xf750c941 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0xf750fd72 in *__GI_abort () at abort.c:92
#3 0x0804da3c in release_cookie (cookie=0xb806b68) at daemon/memcached.c:6693
#4 0xf60a0252 in bucket_engine_release_cookie (cookie=0xb806b68) at bucket_engine.c:2565
#5 0xf2e36234 in EventuallyPersistentEngine::releaseCookie(void const*) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#6 0xf2e64e6b in TapConnection::releaseReference(bool) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#7 0xf2e77adb in TapConnectionReaperCallback::callback(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#8 0xf2e1b9f2 in Task::run(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#9 0xf2e1a104 in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#10 0xf2e1b91a in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#11 0xf7644c39 in start_thread (arg=0xe74eeb70) at pthread_create.c:304
#12 0xf75b223e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

and extra stuff:

(gdb) p thr
$1 = (LIBEVENT_THREAD *) 0xb9ec624
(gdb) p *thr
$2 = {thread_id = 4100967280, base = 0xb9ed828, notify_event = {ev_active_next = {tqe_next = 0x0, tqe_prev = 0xb9edb38}, ev_next = {tqe_next = 0xb9ca8d0, tqe_prev = 0xb9ed8fc}, ev_timeout_pos = {ev_next_with_common_timeout = {tqe_next = 0xffffffff, tqe_prev = 0x0}, min_heap_idx = -1}, ev_fd = 24, ev_base = 0xb9ed828, _ev = {ev_io = {ev_io_next = {tqe_next = 0x0, tqe_prev = 0xb9edbd0}, ev_timeout = {tv_sec = 0, tv_usec = 0}}, ev_signal = {ev_signal_next = {tqe_next = 0x0, tqe_prev = 0xb9edbd0}, ev_ncalls = 0, ev_pncalls = 0x0}}, ev_events = 18, ev_res = 2, ev_flags = 128, ev_pri = 0 '\000', ev_closure = 2 '\002', ev_timeout = {tv_sec = 0, tv_usec = 0}, ev_callback = 0x805b280 <thread_libevent_process>, ev_arg = 0xb9ec624}, notify = {24, 25}, new_conn_queue = 0xb9edbe0, suffix_cache = 0xb9edc38, mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __kind = 376, __nusers = 80, {__spins = 61, __list = {__next = 0x3d}}}, __size = '\000' <repeats 12 times>, "x\001\000\000P\000\000\000=\000\000", __align = 0}, is_locked = 61, pending_io = 0x1, index = 1885431122, type = 543649385, last_checked = 544235892}
(gdb) l
6688 conn *c = (conn *)cookie;
6689 assert(c);
6690
6691 LIBEVENT_THREAD *thr = c->thread;
6692 assert(thr);
6693 LOCK_THREAD(thr);
6694 --c->refcount;
6695
6696 /* Releasing the refererence to the object may cause it to change
6697 * state. (NOTE: the release call shall never be called from the
(gdb) p *c
$3 = {sfd = 254, nevents = 0, sasl_conn = 0xf0205fa8, state = 0x805aa40 <conn_ship_log>, substate = bin_reading_packet, registered_in_libevent = true, event = {ev_active_next = {tqe_next = 0xb7fdf20, tqe_prev = 0xb9edb38}, ev_next = {tqe_next = 0xb7fdf20, tqe_prev = 0xb9ed8fc}, ev_timeout_pos = {ev_next_with_common_timeout = {tqe_next = 0xffffffff, tqe_prev = 0x0}, min_heap_idx = -1}, ev_fd = 254, ev_base = 0xb9ed828, _ev = {ev_io = {ev_io_next = {tqe_next = 0x0, tqe_prev = 0xf0205f28}, ev_timeout = {tv_sec = 0, tv_usec = 0}}, ev_signal = {ev_signal_next = {tqe_next = 0x0, tqe_prev = 0xf0205f28}, ev_ncalls = 0, ev_pncalls = 0x0}}, ev_events = 18, ev_res = 0, ev_flags = 128, ev_pri = 0 '\000', ev_closure = 2 '\002', ev_timeout = {tv_sec = 0, tv_usec = 0}, ev_callback = 0x804f700 <event_handler>, ev_arg = 0xb806b68}, ev_flags = 18, which = 4, rbuf = 0xb806d48 "\201F", rcurr = 0xb806df0 "", rsize = 2048, rbytes = 0, wbuf = 0xb807550 "\200\n", wcurr = 0xb807550 "\200\n", wsize = 2048, wbytes = 1992, write_and_go = 0x805aa40 <conn_ship_log>, write_and_free = 0x0, ritem = 0xb806df0 "", rlbytes = 0, item = 0x0, store_op = 0, sbytes = 0, iov = 0xb8080d8, iovsize = 400, iovused = 0, msglist = 0xb808d60, msgsize = 10, msgused = 1, msgcurr = 0, msgbytes = 0, ilist = 0xb807d58, isize = 200, icurr = 0xb807d58, ileft = 0, suffixlist = 0xb808080, suffixsize = 20, suffixcurr = 0xb808080, suffixleft = 0, protocol = binary_prot, transport = tcp_transport, request_id = 0, request_addr = {ss_family = 0, __ss_align = 0, __ss_padding = '\000' <repeats 119 times>}, request_addr_size = 0, hdrbuf = 0x0, hdrsize = 0, noreply = false, refcount = 2 '\002', dynamic_buffer = {buffer = 0x0, size = 0, offset = 0}, engine_storage = 0xf0205f38, ascii_cmd = 0x0, binary_header = {request = {magic = 129 '\201', opcode = 70 'F', keylen = 0, extlen = 0 '\000', datatype = 0 '\000', vbucket = 0, bodylen = 0, opaque = 654311424, cas = 0}, bytes = "\201F", '\000' <repeats 13 times>, "'\000\000\000\000\000\000\000"}, cas = 0, cmd = 70, opaque = 654311424, keylen = 0, list_state = 0, next = 0x0, thread = 0xb9ec624, aiostat = ENGINE_SUCCESS, ewouldblock = true, tap_iterator = 0xf609ea80 <bucket_tap_iterator_shim>, parent_port = 11999}

[MB-6527] Tools to Index and compact database/indexes when the server is offline

Mon, 8 Oct 2012 13:57:13 -0500

This is from the supportability point of view.
If for whatever reason customer bring their nodes down, eg. maintenance etc.

When they bring the node back up, hopefully we have all the compaction/indexing finished for that particular node.

We need a way to index and compact data (database and index) if possible when the nodes are offline.

[MB-6450] Finalize doc editing API and implementation

Wed, 3 Oct 2012 15:31:02 -0500

We need to:

* avoid loading and displaying blobs on UI

* avoid seeing deleted docs

* handle warmup, node being down in REST API implementation and UI

[MB-6203] delete bucket waiting is raceful

Tue, 16 Oct 2012 19:00:39 -0500

SUBJ.

First we delete bucket in config and _then_ we wait for bucket events until we see bucket is deleted on all nodes. If bucket event(s) fire before we subscribe we'll timeout.

I think new implementation should use active polling for pre-2.0 compat mode and new janitor_agent call for 2.0+ mode.

[MB-6172] XDCR: High CPU utilization on destination nodes

Wed, 7 Nov 2012 18:24:14 -0600

Regardless cluster configuration CPU utilization on destination nodes is too high during initial replication.
Source cluster looks pretty idle at the same time.

See for details:
https://www.yammer.com/couchbase.com/#/Threads/show?threadId=200784413
https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdElKVEhla2NqTFJWdkpkcHFybGgtR3c#gid=4

Build 1550.

[MB-5622] Crash of master node may lead to autofailover in 2 minutes instead of configured shorter autofailover period

Wed, 3 Oct 2012 16:20:57 -0500

SUBJ. That's remaining reason why autofailover may still require 2 minutes even with quick failover code. Fixing it however is not quite trivial.

[MB-5552] Get unit tests running on windows

Tue, 26 Feb 2013 15:26:28 -0600

Get unit tests running on windows

[MB-5430] public api to delete index files

Tue, 16 Oct 2012 18:47:23 -0500

In the case that index becomes corrupted, or contains duplicate, inconsistent, or invalid results, it would be great to delete them so that couch can rebuild.
Motivation: ~~MB-5429~~

We need this functionality for a better implementation of flush. However , not a blocker.

[MB-4975] behavior when running out of disk space should be TMPOOM

Wed, 10 Apr 2013 15:25:27 -0500

(updated by alk: I cannot fix top posting but I took out some names out from this)

Set up a node, fill the filesystem, watch processes run but see memcached take connections and just fail to respond.

Also, set up a node, stop Couchbase. Fill the filesystem. Start Couchbase.

On Wed, Mar 28, 2012 at 5:07 PM, Sharon Barr <XXXX> wrote:

Unix is more mature then Couchbase at the edge cases. we are getting there.. or trying NOT to get there at all (another alternative..).

From: Matt Ingenthron
Sent: Wednesday, March 28, 2012 8:04 AM
To: Frank Weigel; Perry Krug; Dipti Borkar
Cc: Sharon Barr; Alex Ma; support-internal

Subject: Re: YYYY having issues

Incidentally, while testing the hotfix for AAAA with TMP_OOM, I accidentally ran my CentOS out of disk. The OS is running happily and so are our processes, but moxi is just returning errors and the memcached process isn't responding to stats requests.

There is still free memory available, but happily we've (kinda) lived within our quota. Confusingly the quota is set to 512MByte, but the resident memory size of memcached is only 445MByte. The virtual size is larger, but it's likely not tried to allocate.

So at least this UNIX-like OS is fine when out of disk.

Matt

On 3/27/12 9:55 PM, "Frank Weigel" <XXXXXX> wrote:

In principal agree, but if this is the only disk, UNIX doesn't do well when entirely out of disk AFAIK, so we may need to do this when poor man's disk alert kicks in?

That's a myth. Only buggy UNIXes (or UNIX-like OSs) don't do well there. I've worked with many a UNIX that is perfectly fine with a full disk.*

I agree with Perry that it should end in TMP_OOM. We should leave ourselves some memory of course (since we need to receive the packet to respond with TMP_OOM), but there is no reason why this is not doable. It's simply a matter of writing and testing the software.

Matt

* the myth came from BSD that way, way, way back when required 2x the swap possible per process's memory to keep going. that "2x" is another myth that seems to keep perpetuating.

From: Perry Krug <XXX>
Date: Tue, 27 Mar 2012 02:27:01 -0700
To: Frank Weigel <XXX>
Cc: (skipped)
Subject: Re: YYYYY having issues

Can we please actually do something about this in the code so that the entire server doesn't just crash? We should start sending tmp_oom or something as soon as we detect that we are unable to write to disk.

From: Sharon Barr <xxX>
Date: Mon, 26 Mar 2012 17:11:58 -0700
To: Alex Ma <XXX>, Perry Krug <xXX>
Cc: skipped
Subject: RE: YYYYY having issues

Apparently they run out of disk space on all nodes..

[MB-4840] hotfix release should reflect the change(version#) on the web console

Wed, 17 Apr 2013 18:04:43 -0500

hotfix release should reflect the change(version#) on the web console

[MB-4785] Meaningful alert when low-level packet corruption on node

Thu, 20 Sep 2012 02:17:07 -0500

Logs showed that some low-level corruption in network data was apparent. Symptom is that nodes are going up and down. Not clear in the UI that this is happening only on 2 nodes. Not clear in UI that it's low-level corruption. Not clear that these nodes are consistently having a problem, and need to be failed over. No info bubbles up about why the node flaps up and down, or how to report this up to data center or Amazon (in this case on EC2).

Need a clear alert to user, suggesting to fail over a troublesome node. Ideal to have concrete examples of the corrupt data to pass on to data center ops.

[MB-4345] ns_server should use unique memcached admin/password for each cluster instance

Mon, 4 Mar 2013 02:41:50 -0600

steps to reproduce :

1- create a cluster with 3 nodes
2- shut down membase on one of the nodes
3- create a new cluster with 3 nodes and reuse the public IP of one of the nodes
4- now you will see that ns_server from the old cluster now is trying to reset the vbuckets on the new cluster

ns_server should use the otpCookie authentication when running _set_vbucket_state command against memcached (alk: see my comment below)

[MB-4268] We should hash our REST API passwords

Mon, 4 Mar 2013 02:40:03 -0600

We should hash our REST API passwords

[MB-4167] moxi returns "key not found" on ascii-gets during rebalance_in and _out

Dependency — Thu, 8 Nov 2012 19:03:30 -0600

steps to reproduce :

1- insert 10k items
2- run a script which calls mc.get() on those existing items
3- add a node and rebalance
4- after 5-10 minutes , remove a node and rebalance

2 get misses during rebalance_in
2 get misses during rebalance_out

in my testing rebalance took 5 minutes and i got 2 get misses during rebalance ( 2 mins after rebalance started )

[MB-3726] Certain stats cannot be aggregated for all nodes

Dependency — Thu, 19 Jul 2012 16:17:12 -0500

I.e. it's not possible to show e.g. cpu or swap usage across all nodes.

[MB-3971] windows upgrade fails if machine's ip have changed from the original IP address

Thu, 14 Mar 2013 12:30:37 -0500

the workaround is cleanup the old mnesia database under c:\program files\membase
\server\var\lib\membase\mnesia and try again

ref : https://groups.google.com/forum/#!topic/membase/wqoq9EJ6Z5A

[MB-3256] Raise a warning message about time on local machines does not match when adding a node.

Tue, 16 Oct 2012 18:51:31 -0500

When adding a node, we may raise a warning if local time on a node and node/cluster does not match.

[MB-2899] configurable lifespans in moxi should be 64-bit and parsed as unsigned (safe_strtoul)

Thu, 28 Feb 2013 12:34:49 -0600

configurable lifespans in moxi should be 64-bit and parsed as unsigned (safe_strtoul)

[MB-1905] open service after loading x items during warmup( starting a server with many items can potentially take hours to fully load hashtable and values)

Duplicate — Thu, 5 Jul 2012 20:02:55 -0500

Fast startup time for EP engine

We need ep engine to start within 1 minute regardless the amount of data stored on the node.

in addition, we need to load the hot keys (and values) to prevent cold startup.
This means that for some test scenarios, i.e. loading the server with random set of keys, there won't be "hot" keys to load upon startup.

The behavior of the startup time should be documented on the membase wiki for ep engine.

[MB-1875] Our log contains information about the target system we shouldn't reveal

Tue, 12 Jun 2012 18:59:22 -0500

The log contains a list of _all_ of the filesystems I've got on my system, and we certainly don't need information about all of my users home directories, all my installed applications etc.

Instead of listing all of the filesystems we should generate an entry if the filesystem we're configured to use for data storage doesn't contain enough free space.

We might ask the user to send us the logs, and they shouldn't have to edit the logs before sending them to us. This sounds to me like the good old Microsoft reports that contained list of all installed software on the machine....

[MB-7737] Swap usage shoots up with view querying or indexing

Wed, 27 Feb 2013 16:00:39 -0600

- Setup a 7 node cluster,
- RAM (for each physical machine): ~32GB
- create 2 buckets: default (bucket quota: 12000MB per node) and saslbucket (bucket quota: 7000MB per node)
- run a load (with just creates), until the item count hits a mark
- create 2 views over each of the buckets
- Run load that pushes system into DGM, ~80% resident ratio for default, and ~60% resident ratio for saslbucket
- Run queries continuously against the views as the load continues.
(for e.g.:
curl -X GET 'http://10.1.3.235:8092/default/_design/d1/_view/v1?startkey=0&stale=ok&limit=1000')
- Noticed the increased swap usage.
- Start a new mixed load on both the buckets (creates, updates, deletes, expires).
- Setup XDCR to another 5 node cluster.
- Continue querying, noticed considerable increase in swap usage.

[MB-7514] Monitoring and Best Practices Out of Date/Incorrect

Thu, 2 May 2013 16:39:58 -0500

Intake from Perry on Server Guide:

-We also severely need to update our best practices and monitoring guides. These pages are very out of date (and in some cases, patently incorrect):

http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-bestpractice-ongoing.html

http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-bestpractice-ongoing-ui.html

http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-monitoring.html

[MB-1143] Allow for "grouping" servers for replication affinity (rack awareness)

Duplicate — Fri, 3 May 2013 02:33:00 -0500

Customer wants to configure replication so that copies of data are kept in separate racks within a data center to mitigate power failures and/or Amazon EC2 (or other cloud) "zones"

Will need UI/management control as well as underlying ns_server improvements.

[MB-7965] bucket-flush takes over 8 seconds to complete on an empty bucket

Wed, 8 May 2013 09:43:19 -0500

Simply running bucket-flush from the CLI takes over 8 seconds to return when run against a bucket with no items in it.

[MB-8211] making lots of streaming connections in general, or lots of streaming connections in a short period of time causes unexpected resource usage and/or crashing owing to OOM

Wed, 8 May 2013 12:51:10 -0500

Going into version 2.0, we'd integrated changes that make handling of many streaming connections much less expensive in terms of memory, but there is some evidence that creating a lot of them at the same time or creating a large number of streaming connections could cause instability and crashing in ns_server.

Note that in the case of PHP deployments, we've found the number of worker processes to be 256, 512, or 5000 processes in some deployments. That is per server.

Then, deployments can be 50 or more systems.

Thus, we could have 12,000+ or 25,000+ or 250,000 processes which may all try to connect to a cluster.

This issue is to track possible solutions to this kind of thing occurring.

Possible solutions could be to drop connections or to only allow a number of connections tested to work in our minimum system configuration.

Note that at the client library side we have a method of sharing the configuration between multiple processes. However, it is not currently a complete solution and may never be because:
- It is not on by default as it requires the configuration of a file path.
- In some cases, during a rebalance, we can still have a rush of processes trying to get a configuration.
- We do not currently have a good solution for updating the configuration quickly if it's a memcached type bucket, as there is no NOT_MY_VBUCKET response.

Even if we can solve all of those issues at the client library side, a bug or an very large number of client systems can still cause problems, so we will need ns_server to be reliable even in the face of lots of requests.

[MB-8207] moxi does not allow a noop before authentication on binary protocol

Wed, 8 May 2013 15:16:05 -0500

We had added a feature to spymemcached based on a Couchbase user's request to detect hung processes. This tries to complete a noop before doing auth.

It's fine against memcached/ep-engine in all cases, and it appears to be fine for ascii (where there is no authentication and it fails back to the version command), but moxi does not seem to allow auth after the noop. This may be because it's expecting the first command to wire it up to a downstream for the "gateway" moxi?

We're going to search for a workaround, but I wanted to make sure this issue was known.

See also:
https://code.google.com/p/spymemcached/issues/detail?id=272&thanks=272&ts=1364702110
and
https://github.com/mumoshu/play2-memcached/issues/17

[MB-4030] After two nodes crashed, curr_items remained 0 after warmup for extended period of time

Sun, 17 Mar 2013 23:05:03 -0500

we had two nodes crash at Tribal Crossing, possibly related to a disk space issue, but I don't think so.

After they crashed, the nodes warmed up relatively quickly, but immediately "discarded" their items. I say that because I see that they warmed up ~10m items, but the current item counts were both 0.

I tried shutting down the service and had to kill memcached manually (kill -9). Restarting it went through the same process of warming up and then nothing.

While I was looking around, I left it sit for a little while and magically all of the items came back. I seem to recall this bug previously where a node wouldn't be told to be active until all the nodes in the cluster were active...and it got into trouble when not all of the nodes restarted.

Diags for all nodes will be attached

[MB-7930] Find a way to avoid creating too many erlang crash dumps

Fri, 10 May 2013 13:49:39 -0500

We have some new code that ensures unique names of erlang crash dumps. That's useful for preserving multiple different crashes. But it's also dangerous w.r.t. filling disk with those crash dumps. It'll be more dangerous with babysitting VM change.

[MB-8008] [system test]rebalance failure due to bad replicators after rebalance

Mon, 13 May 2013 14:27:43 -0500

Install couchbase server 2.0.1-185 on 9 windows ssd vms.
Create a cluster with 7 nodes
10.3.121.173
10.3.121.169
10.3.121.171
10.3.3.214
10.3.121.47
10.3.3.180
10.3.3.181

Create 2 buckets:
  "buckets" : {"default" : {"quota": "3500 MB", "replicas": "1", "replica_index": "enable"},
                     "sasl": {"count": "1", "quota": "2000MB", "replicas": "1", "replica_index": "disable"}}

Create 2 design docs:
  "ddocs" : {"create": [{"ddoc":"ddoc1", "view":"view1", "map":"function(doc,meta){emit(doc.city,null);}", "bucket":"default"},
                                          {"ddoc":"ddoc1", "view":"view2", "map":"function(doc,meta){emit(doc.city, [doc.st, doc.email]);}", "bucket":"default"},
                                          {"ddoc":"ddoc2", "view":"view1", "map":"function(doc,meta){emit(doc.city,null);}", "bucket":"saslbucket"},
                                          {"ddoc":"ddoc2", "view":"view2", "map":"function(doc,meta){emit(doc.city, [doc.st, doc.email]);}", "bucket":"saslbucket"}]}

no xdcr created

Load 20+ million items to both bucket until resident ratio on both bucket around 70%
Access cluster in 3 hours with spec in this page and run query with 200~500 ops per second on both buckets
5% expire - 5% delete - 5% add - 5% update , 80% gets - 3 hours access phase for default
70% expire - 20% delete - 5% add - 5% gets - 3 hours access phase for saslbucket

Then the test go through a "swap rebalance the orchestrator node" phase and a "rebalance in one node" phase
In "rebalance out one node" phase, rebalance failed with error:

Started rebalancing bucket saslbucket ns_rebalancer000 ns_1@10.3.121.171 03:42:54 - Sat Mar 30, 2013
Starting rebalance, KeepNodes = ['ns_1@10.3.3.181','ns_1@10.3.121.243',
'ns_1@10.3.121.47','ns_1@10.3.3.182',
'ns_1@10.3.3.214','ns_1@10.3.121.171',
'ns_1@10.3.121.169'], EjectNodes = ['ns_1@10.3.121.173']
ns_orchestrator004 ns_1@10.3.121.171 03:42:54 - Sat Mar 30, 2013

Bad replicators after rebalance:
Missing = [{'ns_1@10.3.121.171','ns_1@10.3.3.181',173},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',174},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',175},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',176},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',177},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',178},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',246},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',247},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',248},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',249},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',250},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',251},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',252},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',253},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',254},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',255},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',256},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',257},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',258},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',259},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',260},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',261},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',262},
{'ns_1@10.3.121.171','ns_1@10.3.3.181',263}]
Extras = [] ns_rebalancer002 ns_1@10.3.121.171 06:18:36 - Sat Mar 30, 2013
Rebalance exited with reason bad_replicas ns_orchestrator002 ns_1@10.3.121.171 06:18:36 - Sat Mar 30, 2013
Shutting down bucket "saslbucket" on 'ns_1@10.3.121.173' for deletion ns_memcached002 ns_1@10.3.121.173 06:18:37 - Sat Mar 30, 2013

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-community_x86_64_2.0.1-185-rel.setup.exe.manifest.xml

Diags link https://s3.amazonaws.com/bugdb/jira/9ssd-win2.0.1-185-ns-diag-20130330.txt.zip

[MB-8040] 2.0 needs to support use couchbase for all the REST endpoints. (no membase)

Mon, 13 May 2013 18:42:26 -0500

"Part of membase -> couchbase transition is renaming of bucket type.

But because we also need to support mixed clusters for rebalance upgrade we'll have to handle both membase and couchbase bucket types internally. We won't upgrade membase -> couchbase during upgrade. But rather new buckets will be created with type couchbase.

Heres relevant parts from emails from 1.8 rebranding times:

From: Matt Ingenthron <matt@couchbase.com>
Date: Wed, 30 Nov 2011 14:41:44 -0800
To: Dipti Borkar <dipti@couchbase.com>, Aliaksey <alkondratenko@gmail.com>
Cc: Benjamin Young <benjamin@couchbase.com>, mgmt_dev <mgmt_dev@couchbase.com>, Frank Weigel <frank@couchbase.com>, Perry Krug <perry@couchbase.com>, Farshid Ghods <farshid@couchbase.com>, sdk_dev <sdk_dev@couchbase.com>, Steve Yen <steve@couchbase.com>, MC Brown <mc@couchbase.com>

Subject: Re: Renaming buckettype from membase to couchbase in 1.8.0

Alk brought up one other case which hasn't been considered though. What about mixed clusters?

I think the answer is that it *should* say ""couchbase"" from all nodes if possible. If that's too hard, it's okay for some to say ""membase"" and some to say ""couchbase"", but the key thing is that after rebalancing out all 1.8 nodes, it should say ""couchbase"".

Dipti: if you agree, please let Alk know.

(alk: this is in-line reply by Dipti)
Yes, agreed. ns_server needs to be able to handle both types to handle upgrades.

Thanks,

Matt

p.s.: I hope we're starting testing of upgrade from 1.8 to 2.0 early, so we can wring out any problems to avoid forcing people to take an incremental patch of 1.8 just so they can get to 2.0.

On 11/30/11 2:36 PM, ""Dipti Borkar"" <dipti@couchbase.com> wrote:

Frank, Matt and I talked about it today and agreed on the following.

For 1.8.0
UI reflects ""Couchbase"" as the bucket type â€" on the Manage data buckets page and on the Create bucket page
REST API
Post takes in both ""membase"" or ""couchbase"" as bucket type but internally, it is still stored as ""membase"" (most people will continue to use membase until they are forced to change it to 2.0
Get sends back ""membase"" as buckettype
Documentation section on deprecation will list buckettype ""membase"" as deprecated in 1.8.0
For 2.0
REST API
Post takes in both ""membase"" or ""couchbase"" as bucket type and is stored as ""couchbase""
Get sends back ""couchbase"" as buckettype
Documentation release notes will mention that buckettype ""membase"" was deprecated and ""couchbase"" is the new buckettype name

http://www.pivotaltracker.com/story/show/24607551

[MB-8015] Avoid the full rematerizaliztion when the cluster is restarted.

Dependency — Mon, 13 May 2013 18:56:29 -0500

When the whole cluster is restarted, we sometimes observe that the replication starts from the beginning, which would be very expensive. To avoid this, we should be able to identify the list of items that weren't replicated upon restarting the cluster, and replicate them only when the cluster is ready.

[MB-7216] Allow XDCR configuration at bucket replication level, not just the cluster level

Mon, 13 May 2013 19:00:16 -0500

There are scenarios where you may want to tune the XDCR parameters differently on a per replicator basis. Specifically, if you're using normal XDCR and integration with ElasticSearch you may want to adjust the checkpoint intervals separately.

Currently the XDCR parameters are cluster wide. Allow users to set these on a per XDCR basis.