Couchbase

[MB-8348] Eviction of doc bodies on compaction should be configurable.

Thu, 23 May 2013 20:13:40 -0500

I've added a command line flag to the compactor, http://review.couchbase.org/#/c/26504/

The compactor is called in couch_db_updater.erl https://github.com/couchbase/couchdb/blob/789b84c1e7ff4f870cf88b5bdad194f1ead619b3/src/couchdb/couch_db_updater.erl#L885

Need a configuration option added to ns_server that I can use to decide whether to pass the flag.

[MB-8347] [2.0.2 RN] XDCR REST API endpoint is misspelled as "XCDR"

Thu, 23 May 2013 21:10:38 -0500

Hi Dipit,

The Couchbase manual should show the correct name of endpoints, but in kernel code, the endpoint is misspelled as XCDR in a few places, I am afraid this will

1) make users unable to use the correct endpoints if they follow the manual

2) if more and more users use incorrect endpoints, it would be more expensive for us to fix it later.

A patch has been uploaded to gerrit, please see review comments there.

http://review.couchbase.org/#/c/26472

Leave the decision to product management staff.

[MB-8346] memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 255

Thu, 23 May 2013 17:19:07 -0500

Run test with build 2.0.2-807

./testrunner -i /tmp/6-win-xdcr.ini get-cbcollect-info=True -t xdcr.biXDCR.bidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=bidirection,doc-ops=create-update-delete,doc-ops-dest=create,update,failover=source

Test failed and I saw a lot memcached crashed in log page

Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 255. Restarting. Messages: Mon May 20 17:02:36.236179 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 83 was completed succecssfully.
Mon May 20 17:02:36.236179 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 84 was completed succecssfully.
Mon May 20 17:02:36.241062 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 85 was completed succecssfully.
Mon May 20 17:02:36.242039 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 86 was completed succecssfully.
Mon May 20 17:02:36.243992 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 87 was completed succecssfully.
Mon May 20 17:02:36.243992 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 88 was completed succecssfully.
Mon May 20 17:02:36.246921 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 89 was completed succecssfully.
Mon May 20 17:02:36.259617 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 90 was completed succecssfully.
Mon May 20 17:02:36.259617 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 91 was completed succecssfully.
Mon May 20 17:02:36.259617 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 92 was completed succecssfully.
Mon May 20 17:02:36.265476 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 93 was completed succecssfully.
Mon May 20 17:02:36.265476 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 94 was completed succecssfully.
Mon May 20 17:02:36.265476 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 95 was completed succecssfully.
Mon May 20 17:02:36.266453 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 96 was completed succecssfully.
Mon May 20 17:02:36.266453 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 97 was completed succecssfully.
Mon May 20 17:02:36.268406 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 98 was completed succecssfully.
Mon May 20 17:02:36.275242 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 99 was completed succecssfully.
Mon May 20 17:02:36.275242 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 100 was completed succecssfully.
Mon May 20 17:02:36.276218 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 101 was completed succecssfully.
Mon May 20 17:02:36.276218 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 102 was completed succecssfully.
Mon May 20 17:02:36.276218 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 103 was completed succecssfully.
Mon May 20 17:02:36.277195 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 104 was completed succecssfully.
Mon May 20 17:02:36.277195 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 105 was completed succecssfully.
Mon May 20 17:02:36.278171 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 106 was completed succecssfully.
Mon May 20 17:02:36.278171 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 107 was completed succecssfully.
Mon May 20 17:02:36.279148 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 108 was completed succecssfully.
Mon May 20 17:02:36.283054 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 109 was completed succecssfully.
Mon May 20 17:02:36.285007 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 110 was completed succecssfully.
Mon May 20 17:02:36.286960 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 111 was completed succecssfully.
Mon May 20 17:02:36.286960 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 112 was completed succecssfully.
Mon May 20 17:02:36.296726 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 113 was completed succecssfully.
Mon May 20 17:02:36.296726 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 114 was completed succecssfully.
Mon May 20 17:02:36.301609 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 115 was completed succecssfully.
Mon May 20 17:02:36.317234 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 116 was completed succecssfully.
Mon May 20 17:02:36.317234 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 117 was completed succecssfully.
Mon May 20 17:02:36.326023 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 118 was completed succecssfully.
Mon May 20 17:02:36.326023 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 119 was completed succecssfully.
Mon May 20 17:02:36.326023 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 120 was completed succecssfully.
Mon May 20 17:02:36.326023 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 121 was completed succecssfully.
Mon May 20 17:02:36.326023 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 122 was completed succecssfully.
Mon May 20 17:02:36.327000 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 123 was completed succecssfully.
Mon May 20 17:02:36.327000 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 124 was completed succecssfully.
Mon May 20 17:02:36.328953 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 125 was completed succecssfully.
Mon May 20 17:02:36.328953 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 126 was completed succecssfully.
Mon May 20 17:02:36.329929 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 127 was completed succecssfully.
Mon May 20 17:02:36.329929 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 128 was completed succecssfully.
Mon May 20 17:02:36.329929 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 129 was completed succecssfully.
Mon May 20 17:02:36.332859 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 130 was completed succecssfully.
Mon May 20 17:02:36.333835 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 131 was completed succecssfully.
Mon May 20 17:02:36.333835 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 132 was completed succecssfully.
Mon May 20 17:02:36.333835 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 133 was completed succecssfully.
Mon May 20 17:02:36.333835 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 134 was completed succecssfully.
Mon May 20 17:02:36.334812 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 135 was completed succecssfully.
Mon May 20 17:02:36.340671 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 136 was completed succecssfully.
Mon May 20 17:02:36.340671 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 137 was completed succecssfully.
Mon May 20 17:02:36.343601 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 138 was completed succecssfully.
Mon May 20 17:02:36.362156 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 139 was completed succecssfully.
Mon May 20 17:02:36.366062 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 140 was completed succecssfully.
Mon May 20 17:02:36.366062 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 141 was completed succecssfully.
Mon May 20 17:02:36.366062 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 142 was completed succecssfully.
Mon May 20 17:02:36.369968 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 143 was completed succecssfully.
Mon May 20 17:02:36.369968 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 144 was completed succecssfully.
Mon May 20 17:02:36.369968 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 145 was completed succecssfully.
Mon May 20 17:02:36.371921 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 146 was completed succecssfully.
Mon May 20 17:02:36.371921 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 147 was completed succecssfully.
Mon May 20 17:02:36.372898 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 148 was completed succecssfully.
Mon May 20 17:02:36.373875 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 149 was completed succecssfully.
Mon May 20 17:02:36.373875 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 150 was completed succecssfully.
Mon May 20 17:02:36.392429 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 151 was completed succecssfully.
Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 152 was completed succecssfully.
Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 153 was completed succecssfully.
Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 154 was completed succecssfully.
Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 155 was completed succecssfully.
Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 156 was completed succecssfully.
Mon May 20 17:02:36.396335 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 157 was completed succecssfully.
Mon May 20 17:02:36.398289 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 158 was completed succecssfully.
Mon May 20 17:02:36.398289 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 159 was completed succecssfully.
Mon May 20 17:02:36.399265 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 160 was completed succecssfully.
Mon May 20 17:02:36.399265 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 161 was completed succecssfully.
Mon May 20 17:02:36.403171 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 162 was completed succecssfully.
Mon May 20 17:02:36.403171 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 163 was completed succecssfully.
Mon May 20 17:02:36.406101 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 164 was completed succecssfully.
Mon May 20 17:02:36.407078 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 165 was completed succecssfully.
Mon May 20 17:02:36.407078 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 166 was completed succecssfully.
Mon May 20 17:02:36.409031 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 167 was completed succecssfully.
Mon May 20 17:02:36.412937 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 168 was completed succecssfully.
Mon May 20 17:02:36.412937 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 169 was completed succecssfully.
Mon May 20 17:02:36.425632 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_30 - Reset vbucket 170 was completed succecssfully.
Mon May 20 17:05:23.420750 Pacific Daylight Time 3: (default) TAP (Consumer) eq_tapq:anon_29 - disconnected
Mon May 20 17:05:23.825046 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:anon_29"
Mon May 20 17:05:23.987156 Pacific Daylight Time 3: (default) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.125 - disconnected, keep alive for 300 seconds
EOL on stdin. Initiating shutdown
Mon May 20 17:05:53.190281 Pacific Daylight Time 3: (default) Shutting down tap connections!
Mon May 20 17:05:53.191257 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.125"
Mon May 20 17:05:53.205906 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.123"
Mon May 20 17:05:53.205906 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:anon_30"
Mon May 20 17:05:54.208835 Pacific Daylight Time 3: (default) Had to wait 999 ms for shutdown
Mon May 20 17:05:55.209812 Pacific Daylight Time 3: (default) Had to wait 1000 ms for shutdown
Mon May 20 17:05:56.210789 Pacific Daylight Time 3: (default) Had to wait 1000 ms for shutdown
Mon May 20 17:05:57.212742 Pacific Daylight Time 3: (default) Had to wait 1001 ms for shutdown ns_log000 ns_1@10.3.2.124 17:06:25 - Mon May 20, 2013
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.3.2.124 17:06:09 - Mon May 20, 2013
Node 'ns_1@10.3.2.124' saw that node 'ns_1@10.3.2.123' went down. Details: [{nodedown_reason,
connection_closed}] ns_node_disco005 ns_1@10.3.2.124 17:06:01 - Mon May 20, 2013

Link to manifest file of this build
http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-807-rel.setup.exe.manifest.xml

This bug may relate to bug ~~MB-8296~~

[MB-8345] nodes down due to memcached crash at destination cluster

Thu, 23 May 2013 17:18:56 -0500

Environment: 6 windows 2008 r2 64 bit
Run test
./testrunner -i /tmp/6-win-xdcr.ini get-cbcollect-info=True -t xdcr.biXDCR.bidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=bidirection,doc-ops=create-update-delete,doc-ops-dest=create,update,failover=source

Test failed with error could not reach node
https://friendpaste.com/4gUHpCTiJLdRDYYzp0hRUd

Check diag around the failed time, I saw erlang crash in one node

Mon May 20 17:05:53.190281 Pacific Daylight Time 3: (default) Shutting down tap connections!
Mon May 20 17:05:53.191257 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.125"
Mon May 20 17:05:53.205906 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.123"
Mon May 20 17:05:53.205906 Pacific Daylight Time 3: (default) Schedule cleanup of "eq_tapq:anon_30"
Mon May 20 17:05:54.208835 Pacific Daylight Time 3: (default) Had to wait 999 ms for shutdown
Mon May 20 17:05:55.209812 Pacific Daylight Time 3: (default) Had to wait 1000 ms for shutdown
Mon May 20 17:05:56.210789 Pacific Daylight Time 3: (default) Had to wait 1000 ms for shutdown
Mon May 20 17:05:57.212742 Pacific Daylight Time 3: (default) Had to wait 1001 ms for shutdown
[user:info,2013-05-20T17:06:26.069,ns_1@10.3.2.124:ns_memcached-default<0.31624.6>:ns_memcached:terminate:735]Control connection to memcached on 'ns_1@10.3.2.124' disconnected: {{badmatch,
                                                                     {error,
                                                                      closed}},
                                                                    [{mc_client_binary,
                                                                      cmd_binary_vocal_recv,
                                                                      5},
                                                                     {mc_client_binary,
                                                                      select_bucket,
                                                                      2},
                                                                     {ns_memcached,
                                                                      ensure_bucket,
                                                                      2},
                                                                     {ns_memcached,
                                                                      handle_info,
                                                                      2},
                                                                     {gen_server,
                                                                      handle_msg,
                                                                      5},
                                                                     {ns_memcached,
                                                                      init,1},
                                                                     {gen_server,
                                                                      init_it,
                                                                      6},
                                                                     {proc_lib,
                                                                      init_p_do_apply,
                                                                      3}]}
[error_logger:error,2013-05-20T17:06:26.069,ns_1@10.3.2.124:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
  crasher:
    initial call: erlang:apply/2
    pid: <0.31640.6>
    registered_name: []
    exception error: no match of right hand side value {error,closed}
      in function mc_binary:quick_stats_recv/3
      in call from mc_binary:quick_stats_loop/5
      in call from mc_binary:quick_stats/5
      in call from ns_memcached:do_handle_call/3
      in call from ns_memcached:worker_loop/3
    ancestors: ['ns_memcached-default','single_bucket_sup-default',
                  <0.31607.6>]
    messages: []
    links: [<0.31624.6>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 2584
    stack_size: 24
    reductions: 567619
  neighbours:

Link to manifest of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-807-rel.setup.exe.manifest.xml

This bug may relate to bug MB-8266

[MB-8344] UI: Document viewer within view page spins indefinitely when trying to access an item that does not exist

Thu, 23 May 2013 13:17:11 -0500

See attached screenshot. I was able to reproduce this by clicking on an item from a view response that does not exist.

[MB-8343] In Memory-full scenarios, replication incorrectly reports 100 percent complete status, XDCR queue shows incorrect numbers, XDCR is non-functioning at this point.

Thu, 23 May 2013 12:17:41 -0500

Cluster1
1. Load 24M items on bucket0,
2, Setup unidirecrtional replications to cluster2, 3, 4
3. Run system over 10 hours.
4. Replicated data on Cluster3 and 4 show only 16M out of 24M items.

- Replication progress shows "100 percent complete" for all outbound replication - This is incorrect.
- Replication Queue is around 400-500k ( This is aggregated across all the replication streams) - This is incorrect, it should be at least 8+8 M items in outbound queue.
- Replication errors show " Post Condition error 500" - Can we have more detailed errors here? What is the best way to debug this error code/ message.

Remote Cluster 3, 4 stats
- Memory is nearly fully used up with Active resident ratio - 1 percent on both the clusters.
- Incoming XDCR stats show
- Zero setMeta ( which is expected)
- 5-10K get meta ops/sec

Added screenshot from Source Cluster - cluster1

and cluster3

Adding logs from the clusters.

[MB-8342] [windows] Disk Write Queue doesn't drain completely after rebalancing out nodes from the cluster

Thu, 23 May 2013 12:29:59 -0500

Disk Write Queue doesn't drain completely after rebalancing out some nodes from the cluster

Test to reproduce:
./testrunner -i vm-4win.ini -t rebalance.rebalanceout.RebalanceOutTests.incremental_rebalance_out_with_ops,replicas=2,items=100000,max_verify=100000,doc_ops=create

The test creates a 7 node cluster and then keeps rebalancing out 2 nodes at a time while data loading is in progress.

After rebalance is done, one of the nodes in the cluster has ep_queue_size=1 and doesn't drain for > 150 seconds.

Please see attached screenshot, all data activity is finished and compaction also completed, but DWQ is still 1.

Reproducible on both internal VMs and EC2 consistently.

Diagnostics are attached.

[MB-8341] Per node stats for Replication progress i.e. "Percent complete" shows "?".

Thu, 23 May 2013 18:06:33 -0500

Per server stats for Replication progress shows "?".
- Looks like they are not calculated per Server? Attached Screenshot

Steps to repro
1. Setup a 4 node cluster.
2, Load 1M data
3, Setup replication from cluster1 to cluster2.
4. View Outbound XDCR stats " Percentage Complete" for XDC Replicaiton cluster-wide, and then per-node

Results
- Aggregated stats show correct value ~ 30 percent,
- View per node, shows "?" for each node.

[MB-8340] [system test][windows] erlang crash due to memory overflow

Thu, 23 May 2013 18:58:29 -0500

Environment:
7 windows server 2008 r2 64bit with SSD and 8GB RAM

1:10.3.121.173
2:10.3.121.169
3:10.3.121.171
4:10.3.3.214
5:10.3.121.47
6:10.3.3.180
7:10.3.3.181

Cluster setup:
7 nodes cluster
1 default bucket (3GB with replica index enable)
1 sasl bucket (3GB with replica index disable)

Load item to both buckets until active resident ratio down to 70%
Run access phase more than one day with
default bucket: set 5% get 80% expired 5%
sasl bucket: set 5% get 5% delete 20% and expired 80%
After more than one day running, one node in cluster (10.3.121.171) become unstable with erlang crashed
See memory use is used over 3GB over mem quota for couchbase server as show in capture

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-807-rel.setup.exe.manifest.xml

Link to collect info of 6 nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_2/2013_05/7nodes_202-807_nodeDown_erlCrash_20130522-174845.tgz
Due to node 171 is down, I could not get the collect info file
erlang crash dump also attached

Cluster is in failed state now

Impact of this bug. Node 10.3.121.171 is in unstable state. One solution is failover that node, add another node in and rebalance out failed node.

.

[MB-8339] Binary Protocol on Metadata

Thu, 23 May 2013 16:23:10 -0500

http://www.couchbase.com/issues/browse/MB-8217

[MB-8338] [windows] moxi crashed when create bucket with error: bad json configuration

Wed, 22 May 2013 16:08:28 -0500

Install couchbase server 2.0.2-811 on one node windows 2012 64-bit

Create default bucket, see moxi crash right after bucket created with error bad json configuration

Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: WARNING: curl error: transfer closed with outstanding read data remaining from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
2013-05-22 13:11:46: (agent_config.c.705) ERROR: bad JSON configuration from http://127.0.0.1:8091/pools/default/saslBucketsStreaming: Number of vBuckets must be a power of two > 0 and <= 65536 ({
"name": "default",
"nodeLocator": "vbucket",
"saslPassword": "",
"nodes": [{
"hostname": "127.0.0.1:8091",
"ports": {
"direct": 11210,
"proxy": 11211
}
}],
"vBucketServerMap": {
"hashAlgorithm": "CRC",
"numReplicas": 1,
"serverList": ["127.0.0.1:11210"],
"vBucketMap": []
}
})
EOL on stdin. Exiting ns_log000 ns_1@127.0.0.1 13:11:26 - Wed May 22, 2013
Bucket "default" loaded on node 'ns_1@127.0.0.1' in 0 seconds. ns_memcached001 ns_1@127.0.0.1 13:11:16 - Wed May 22, 2013
Created bucket "default" of type: membase
[{num_replicas,1},
{replica_index,false},
{ram_quota,2571108352},
{auth_type,sasl},
{flush_enabled,false}] menelaus_web012 ns_1@127.0.0.1 13:11:16 - Wed May 22, 2013

after moxi restarts, I could use cbworkloadgen.exe to load and read items from bucket

Link to manifest file for this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-811-rel.setup.exe.manifest.xml

[MB-8337] UI fails to retrieve 181 node information during online upgrade

Thu, 23 May 2013 21:07:49 -0500

While Performing rolling upgrade of nodes from 181 -> 202 I am unable to view the server info of the 181 nodes from a node running the 202 UI. Instead the UI shows "loading" string.

Error from logs:

Client-side error-report for user "Administrator" on node 'ns_1@10.3.2.92':

User-Agent:Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7
Got unhandled error: Uncaught TypeError: Cannot read property 'length' of undefined
At: http://10.3.2.92:8091/js/cells.js:64
Backtrace:
Function: collectBacktraceViaCaller
Args:

---------
Function: appOnError
Args:
"Uncaught TypeError: Cannot read property 'length' of undefined"
"http://10.3.2.92:8091/js/cells.js"
64
---------

[MB-8336] Updated/New Front Matter

Thu, 23 May 2013 15:10:09 -0500

Front matter, introduction for 2.0.2 in need of overhaul check of features.

[MB-8335] Move RAM stuff to RAM chapter from Disk Storage

Wed, 22 May 2013 14:11:59 -0500

As a separate task, I think we need to take a look at how we are describing what goes on with RAM and disk. The "Disk Storage" section covers a lot of the RAM stuff...

[MB-8334] [windows] unable to get back to previous step in installer

Wed, 22 May 2013 13:09:47 -0500

On the final step of installation wizard(see screenshot) try to click Back btn.
there is no any reaction on clicking it

[MB-8333] [DOC 2.0.2] [windows] unable to add a 1.8.0 node to 2.0.2 cluster

Wed, 22 May 2013 16:53:33 -0500

2 nodes cluster (2.0.2), trying to add a 1.8.0 node

2013-05-22 00:40:06.346 ns_cluster:5:info:message(ns_1@172.23.105.74) - Failed to add node 172.23.105.72:8091 to cluster. This node cannot add another node ('ns_1@172.23.105.72') because of cluster version compatibility mismatch. Cluster works in [2, 0] mode and node only supports [1, 8]

[MB-8332] moxi crashed when delete non default bucket

Wed, 22 May 2013 12:48:16 -0500

Test on build 2.0.2-809 on windows 8 64bit
Create default bucket,
Delete default bucket ==> no moxi crashed

Create sasl bucket
Delete sasl bucket ==> moxi crashed as show in log

Deleted bucket "saslbucket"
menelaus_web011 ns_1@127.0.0.1 18:43:37 - Tue May 21, 2013
Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: 2013-05-21 18:40:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2013-05-21 18:40:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)
2013-05-21 18:40:36: (agent_config.c.705) ERROR: bad JSON configuration from http://127.0.0.1:8091/pools/default/saslBucketsStreaming: Number of vBuckets must be a power of two > 0 and <= 65536 ({
"name": "default",
"nodeLocator": "vbucket",
"saslPassword": "",
"nodes": [{
"hostname": "127.0.0.1:8091",
"ports": {
"direct": 11210,
"proxy": 11211
}
}],
"vBucketServerMap": {
"hashAlgorithm": "CRC",
"numReplicas": 1,
"serverList": ["127.0.0.1:11210"],
"vBucketMap": []
}
})
2013-05-21 18:43:11: (agent_config.c.705) ERROR: bad JSON configuration from http://127.0.0.1:8091/pools/default/saslBucketsStreaming: Number of vBuckets must be a power of two > 0 and <= 65536 ({
"name": "saslbucket",
"nodeLocator": "vbucket",
"saslPassword": "",
"nodes": [{
"hostname": "127.0.0.1:8091",
"ports": {
"direct": 11210,
"proxy": 11211
}
}],
"vBucketServerMap": {
"hashAlgorithm": "CRC",
"numReplicas": 1,
"serverList": ["127.0.0.1:11210"],
"vBucketMap": []
}
})
EOL on stdin. Exiting ns_log000 ns_1@127.0.0.1 18:43:37 - Tue May 21, 2013
Shutting down bucket "saslbucket" on 'ns_1@127.0.0.1' for deletion ns_memcached002 ns_1@127.0.0.1 18:43:33 - Tue May 21, 2013
Bucket "saslbucket" loaded on node 'ns_1@127.0.0.1' in 0 seconds. ns_memcached001 ns_1@127.0.0.1 18:43:09 - Tue May 21, 2013
Created bucket "saslbucket" of type: membase
[{num_replicas,1},
{replica_index,false},
{ram_quota,2515533824},
{auth_type,sasl},
{autocompaction,false},
{flush_enabled,false}] menelaus_web012 ns_1@127.0.0.1 18:43:09 - Tue May 21, 2013
Shutting down bucket "default" on 'ns_1@127.0.0.1' for deletion ns_memcached002 ns_1@127.0.0.1 18:41:59 - Tue May 21, 2013
Bucket "default" loaded on node 'ns_1@127.0.0.1' in 0 seconds. ns_memcached001 ns_1@127.0.0.1 18:40:34 - Tue May 21, 2013
Created bucket "default" of type: membase
[{num_replicas,1},
{replica_index,false},
{ram_quota,2515533824},
{auth_type,sasl},
{autocompaction,false},
{flush_enabled,false}] menelaus_web012 ns_1@127.0.0.1 18:40:34 - Tue May 21, 2013
Deleted bucket "new-bucket-97c90481-0efb-4ad3-84ed-8a0bc00799ad"
(repeated 1 times) menelaus_web011 ns_1@127.0.0.1 18:40:27 - Tue May 21, 2013
Client-side error-report for user "Administrator" on node 'ns_1@127.0.0.1':
User-Agent:Python-httplib2/$Rev: 259 $
2013-05-21 18:39:58.017545 : test_non_default_moxi finished
menelaus_web102 ns_1@127.0.0.1 18:40:02 - Tue May 21, 2013
Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: 2013-05-21 18:39:43: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2013-05-21 18:39:43: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)
EOL on stdin. Exiting ns_log000 ns_1@127.0.0.1 18:40:02 - Tue May 21, 2013
Shutting down bucket "new-bucket-97c90481-0efb-4ad3-84ed-8a0bc00799ad" on 'ns_1@127.0.0.1' for deletion

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-809-rel.setup.exe.manifest.xml

[MB-8331] Need to move to 5.8.5.cb1

Dependency — Tue, 21 May 2013 14:15:58 -0500

We need to move to our patched Erlang version 5.8.5.cb1 as we decided not to move to newer version of Erlang that has Filipe's changes upstream.

[MB-8330] Documentation for reduce (specially rereduce) function is missleading

Wed, 22 May 2013 13:54:08 -0500

The documentation for reduce-functions is very missleading.

If you just read the first examples, you get the impression, that this function can be called by having a collection of reduced and non-reduced data.

I got informed via IRC by someone, working with CouchDB, that this isn't the case. You have either a collection of reduced or a collection of non-reduced data you get as arguments for this function.

The documentation there also talks about using the build-in function typeof ... I can hardly see that you've made use of that function ...

[MB-8329] ui doesn't show document content

Wed, 22 May 2013 12:53:26 -0500

earlier documents page shown id and content, now content is omitted

[MB-8328] High latency variation and regression in query throughput

Thu, 23 May 2013 16:58:30 -0500

Build vs. query throughput
-----------------
| 170 | 1357 |
-----------------
| 740 | 1210 |
| 745 | 1338 |
| 755 | 1363 |
-----------------
| 758 | 579 |
| 759 | 571 |
| 760 | 583 |
| 765 | 531 |
| 770 | 597 |
| 790 | 530 |
| 799 | 561 |
| 803 | 587 |
| 807 | 573 |
| 809 | 576 |
---------------

80th and 90th percentiles (that we usually track) are not affected but 99th and 99.9th percentiles are much worse.
Obviously average throughput regressed (>2x).

Diags, stats, atop:
2.0.1-170 - http://172.23.96.10:8080/view/views/job/vesta-views/45/artifact/
2.0.2-755 - http://172.23.96.10:8080/view/views/job/vesta-views/62/artifact/
2.0.2-758 - http://172.23.96.10:8080/view/views/job/vesta-views/60/artifact/
2.0.2-809 - http://172.23.96.10:8080/view/views/job/vesta-views/63/artifact/

Manifests:
http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.2-755-rel.rpm.manifest.xml
http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.2-758-rel.rpm.manifest.xml

Changes in bucket_engine:
2a797a8 ~~MB-6144~~: fixed typo
0992a55 ~~MB-6144~~: Some packages used incorrect offsets

Changes in ns_server:
301435b MB-100: fixed build error due to bad .PHONY and rule interaction
414e304 MB-100: provide api for testrunner for messing with memcached
759135b ~~MB-7398~~: added CHANGES entry for hostname management feature
4ac8e1b CBD-220: run memcached and ns_server from babysitting VM
ec84aad CBD-220: [ale] allow syncing all log messages to all sinks
717902e CBD-220: switched ns_server_sup to supervisor2
68c1e9f MB-100: added typespec to ns_pubsub:subscribe_link
f1032ac ~~MB-7398~~: don't allow hostnames without dot
337c98e MB-100: moved samples_loader_tasks earlier in ns_server_sup list
89661aa MB-5307 Handle failure to save address after rename.
1dd0e9e ~~MB-7398~~ Rename added node to what was specified as its address.
c2b0108 ~~MB-7398~~ Ability to specify server hostname in wizard.
3fc9d9d ~~MB-7398~~ REST call to rename a node.
e6aa76a ~~MB-7398~~ Handle ip_start file in dist_manager.
e3b488a ~~MB-7398~~ Move is_good_address to misc.
531d168 MB-5307 Try to stop net_kernel on dist_manager start.
400c20a MB-5307 Remove references to non-existent path-box class.

[MB-8326] during svc restart, node that has already stopped svc, displays "{badrpc,nodedown}"

Thu, 23 May 2013 13:11:07 -0500

centos 5.8
build 803

1). stopped couchbase service
2). do a restart (see command below) -- you will see shutdown failed:

[root@cen-1910 ~]# /etc/init.d/couchbase-server restart
Stopping couchbase-serverNOTE: shutdown failed
{badrpc,nodedown}

Starting couchbase-server [ OK ]
[root@cen-1910 ~]# rpm -qa|grep couchbase
couchbase-server-2.0.2-803
[root@cen-1910 ~]#

Service is ok and restarted fine. No problems. This {badrpmc, nodedown} msg should be eliminated.

[MB-8325] ns_server should collect various stats about supervised processes

Tue, 21 May 2013 19:22:05 -0500

ns_server should collect various stats about supervised processes

[MB-8324] some nodes have 0 active items after offline upgrade

Tue, 21 May 2013 16:13:53 -0500

1) Loaded data into a 9 node cluster at 181.
2) Stopped load, waited for disk-write-queue = 0.
3) shutdown all nodes simultaneously via /etc/init.d/couchbase-server stop
4) upgrade to 2.0.2-807.

Nodes come back up but with less items than before upgrade. 2 nodes have 0 active items.

Impact: apparent data loss
Cluster live ( 10.3.2.89:8091)

all node diags attached.
shutdown occured at timestamp: [2013-05-20 14:37...]
followed by upgrade.

[MB-8323] Crash caused by task running while a bucket is shutting down

Tue, 21 May 2013 13:44:23 -0500

Crash caused by task running while a bucket is shutting down

[MB-8322] Use couch_set_view module for index deletion

Tue, 21 May 2013 09:02:03 -0500

At the moment ns_server implements it's own code to cleanup index files. The couch_set_view module implements a similar logic. ns_server should just call the logic from couch_set_view.

[MB-8321] [RN 2.0.2] windows32: Rebalance exited with reason {mover_failed,{badmatch,{error,eaddrinuse}}}

Wed, 22 May 2013 14:30:58 -0500

http://qa.hq.northscale.net/job/windows32_rebalance-kv/11/consoleFull

./testrunner -i /tmp/4-w-32.ini get-cbcollect-info=True -t rebalancetests.IncrementalRebalanceOut.test_load,replica=1,do-stop=True

2013-05-19 06:24:15,085] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.2.179%2Cns_1%4010.3.2.185%2Cns_1%4010.3.2.178%2Cns_1%4010.3.2.183
[2013-05-19 06:24:15,296] - [rest_client:929] INFO - rebalance operation started
[2013-05-19 06:24:15,334] - [rest_client:1031] INFO - rebalance percentage : 0 %
[2013-05-19 06:24:17,376] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-19 06:24:19,438] - [rest_client:1031] INFO - rebalance percentage : 0.703125 %
[2013-05-19 06:24:21,473] - [rest_client:1031] INFO - rebalance percentage : 1.640625 %
[2013-05-19 06:24:23,499] - [rest_client:1031] INFO - rebalance percentage : 2.6953125 %
[2013-05-19 06:24:25,523] - [rest_client:1031] INFO - rebalance percentage : 3.6328125 %
[2013-05-19 06:24:26,544] - [rest_client:1031] INFO - rebalance percentage : 4.21875 %
[2013-05-19 06:24:27,572] - [rest_client:1031] INFO - rebalance percentage : 4.8046875 %
[2013-05-19 06:24:28,601] - [rest_client:1031] INFO - rebalance percentage : 5.5078125 %
[2013-05-19 06:24:30,639] - [rest_client:1031] INFO - rebalance percentage : 6.4453125 %
[2013-05-19 06:24:31,663] - [rest_client:1014] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2013-05-19 06:24:31,663] - [rest_client:1015] INFO - Latest logs from UI:
[2013-05-19 06:24:31,764] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.179', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.179\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368969856238, u'type': u'info'}
[2013-05-19 06:24:31,764] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.179', u'code': 4, u'text': u"Node 'ns_1@10.3.2.179' saw that node 'ns_1@10.3.2.185' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1368969855691, u'type': u'info'}
[2013-05-19 06:24:31,765] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.179', u'code': 3, u'text': u'Node ns_1@10.3.2.179 joined cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1368969854738, u'type': u'info'}
[2013-05-19 06:24:31,765] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.179', u'code': 4, u'text': u"Node 'ns_1@10.3.2.179' saw that node 'ns_1@10.3.2.183' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1368969854738, u'type': u'info'}
[2013-05-19 06:24:31,765] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.179', u'code': 0, u'text': u"Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: WARNING: curl error: transfer closed with outstanding read data remaining from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming\nEOL on stdin. Exiting", u'shortText': u'message', u'module': u'ns_log', u'tstamp': 1368969854628, u'type': u'info'}
[2013-05-19 06:24:31,765] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.179', u'code': 1, u'text': u"Couchbase Server has started on web port 8091 on node 'ns_1@10.3.2.179'.", u'shortText': u'web start ok', u'module': u'menelaus_sup', u'tstamp': 1368969854597, u'type': u'info'}
[2013-05-19 06:24:31,766] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.178', u'code': 2, u'text': u'Rebalance exited with reason {mover_failed,{badmatch,{error,eaddrinuse}}}\n', u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368969854354, u'type': u'info'}
[2013-05-19 06:24:31,766] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.178', u'code': 0, u'text': u'<0.26161.33> exited with {mover_failed,{badmatch,{error,eaddrinuse}}}', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368969854339, u'type': u'critical'}
[2013-05-19 06:24:31,767] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.185', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.185\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368969850886, u'type': u'info'}
[2013-05-19 06:24:31,767] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.185', u'code': 3, u'text': u'Node ns_1@10.3.2.185 joined cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1368969850479, u'type': u'info'}

[MB-8320] [windows] incorrect text is shown in installer wizard

Mon, 20 May 2013 14:19:25 -0500

[windows] incorrect text is shown in installer wizard

[MB-8319] [DOC? 2.0.2] View compaction is triggered during rebalance when auto-compaction is disabled

Tue, 21 May 2013 18:20:45 -0500

Steps:
1. Set fragmentation threshold to 100% (both data and index)
2. Start workload, view fragmentation is high, >90%.
3. Rebalance cluster

As result view compaction kicks in. Bucket compaction doesn't kick in.

Not sure if it's a bug but behavior is very confusing. At least I didn't find documentation that explains it.

Diags for 2.0.2, just in case:
http://172.23.96.10:8080/view/views/job/apollo-views/116/artifact/

Basically I just want to run a test with no view compaction.

[MB-8318] memcached crashed in CouchKVStore::setVBucketState

Tue, 21 May 2013 10:53:38 -0500

http://qa.hq.northscale.net/job/cen-ubu-64-2.0-new-rebalance-mixed-cluster-P1/9/consoleFull

1.8.1 & 2.0.2-807 ( mixed cluster)
1_8_1:
10.5.2.11
10.6.2.81
10.6.2.82

2.0.2-807-rel:
10.3.121.119
10.3.121.121
10.3.121.122
10.3.121.123

on 10.3.121.121( root/couchbase):

[root@localhost tmp]# ls -la
total 1345100
drwxrwxrwt 10 root root 4096 May 20 01:09 .
drwxr-xr-x 24 root root 4096 May 18 09:40 ..
drwx------ 2 root root 4096 May 20 00:00 atop.d
drwxr-xr-x 2 root root 4096 May 18 03:24 backup
-rw------- 1 couchbase couchbase 698519552 May 19 17:13 core.memcached.14586
-rw------- 1 couchbase couchbase 947982336 May 19 13:57 core.memcached.3876
-rw-r--r-- 1 root root 129834728 May 18 00:32 couchbase-server-enterprise_x86_64_2.0.2-807-rel.rpm
drwxrwxrwt 2 root root 4096 May 18 09:40 .font-unix
-rw------- 1 root root 66 May 18 00:33 .gdm6DAMUW
srw-rw-rw- 1 root root 0 May 18 09:40 .gdm_socket
drwxrwxrwt 2 root root 4096 May 18 09:40 .ICE-unix
drwx------ 2 root root 4096 Apr 2 12:32 keyring-iUmHcT
drwx------ 2 couchbase couchbase 4096 Apr 11 2012 keyring-JYEPdw
drwx------ 2 root root 4096 May 18 09:40 vmware-root
-r--r--r-- 1 root root 11 May 18 09:40 .X0-lock
drwxrwxrwt 2 root root 4096 May 18 09:40 .X11-unix
[root@localhost tmp]# gdb /opt/couchbase/bin/memcached core.memcached.14586
GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 15231]
[New Thread 20939]
[New Thread 20938]
[New Thread 20937]
[New Thread 15236]
[New Thread 15235]
[New Thread 15234]
[New Thread 15233]
[New Thread 15232]
[New Thread 15230]
[New Thread 14593]
[New Thread 14592]
[New Thread 14591]
[New Thread 14590]
[New Thread 14589]
[New Thread 14588]
[New Thread 14587]
[New Thread 14586]

warning: .dynamic section for "/usr/lib64/libstdc++.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff41d5f000
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x0000003018630265 in raise () from /lib64/libc.so.6
(gdb) t a a bt

Thread 18 (Thread 0x2b6944d5f220 (LWP 14586)):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772000, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772000, flags=<value optimized out>) at event.c:1558
#3 0x00000000004097d6 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7926

Thread 17 (Thread 14587):
#0 0x00000030186c678b in read () from /lib64/libc.so.6
#1 0x000000301866cd57 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x000000301866d71e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x0000003018662804 in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00000030186616a9 in fgets () from /lib64/libc.so.6
#5 0x00002b6944d60939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#6 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#7 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 16 (Thread 14588):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0xef0a040) at extensions/loggers/file_logger.c:368
#2 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#3 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 15 (Thread 14589):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772500, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772500, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0d900) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 14 (Thread 14590):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772280, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0d9f8) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 13 (Thread 14591):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772c80, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772c80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0daf0) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 12 (Thread 14592):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772a00, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772a00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0dbe8) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 11 (Thread 14593):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772780, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0dce0) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 10 (Thread 15230):
#0 0x000000301869a541 in nanosleep () from /lib64/libc.so.6
#1 0x00000030186cded4 in usleep () from /lib64/libc.so.6
#2 0x00002aaaaaf32805 in updateStatsThread (arg=0xef0a4c0) at src/memory_tracker.cc:31
#3 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#4 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 9 (Thread 15232):

#0 0x000000301920d524 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003019208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003019208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002aaaaaf336fa in Mutex::acquire (this=0x137ed110) at src/mutex.cc:79
#4 0x00002aaaaaf79dc3 in lock (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00002aaaaaf70163 in CouchKVStore::setVBucketState (this=0x195f6300, vbucketId=1, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#8 0x00002aaaaaf71069 in CouchKVStore::snapshotVBuckets (this=0x195f6300, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#9 0x00002aaaaaf021f3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2ca5b800, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#10 0x00002aaaaaf51cff in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#11 0x00002aaaaaf371c1 in ExecutorThread::run (this=0x1381da00) at src/scheduler.cc:153
#12 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#13 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#14 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 8 (Thread 15233):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x1381d860) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x1381d860) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 7 (Thread 15234):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x13844ea0) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x13844ea0) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 6 (Thread 15235):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x13844d00) at src/syncobject.hh:57
---Type <return> to continue, or q <return> to quit---
#2 ExecutorThread::run (this=0x13844d00) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 5 (Thread 15236):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x13844b60) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x13844b60) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 4 (Thread 20937):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf0ebdf in wait (this=0x13770400) at src/syncobject.hh:57
#2 wait (this=0x13770400) at src/syncobject.hh:73
#3 wait (this=0x13770400) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x13770400) at src/ep_engine.cc:3377
#5 0x00002aaaaaf0ecc3 in EvpNotifyPendingConns (arg=0x13770400) at src/ep_engine.cc:1153
#6 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#7 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 3 (Thread 20938):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef31c8 in wait (this=0x1428d440, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x1428d440, d=...) at src/dispatcher.cc:342
#3 0x00002aaaaaef5d2a in Dispatcher::run (this=0x137b2a80) at src/dispatcher.cc:184
#4 0x00002aaaaaef64ed in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
#5 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 2 (Thread 20939):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef31c8 in wait (this=0x1428ccf0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x1428ccf0, d=...) at src/dispatcher.cc:342
#3 0x00002aaaaaef5d2a in Dispatcher::run (this=0x137b2fc0) at src/dispatcher.cc:184
#4 0x00002aaaaaef64ed in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
#5 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x4651e940 (LWP 15231)):
#0 0x0000003018630265 in raise () from /lib64/libc.so.6
#1 0x0000003018631d10 in abort () from /lib64/libc.so.6
#2 0x00002aaaaaf7a779 in BinaryPacketHandler::implicitResponse (this=0x38fa) at src/couch-kvstore/couch-notifier.cc:41
#3 0x00002aaaaaf77009 in CouchNotifier::handleResponse (this=0x137ed000, res=0x137fc000) at src/couch-kvstore/couch-notifier.cc:216
#4 0x00002aaaaaf794eb in CouchNotifier::processInput (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:571
#5 0x00002aaaaaf79185 in waitOnce (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:677
#6 CouchNotifier::selectBucket (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:725
#7 0x00002aaaaaf7966f in CouchNotifier::processInput (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:608
#8 0x00002aaaaaf79f75 in waitOnce (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:677
#9 CouchNotifier::notify_update (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:755
#10 0x00002aaaaaf70163 in CouchKVStore::setVBucketState (this=0x195f6000, vbucketId=0, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#11 0x00002aaaaaf71069 in CouchKVStore::snapshotVBuckets (this=0x195f6000, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
---Type <return> to continue, or q <return> to quit---
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#12 0x00002aaaaaf021f3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2ca5b800, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#13 0x00002aaaaaf51cff in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#14 0x00002aaaaaf371c1 in ExecutorThread::run (this=0x1381dba0) at src/scheduler.cc:153
#15 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#16 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#17 0x00000030186d44bd in clone () from /lib64/libc.so.6
(gdb) quit

[MB-8317] Rebalance fails when ejecting out nodes in a 1.8.1 & 2.0.2 mixed cluster while data is being inserted/mutated

Mon, 20 May 2013 13:30:23 -0500

1.8.1 & 2.0.2 mixed cluster: Rebalance exited with reason {{change_filter_failed,{'EXIT',{noproc,[{erlang,link,[<17560.12069.16>]},{ns_vbm_new_sup,'-perform_vbucket_filter_change/6-fun-1-'

very similar bug ~~MB-8309~~

1.8.1 & 2.0.2-807 ( mixed cluster)http://qa.hq.northscale.net/job/cen-ubu-64-2.0-new-rebalance-mixed-cluster-P1/9/consoleFull

./testrunner -i /tmp/rebalance_in.ini wait_timeout=180,GROUP=P1,EXCLUDE_GROUP=FROM_2_0,get-cbcollect-info=True -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops,nodes_out=3,replicas=2,items=50000,doc_ops=create;update;delete,GROUP=OUT;P1

1_8_1:
10.5.2.11
10.6.2.81
10.6.2.82

2.0.2-807-rel:
10.3.121.119
10.3.121.121
10.3.121.122
10.3.121.123

2013-05-19 14:30:01,445] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.121.119%2Cns_1%4010.3.121.122%2Cns_1%4010.6.2.81%2Cns_1%4010.3.121.123%2Cns_1%4010.6.2.82%2Cns_1%4010.5.2.11%2Cns_1%4010.3.121.121
[2013-05-19 14:30:02,062] - [rest_client:929] INFO - rebalance operation started
[2013-05-19 14:30:02,080] - [rest_client:1031] INFO - rebalance percentage : 0 %
[2013-05-19 14:30:12,106] - [task:340] INFO - rebalancing was completed with progress: 100% in 10.0423641205 sec
....
[2013-05-19 14:30:47,707] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.121.123%2Cns_1%4010.3.121.122%2Cns_1%4010.3.121.121&user=Administrator&knownNodes=ns_1%4010.3.121.119%2Cns_1%4010.3.121.122%2Cns_1%4010.6.2.81%2Cns_1%4010.3.121.123%2Cns_1%4010.6.2.82%2Cns_1%4010.5.2.11%2Cns_1%4010.3.121.121
[2013-05-19 14:30:47,720] - [rest_client:929] INFO - rebalance operation started
[2013-05-19 14:30:47,730] - [rest_client:1031] INFO - rebalance percentage : 0 %
[2013-05-19 14:30:57,756] - [rest_client:1014] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2013-05-19 14:30:57,757] - [rest_client:1015] INFO - Latest logs from UI:
[2013-05-19 14:30:57,815] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 0, u'text': u"Current master is older (before 2.0.1) and I'll try to takeover (repeated 1 times)", u'shortText': u'message', u'module': u'mb_master', u'tstamp': 1368999050547.0, u'type': u'warning'}
[2013-05-19 14:30:57,816] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 2, u'text': u"Rebalance exited with reason {{change_filter_failed,\n {'EXIT',\n {noproc,\n [{erlang,link,[<17560.12069.16>]},\n {ns_vbm_new_sup,\n '-perform_vbucket_filter_change/6-fun-1-',\n 7},\n {misc,'-executing_on_new_process/1-fun-0-',\n 3}]}}},\n [{ns_vbm_sup,change_vbucket_filter,4},\n {ns_vbm_sup,'-set_replicas/3-fun-2-',5},\n {lists,foldl,3},\n {ns_vbm_sup,set_replicas,3},\n {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',\n 3},\n {lists,foreach,2},\n {janitor_agent,\n do_bulk_set_vbucket_state_old_style,4},\n {ns_vbucket_mover,handle_call,3}]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368999050224.0, u'type': u'info'}
[2013-05-19 14:30:57,818] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368999049409.0, u'type': u'critical'}
[2013-05-19 14:30:57,819] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368999049155.0, u'type': u'info'}
[2013-05-19 14:30:57,820] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'module': u'ns_rebalancer', u'tstamp': 1368999048756.0, u'type': u'info'}
[2013-05-19 14:30:57,821] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.119', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.3.121.119','ns_1@10.6.2.81',\n 'ns_1@10.6.2.82','ns_1@10.5.2.11'], EjectNodes = ['ns_1@10.3.121.123',\n 'ns_1@10.3.121.122',\n 'ns_1@10.3.121.121']\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368999048675.0, u'type': u'info'}
[2013-05-19 14:30:57,822] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.5.2.11', u'code': 102, u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@10.5.2.11\':\nUser-Agent:Python-httplib2/$Rev: 259 $\n2013-05-19 14:30:19.598873 : rebalance_out_with_ops started \n', u'shortText': u'client-side error report', u'module': u'menelaus_web', u'tstamp': 1368999020803.0, u'type': u'warning'}
[2013-05-19 14:30:57,823] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.5.2.11', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.5.2.11\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368999014887.0, u'type': u'info'}
[2013-05-19 14:30:57,825] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.6.2.81', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.6.2.81\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368999013947.0, u'type': u'info'}
[2013-05-19 14:30:57,827] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.6.2.82', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.6.2.82\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368999013933.0, u'type': u'info'}

[MB-8316] Fix cbtransfer --help output

Sun, 19 May 2013 00:02:06 -0500

Need to Engineering to update descriptions for cbtransfer --h to match with documentation. We just edited these for 2.0.2 int he docs and made them readable/understandable.

Not sure what this release this should be in.

-h, --help Command help
-b BUCKET_SOURCE Single named bucket from source cluster to transfer
-B BUCKET_DESTINATION, --bucket-destination=BUCKET_DESTINATION Single named bucket on destination cluster which receives transfer. This allows you to transfer to a bucket with a different name as your source bucket. If you do not provide defaults to the same name as the bucket-source
-i ID, --id=ID Transfer only items that match a vbucketID
-k KEY, --key=KEY Transfer only items with keys that match a regexp
-n, --dry-run No actual transfer; just validate parameters, files, connectivity and configurations
-u USERNAME, --username=USERNAME REST username for source cluster or server node
-p PASSWORD, --password=PASSWORD REST password for cluster or server node
-t THREADS, --threads=THREADS Number of concurrent workers threads performing the transfer. Defaults to 4.
-v, --verbose Verbose logging; provide more verbosity
-x EXTRA, --extra=EXTRA Provide extra, uncommon config parameters
--single-node Transfer from a single server node in a source cluster. This single server node is a source node URL
--source-vbucket-state=SOURCE_VBUCKET_STATE Only transfer from source vbuckets in this state, such as 'active' (default) or 'replica'. Must be used with Couchbase cluster as source.
--destination-vbucket-state=DESTINATION_VBUCKET_STATE Only transfer to destination vbuckets in this state, such as 'active' (default) or 'replica'. Must be used with Couchbase cluster as destination.
--destination-operation=DESTINATION_OPERATION Perform this operation on transfer. "set" will override an existing document, 'add' will not override, 'get' will load all keys transferred from a source cluster into the caching layer at the destination.
The following are extra, specialized command options you use in this form cbtransfer -x [EXTRA OPTIONS]:
batch_max_bytes=400000 Transfer this # of bytes per batch.
batch_max_size=1000 Transfer this # of documents per batch
cbb_max_mb=100000 Split backup file on destination cluster if it exceeds MB
max_retry=10 Max number of sequential retries if transfer fails
nmv_retry=1 0 or 1, where 1 retries transfer after a NOT_MY_VBUCKET message. Default of 1.
recv_min_bytes=4096 Amount of bytes for every TCP/IP batch transferred
report=5 Number batches transferred before updating progress bar in console
report_full=2000 Number batches transferred before emitting progress information in console
try_xwm=1 As of 2.0.2, transfer documents with metadata. 1 is default. 0 should only be used if you transfer from 1.8.x to 1.8.x.
For 2.0.2
   <row>
   <entry>data_only=0</entry>
   <entry>For value 1, only transfer data from a backup file or cluster.</entry>
   </row>
   <row>
   <entry>design_doc_only=0</entry>
   <entry>For value 1, transfer design documents only from a backup file or cluster. Defaults to 0.</entry>
   </row>
   For 2.0.2

[MB-8315] Need Decision on cbtransfer, recv_min_bytes=4096

Sun, 19 May 2013 00:02:21 -0500

It is a batch side for TCP/IP used in the transfer. Not sure if it should be public/tunable or not......

Public for now.

[MB-8314] rebalance exited ns_vbucket_mover failed to initiate_indexing

Wed, 22 May 2013 16:21:42 -0500

Have a 5 node cluster that was upgraded from 181 -> 202 (build 805).
After upgrade attempted to swap out orchestrator(10.3.121.69) and add in a new 202 node (10.3.3.131)

Rebalance fails with ns_vbucket_mover000 on orchestrator reporting:
<0.11037.1> exited with {noproc,
{gen_server,call,
[{'janitor_agent-saslbucket',
'ns_1@10.3.3.131'},
{if_rebalance,<0.1.1>,initiate_indexing},
infinity]}}

On new node 202, couchdb went down and erlang crash dump was generated(attached).

Saw these errors from mccouch which may be why vbuckets couldn't be moved to this node:
Fri May 17 07:27:36.981472 PDT 3: (saslbucket) Trying to connect to mccouch: "127.0.0.1:11213"
Fri May 17 07:27:36.981615 PDT 3: (saslbucket) Connected to mccouch: "127.0.0.1:11213"
Fri May 17 07:27:37.019496 PDT 3: (saslbucket) Connection closed by mccouch
Fri May 17 07:27:37.019527 PDT 3: (saslbucket) Resetting connection to mccouch, lastReceivedCommand = notify_vbucket_update lastSentCommand = notify_vbucket_update currentCommand =unknown
Fri May 17 07:27:37.019595 PDT 3: (saslbucket) Trying to connect to mccouch: "127.0.0.1:11213"
Fri May 17 07:27:37.019730 PDT 3: (saslbucket) Connected to mccouch: "127.0.0.1:11213"
Fri May 17 07:27:37.021763 PDT 3: (saslbucket) Connection closed by mccouch
Fri May 17 07:27:37.021788 PDT 3: (saslbucket) Resetting connection to mccouch, lastReceivedCommand = select_bucket lastSentCommand = notify_vbucket_update currentCommand =unknown

=========================CRASH REPORT=========================
  crasher:
    initial call: mc_connection:init/1
    pid: <0.965.0>
    registered_name: []
    exception error: no case clause matching {error,system_limit}
      in function mc_connection:do_notify_vbucket_update/3
      in call from mc_connection:handle_message/9
      in call from mc_connection:read_full_message/2
      in call from mc_connection:run_loop/2
    ancestors: [mc_conn_sup,mc_sup,ns_server_sup,ns_server_cluster_sup,
                  <0.59.0>]
    messages: []
    links: [<0.641.0>,#Port<0.6784>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 1094838
  neighbours:

[error_logger:error,2013-05-17T7:27:36.622,ns_1@10.3.3.131:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
     Supervisor: {local,mc_conn_sup}
     Context: child_terminated
     Reason: {case_clause,{error,system_limit}}
     Offender: [{pid,<0.965.0>},
                  {name,mc_connection},
                  {mfargs,{mc_connection,start_link,undefined}},
                  {restart_type,temporary},
                  {shutdown,brutal_kill},
                  {child_type,worker}]

=========================CRASH REPORT=========================
  crasher:
    initial call: couch_file:spawn_reader/2
    pid: <0.652.0>
    registered_name: []
    exception exit: {problem_reopening_file,
                        {error,system_limit},
                        {set_close_after,infinity,<0.650.0>},
                        <0.652.0>,
                        "/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1",
                        10}
      in function couch_file:reader_loop/3
    ancestors: [<0.650.0>,couch_server,couch_primary_services,
                  couch_server_sup,cb_couch_sup,ns_server_cluster_sup,
                  <0.59.0>]
    messages: []
    links: [<0.650.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 377
    stack_size: 24
    reductions: 504
  neighbours:

[MB-8313] Table 7.4. Administration — Standard couchbase Tool Options should be updated with new options

Wed, 22 May 2013 15:35:32 -0500

http://www.couchbase.com/docs/couchbase-manual-2.0.pdf

Table 7.4. Administration — Standard couchbase Tool Options doesn't contains the full list of available options.

Actual info we can get from from './couchbase-cli -h":

CLUSTER:
  --cluster=HOST[:PORT] or -c HOST[:PORT]

OPTIONS:
  -u USERNAME, --user=USERNAME admin username of the cluster
  -p PASSWORD, --password=PASSWORD admin password of the cluster
  -o KIND, --output=KIND KIND is json or standard
  -d, --debug

server-add OPTIONS:
  --server-add=HOST[:PORT] server to be added
  --server-add-username=USERNAME admin username for the
                                    server to be added
  --server-add-password=PASSWORD admin password for the
                                    server to be added

server-readd OPTIONS:
  --server-add=HOST[:PORT] server to be added
  --server-add-username=USERNAME admin username for the
                                    server to be added
  --server-add-password=PASSWORD admin password for the
                                    server to be added

rebalance OPTIONS:
  --server-add* see server-add OPTIONS
  --server-remove=HOST[:PORT] the server to be removed

failover OPTIONS:
  --server-failover=HOST[:PORT] server to failover

cluster-* OPTIONS:
  --cluster-username=USER new admin username
  --cluster-password=PASSWORD new admin password
  --cluster-port=PORT new cluster REST/http port
  --cluster-ramsize=RAMSIZEMB per node ram quota in MB

node-init OPTIONS:
  --node-init-data-path=PATH per node path to store data
  --node-init-index-path=PATH per node path to store index

bucket-* OPTIONS:
  --bucket=BUCKETNAME bucket to act on
  --bucket-type=TYPE memcached or couchbase
  --bucket-port=PORT supports ASCII protocol and is auth-less
  --bucket-password=PASSWORD standard port, exclusive with bucket-port
  --bucket-ramsize=RAMSIZEMB ram quota in MB
  --bucket-replica=COUNT replication count
  --enable-flush=[0|1] enable/disable flush
  --enable-index-replica=[0|1] enable/disable index replicas
  --wait wait for bucket create to be complete before returning
  --force force to execute command without asking for confirmation
  --data-only compact datbase data only
  --view-only compact view data only

setting-compacttion OPTIONS:
  --compaction-db-percentage=PERCENTAGE at which point database compaction is triggered
  --compaction-db-size=SIZE[MB] at which point database compaction is triggered
  --compaction-view-percentage=PERCENTAGE at which point view compaction is triggered
  --compaction-view-size=SIZE[MB] at which point view compaction is triggered
  --compaction-period-from=HH:MM allow compaction time period from
  --compaction-period-to=HH:MM allow compaction time period to
  --enable-compaction-abort=[0|1] allow compaction abort when time expires
  --enable-compaction-parallel=[0|1] allow parallel compaction for database and view

setting-notification OPTIONS:
  --enable-notification=[0|1] allow notification

setting-alert OPTIONS:
  --enable-email-alert=[0|1] allow email alert
  --email-recipients=RECIPIENT email recipents, separate addresses with , or ;
  --email-sender=SENDER sender email address
  --email-user=USER email server username
  --email-password=PWD email server password
  --email-host=HOST email server host
  --email-port=PORT email server port
  --enable-email-encrypt=[0|1] email encrypt
  --alert-auto-failover-node node was auto failover
  --alert-auto-failover-max-reached maximum number of auto failover nodes was reached
  --alert-auto-failover-node-down node wasn't auto failover as other nodes are down at the same time
  --alert-auto-failover-cluster-small node wasn't auto fail over as cluster was too small
  --alert-ip-changed node ip address has changed unexpectedly
  --alert-disk-space disk space used for persistent storgage has reached at least 90% capacity
  --alert-meta-overhead metadata overhead is more than 50%
  --alert-meta-oom bucket memory on a node is entirely used for metadata
  --alert-write-failed writing data to disk for a specific bucket has failed

setting-autofailover OPTIONS:
  --enable-auto-failover=[0|1] allow auto failover
  --auto-failover-timeout=TIMEOUT (>=30) specify timeout that expires to trigger auto failover

setting-xdcr OPTIONS:
  --max-concurrent-reps=[32] maximum concurrent replications per bucket, 8 to 256.
  --checkpoint-interval=[1800] intervals between checkpoints, 60 to 14400 seconds.
  --worker-batch-size=[500] doc batch size, 500 to 10000.
  --doc-batch-size=[2048]KB document batching size, 10 to 100000 KB
  --failure-restart-interval=[30] interval for restarting failed xdcr, 1 to 300 seconds
  --optimistic-replication-threshold=[256] document body size threshold (bytes) to trigger optimistic replication

xdcr-setup OPTIONS:
  --create create a new xdcr configuration
  --edit modify existed xdcr configuration
  --delete delete existed xdcr configuration
  --xdcr-cluster-name=CLUSTERNAME cluster name
  --xdcr-hostname=HOSTNAME remote host name to connect to
  --xdcr-username=USERNAME remote cluster admin username
  --xdcr-password=PASSWORD remtoe cluster admin password

xdcr-replicate OPTIONS:
  --create create and start a new replication
  --delete stop and cancel a replication
  --xdcr-from-bucket=BUCKET local bucket name to replicate from
  --xdcr-clucter-name=CLUSTERNAME remote cluster to replicate to
  --xdcr-to-bucket=BUCKETNAME remote bucket to replicate to

[MB-8312] we don't have Table 7.3. and Table 7.4. (clouhbase-cli stuff)in online documentation

Fri, 17 May 2013 14:34:42 -0500

I see that they are presented in http://www.couchbase.com/docs/couchbase-manual-2.0.pdf
Table 7.3. Administration — couchbase Tool Commands
Table 7.4. Administration — Standard couchbase Tool Options

but missed in online documentation for '7.4. couchbase-cli Tool' in http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-cmdline-couchbase-cli.html

[MB-8311] Table 7.3. Administration — couchbase Tool Commands should be updated with new features

Wed, 22 May 2013 15:35:00 -0500

according to http://www.couchbase.com/docs/couchbase-manual-2.0.pdf

Table 7.3. Administration — couchbase Tool Commands
Command Description
server-list List all servers in a cluster
server-info Show details on one server
server-add Add one or more servers to the cluster
server-readd Re-add a server that was failed over to the cluster
rebalance Start a cluster rebalancingCommand-line Interface for Administration
169
Command Description
rebalance-stop Stop current cluster rebalancing
rebalance-status Show status of current cluster rebalancing
failover Failover one or more servers
cluster-init Set the username, password and port of the cluster
node-init Set node specific parameters
bucket-list List all buckets in a cluster
bucket-create Add a new bucket to the cluster
bucket-edit Modify an existing bucket
bucket-delete Delete an existing bucket
bucket-flush Flush a given bucket
help Show longer usage/help and examples

but the full list should be for 2.0.2 as:

./couchbase-cli
usage: couchbase-cli COMMAND CLUSTER [OPTIONS]

CLUSTER is --cluster=HOST[:PORT] or -c HOST[:PORT]

COMMANDs include
  server-list list all servers in a cluster
  server-info show details on one server
  server-add add one or more servers to the cluster
  server-readd readd a server that was failed over
  rebalance start a cluster rebalancing
  rebalance-stop stop current cluster rebalancing
  rebalance-status show status of current cluster rebalancing
  failover failover one or more servers
  cluster-init set the username,password and port of the cluster
  cluster-edit modify cluster settings
  node-init set node specific parameters
  bucket-list list all buckets in a cluster
  bucket-create add a new bucket to the cluster
  bucket-edit modify an existing bucket
  bucket-delete delete an existing bucket
  bucket-flush flush all data from disk for a given bucket
  bucket-compact compact database and index data
  setting-compaction set auto compaction settings
  setting-notification set notification settings
  setting-alert set email alert settings
  setting-autofailover set auto failover settings
  setting-xdcr set xdcr related settings
  xdcr-setup set up XDCR connection
  xdcr-replicate xdcr operations
  help show longer usage/help and examples

[MB-8310] Couchdb gerrit changes for master must use manifest 2.1-unstable.xml

Fri, 17 May 2013 06:28:32 -0500

2.1-stable.xml was abandoned, and any couchdb master gerrit change depends on couchstore revision more recent than the one listed in 2.1-stable.xml - this means the jobs couchdb-gerrit-views-master and couchdb-gerrit-views-pre-merge-master always fail.

[MB-8309] online_upgrade_swap_rebalance 1.8.1->2.0.2: Rebalance exited with reason {{change_filter_failed,{'EXIT',{{unexpected_reason,killed}

Mon, 20 May 2013 17:15:48 -0500

http://qa.hq.northscale.net/job/centos-32-2.0-upgrade-P1/4/consoleFull

./testrunner -i /tmp/centos-32-2.0-upgrade.ini upgrade_version=2.0.2-805-rel,get-cbcollect-info=True,GROUP=P1 -t newupgradetests.MultiNodesUpgradeTests.online_upgrade_swap_rebalance,initial_version=1.8.1-942-rel,standard_buckets=1,items=500000,max_verify=1000,GROUP=1_8;ONLINE;WINDOWS;P1

steps:
1)10.3.3.151, 10.3.3.152 with 1.8.1 build
10.3.3.153 with 2.0.2-805-rel

2)[2013-05-17 01:11:42,198] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.151&user=Administrator&knownNodes=ns_1%4010.3.3.152%2Cns_1%4010.3.3.151%2Cns_1%4010.3.3.153
...
[2013-05-17 01:26:15,908] - [task:340] INFO - rebalancing was completed with progress: 100.0% in 873.699863911 sec

3)[2013-05-17 01:26:37,009] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.152&user=Administrator&knownNodes=ns_1%4010.3.3.152%2Cns_1%4010.3.3.153
...

[2013-05-17 01:31:07,549] - [rest_client:1014] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2013-05-17 01:31:07,550] - [rest_client:1015] INFO - Latest logs from UI:
[2013-05-17 01:31:07,601] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 2, u'text': u"Rebalance exited with reason {{change_filter_failed,\n {'EXIT',\n {{unexpected_reason,killed},\n [{misc,executing_on_new_process,1},\n {ns_vbm_sup,local_change_vbucket_filter,4},\n {rpc,local_call,3},\n {ns_vbm_sup,change_vbucket_filter,4},\n {ns_vbm_sup,'-set_replicas/3-fun-2-',5},\n {lists,foldl,3},\n {ns_vbm_sup,set_replicas,3},\n {ns_vbm_sup,\n '-set_replicas_on_nodes/3-fun-1-',3}]}}},\n [{ns_vbm_sup,change_vbucket_filter,4},\n {ns_vbm_sup,'-set_replicas/3-fun-2-',5},\n {lists,foldl,3},\n {ns_vbm_sup,set_replicas,3},\n {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',\n 3},\n {lists,foreach,2},\n {janitor_agent,\n do_bulk_set_vbucket_state_old_style,4},\n {ns_vbucket_mover,\n update_replication_post_move,3}]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368779444497.0, u'type': u'info'}
[2013-05-17 01:31:07,602] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 43 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779418787.0, u'type': u'critical'}
[2013-05-17 01:31:07,603] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779358861.0, u'type': u'critical'}
[2013-05-17 01:31:07,603] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 179 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779358787.0, u'type': u'critical'}
[2013-05-17 01:31:07,604] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779299138.0, u'type': u'critical'}
[2013-05-17 01:31:07,604] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 154 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779298786.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779238798.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 132 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779238787.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779184396.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Bucket "standard_bucket0" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368779184144.0, u'type': u'info'}

[MB-8308] Rebalance time regression in large scale DGM vperf tests

Relates to — Wed, 22 May 2013 14:06:58 -0500

Timings (builds vs. rebalance time in sec):
--------------------
| 170 | 19763 |
--------------------
| 740 | 28559 |
| 761 | 26748 |
| 799 | 37526 |
| 803 | 34903 |
| 807 | 26430 |
| 809 | 26430 |
--------------------

diags, master events, atop:
2.0.1-170 - http://172.23.96.10:8080/view/views/job/apollo-views/105/artifact/
2.0.2-740 - http://172.23.96.10:8080/view/views/job/apollo-views/118/artifact/
2.0.2-809 - http://172.23.96.10:8080/view/views/job/apollo-views/121/artifact/

[MB-8307] Litmus dashboard becomes unresponsive after fetching big set of results

Fri, 17 May 2013 14:27:12 -0500

I can see that it eats one of my cores for a couple of seconds.

Firefox tries to kill the script from time to time.

[MB-8306] Number of document mutations pending XDC replication is rapidly growing

Thu, 23 May 2013 17:56:22 -0500

Also time spent waiting for checkpointing is much higher.

Diags (internal IP address):
http://172.23.96.10:8080/view/xdcr/job/xperf-lnx-bi/25/artifact/

[MB-8305] 2.1 manifests have reference to membase-cli

Fri, 17 May 2013 00:16:05 -0500

membase/membase-cli instead of couchbase/couchbase-cli

For instance:
https://github.com/couchbase/manifest/blob/master/2.1-stable.xml

I'm not sure that we are using those manifest though.

[MB-8304] Grommit synchronization is not automated

Fri, 17 May 2013 00:11:26 -0500

Currently any project update requires manual checkout at build slaves.

Apart from being annoying it introduces bugs.