Couchbase

[MB-8350] [system test] [windows] rebalance hang at 0% after add node in

Sat, 25 May 2013 00:57:46 -0500

Environment:
9 node windows server 2008 R2 64bit with (8GB RAM and SSD)

Cluster
  Create a 7 nodes cluster with build 2.0.2-812
    default bucket: 3GB RAM, replica index enable
    sasl bucket: 3GB RAM, replica index disable

Load phase:
  Load items (size from 128 to 512 bytes) to both bucket until the active residents go down to 70%
Access phase:
  Run in one day with different load specification to each bucket as in the link without view
    http://hub.internal.couchbase.com/confluence/display/QA/views+%28and+now+with+XDCR%29+tests

See the memory overflow as in bug MB-8340
Continue to run next phase by adding one node (182) and rebalance.
Rebalance hang at 0% for long time.
We decide to stop rebalance then restart again.
Test system add one more node (243) to cluster continue rebalance.
Rebalance still hang without moving any vbucket in sasl bucket.
Cluster is in failed state now

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-812-rel.setup.exe.manifest.xml

Link to collect info file of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_2/2013_05/9nodes_202-812_reb-in-hang-at-0_20130524-161822.tgz

[MB-8328] High latency variation and regression in query throughput

Fri, 24 May 2013 21:18:20 -0500

Build vs. query throughput
-----------------
| 170 | 1357 |
-----------------
| 740 | 1210 |
| 745 | 1338 |
| 755 | 1363 |
-----------------
| 758 | 579 |
| 759 | 571 |
| 760 | 583 |
| 765 | 531 |
| 770 | 597 |
| 790 | 530 |
| 799 | 561 |
| 803 | 587 |
| 807 | 573 |
| 809 | 576 |
---------------

80th and 90th percentiles (that we usually track) are not affected but 99th and 99.9th percentiles are much worse.
Obviously average throughput regressed (>2x).

Diags, stats, atop:
2.0.1-170 - http://172.23.96.10:8080/view/views/job/vesta-views/45/artifact/
2.0.2-755 - http://172.23.96.10:8080/view/views/job/vesta-views/62/artifact/
2.0.2-758 - http://172.23.96.10:8080/view/views/job/vesta-views/60/artifact/
2.0.2-809 - http://172.23.96.10:8080/view/views/job/vesta-views/63/artifact/

Manifests:
http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.2-755-rel.rpm.manifest.xml
http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.2-758-rel.rpm.manifest.xml

Changes in bucket_engine:
2a797a8 ~~MB-6144~~: fixed typo
0992a55 ~~MB-6144~~: Some packages used incorrect offsets

Changes in ns_server:
301435b MB-100: fixed build error due to bad .PHONY and rule interaction
414e304 MB-100: provide api for testrunner for messing with memcached
759135b ~~MB-7398~~: added CHANGES entry for hostname management feature
4ac8e1b CBD-220: run memcached and ns_server from babysitting VM
ec84aad CBD-220: [ale] allow syncing all log messages to all sinks
717902e CBD-220: switched ns_server_sup to supervisor2
68c1e9f MB-100: added typespec to ns_pubsub:subscribe_link
f1032ac ~~MB-7398~~: don't allow hostnames without dot
337c98e MB-100: moved samples_loader_tasks earlier in ns_server_sup list
89661aa MB-5307 Handle failure to save address after rename.
1dd0e9e ~~MB-7398~~ Rename added node to what was specified as its address.
c2b0108 ~~MB-7398~~ Ability to specify server hostname in wizard.
3fc9d9d ~~MB-7398~~ REST call to rename a node.
e6aa76a ~~MB-7398~~ Handle ip_start file in dist_manager.
e3b488a ~~MB-7398~~ Move is_good_address to misc.
531d168 MB-5307 Try to stop net_kernel on dist_manager start.
400c20a MB-5307 Remove references to non-existent path-box class.

[MB-7972] Sever performance dip during rebalancing observed in demo environment

Fri, 24 May 2013 20:55:52 -0500

Environment:
-memcache-test running through client-side Moxi.
-memcache-test workload: /root/mctest/Users/ingenthr/tmp/memcachetest-centos_x86 -h localhost -i 1000 -t 4 -K test -M 10 -F -S -l

-Moxi config: CLI: /opt/moxi/bin/moxi -u nobody -Z usr=Administrator,pwd=password http://10.197.1.147:8091/pools/default/bucketsStreaming/beer-sample -vv -O /var/log/moxi,
config:
port_listen=11211,
default_bucket_name=default,
downstream_max=1024,
downstream_conn_max=4,
downstream_conn_queue_timeout=200,
downstream_timeout=5000,
wait_queue_timeout=200,
connect_max_errors=5,
connect_retry_interval=30000,
connect_timeout=400,
auth_timeout=100,
cycle=200

-4 nodes of Couchbase Server 2.0.1. c1.medium EC2 instances, beer-sample database bucket
-2 nodes of same config added to cluster and rebalanced

Observation:
-Workload very steady around 8k ops/sec before rebalance
-Workload drops to around 4k ops/sec during rebalance. Both gets and sets affected, compaction not running when load initially drops (kicks in eventually). Workload very choppy during rebalance.
-Workload returns to 8k ops/sec after rebalance

Client memcachetest and moxi logs:
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/client1.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/client2.zip

Cluster logs:
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node1.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node2.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node3.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node4.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node5.zip (aded to first 4)
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node6.zip (added to first 4)

[MB-7991] Need generic workload generator

Fri, 24 May 2013 20:17:01 -0500

Key requirements:
1. Easy to install
2. Simple configuration API (similar to memcachetest)
3. High throughput
4. Stable during rebalance

[MB-8066] Observed rebalance regression from 2.0.1 to 2.0.2

Relates to — Fri, 24 May 2013 19:35:50 -0500

Noticed that Rebalance consistently taking longer time in 2.0.2-755 than 2.0.1-170.

Consider the last or last 2 results from each rebalance job on:
http://dashboard.hq.couchbase.com/litmus/dashboard/?env=thor

Also consider the graphs of rebalance jobs attached in http://www.couchbase.com/issues/browse/CBD-914, to note the increase in rebalance times.

Logs and graphs from reb-out-large-2 attached.

[MB-7919] [windows] Rebalance exited with reason {bulk_set_vbucket_state_failed, {timeout, when data mutation is in progress

Fri, 24 May 2013 19:32:45 -0500

Rebalance exited with below reason when data mutation is in progress with rebalance

{{bulk_set_vbucket_state_failed,
                               [{'ns_1@10.142.85.118',
                                 {'EXIT',
                                  {{{timeout,

Test to repro:
nohup ./testrunner -i win-aws.ini -t rebalance.rebalancein.RebalanceInTests.incremental_rebalance_in_with_ops,replicas=2,items=100000,doc_ops=update &

The test failed at 2013-03-15 20:53:08 and the info.18 log file at the orchestrator nodes has the logs. Below crash reports can be seen.

=========================CRASH REPORT=========================
  crasher:
    initial call: ns_single_vbucket_mover:mover/6
    pid: <0.32365.663>
    registered_name: []
    exception exit: {{{timeout,
                          {gen_server,call,
                              [<21714.30742.56>,
                               {start_vbucket_filter_change,


=========================CRASH REPORT=========================
  crasher:
    initial call: ns_single_vbucket_mover:mover/6
    pid: <0.32167.663>
    registered_name: []
    exception exit: {{{timeout,
                          {gen_server,call,
                              [<21714.30742.56>,
                               {start_vbucket_filter_change,

=========================CRASH REPORT=========================
  crasher:
    initial call: ns_single_vbucket_mover:mover/6
    pid: <0.31877.663>
    registered_name: []
    exception exit: {unexpected_exit,
                     {'EXIT',<0.48.664>,
                      {{wait_checkpoint_persisted_failed,"default",853,4,
                        [{'ns_1@10.142.85.118',
                          {'EXIT',
                           {{{timeout,
                              {gen_server,call,
                               [<21714.30742.56>,
                                {start_vbucket_filter_change,


Attaching the testrunner logs and collect_info.zip.

I executed the same test again and it passed so this is not consistently reproducible.

[MB-8341] Per node stats unable to show lazy computed xdcr stats correctly

Fri, 24 May 2013 19:19:11 -0500

Per server stats for Replication progress shows "?".
- Looks like they are not calculated per Server? Attached Screenshot

Steps to repro
1. Setup a 4 node cluster.
2, Load 1M data
3, Setup replication from cluster1 to cluster2.
4. View Outbound XDCR stats " Percentage Complete" for XDC Replicaiton cluster-wide, and then per-node

Results
- Aggregated stats show correct value ~ 30 percent,
- View per node, shows "?" for each node.

[MB-8329] ui doesn't show document content

Fri, 24 May 2013 19:17:54 -0500

earlier documents page shown id and content, now content is omitted

[MB-7959] [Doc'd]: New 2.0.2 XDCR Contin. Optimistic, new XDCR stats, and more XDCR background

Fri, 24 May 2013 19:14:04 -0500

Attached is FINAL draft content for review on XDCR Optimistic Replication, new XDCR stats, updated replication discussion and configuration

Configuring Replication: pages 104-108
Conflict Resolution: page 109
'Optimistic Replication': pages 110-111
Monitoring Outgoing XDCR in Web Console: pages 144-147
Getting XDCR Stats via REST: pages 227-228
Setting 'optimistic replication' via REST: page 227
Mandatory Reviewers: Junyi, Ketaki

Optional Reviewers: PerryK

Bug tracking doc update:http://www.couchbase.com/issues/browse/MB-7959

Review timeframe: *Please provide your review input by EOD Wednesday, May 22 of next week.

*Please add comments directly to the PDF or to the bug link above.

==============

Provide information to document on new parameter for 2.0.2, then assign to Karen to add.

[MB-8302] [Doc'd] READMEs Win, Linux, Mac

Fri, 24 May 2013 19:06:36 -0500

Attached are three README's 1) updated Mac for 2.0.2, 2) new 2.0.2 Linux for Centos and Ubuntu, and 3) new 2.0.2 for Windows.

They include the front-matter from Release Notes 2.0.2, fixes from Jira and any important notes before starting the server.

Here is the ticket where they are also attached: http://www.couchbase.com/issues/browse/MB-8302

per bug triage, we need couchbase installation readme.txt for Windows and Linux OS.

[MB-8277] [Doc'd ]Release Notes 2.0.2

Fri, 24 May 2013 19:06:29 -0500

Attached is first draft content for 2.0.2 Release Notes, pages 334-346.

Mandatory Reviewers: Dipti, Anil

Please provide feedback and updates to me by Monday, May 20 by end of day.

--------------------------

As we did for 2.0.1, I need the intro/front-matter for release notes. Currently:

Couchbase Server 2.0.2 is second maintenance release for
Couchbase Server 2.0. This release contains a number of enhancements
including:

[MB-8065] [Doc'd]: CLI tool: All new settings

Fri, 24 May 2013 19:00:15 -0500

~~MB-7851~~ : CLI tool: Support cluster settings

[MB-8221] [Doc'd] Upgrade, Prerequisites, Initial Install for 2.0.2

Fri, 24 May 2013 19:00:10 -0500

Attached is draft content for review for 2.0.2 prerequisites, install, initial setup and upgrade:

"Preparation" on pages 17-19.
"Installing" on pages 20-23.
"Initial Server Setup: on pages 23-29
"Upgrading to Couchbase Server 2.0.x" on pages 30-38

Mandatory Reviewers: BinC, Tony/Thuan, Chisheng

Optional Reviewers: PerryK, AnilK

Bug tracking doc update:http://www.couchbase.com/issues/browse/MB-8221

Review timeframe: *Please provide your review input by EOD Friday, May 24 of this week.

==============
Review session on 2.0.2 upgrade process: see what overhaul required to existing chapter.

[MB-8061] [Doc'd]: Host name and IP address management in Couchbase

Fri, 24 May 2013 18:59:38 -0500

Attached is new draft content for review on hostname and IP address management for 2.0.2 and updates/fixes to 2.0.1

Using Hostnames with Couchbase Server – Page 29-32
Handling Changes in IP Addresses (Cloud) – Page 59
Mandatory Reviewers: Bin Cui, AliaskeyA, Ketaki/or Delegate

Optional Reviewers: JamesM, PerryK

Bug tracking doc update: http://www.couchbase.com/issues/browse/MB-8061

Review timeframe: *Please provide your review input by EOD Wednesday, May 22 of next week.

*Please add comments directly to the PDF or to the bug link above.
--------------

http://www.couchbase.com/issues/browse/MB-7398

Need to update: http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-bestpractice-cloud-ip.html

AND

http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-getting-started-hostnames.html

[MB-8339] Binary Protocol on Metadata

Fri, 24 May 2013 18:28:38 -0500

http://www.couchbase.com/issues/browse/MB-8217

[MB-8259] Disk not flush on xdcr dest node

Fri, 24 May 2013 17:32:02 -0500

On the ec2 cluster I used for cbrecovery, the disk write queue didn't flush for a very long time. (this symptom was observed twice)

It's an xdcr dest cluster, no other ops rather than xdcr incoming traffic.

Fired as blocker, since it blocks the cbrecovery testing. The clusters are left intact as they are for developer to diagnose, please email back to me when the problem got resolved then I could continue the test. Thanks!

Ronnie

Destination:
http://ec2-184-169-190-197.us-west-1.compute.amazonaws.com:8091

And the source:
http://ec2-23-21-15-103.compute-1.amazonaws.com:8091

User: Administrator
Passwd: password

SSH:
user: root
passwd: couchbase

Now attach the original email thread with diagnosis from Jin and Junyi:

This should be a bug. There are a lot of crash in ns_server/baby_sitter and couchdb/writer process. Couchdb is unable to spawn writer process for some reasons.

I am not 100% sure which caused which but It should be unrelated to XDCR.

Please file a 2.0.2 bug and assign to Filipe to take first look. Thanks.

Junyi

On May 13, 2013, at 10:56 AM, Jin Lim wrote:

Logs from the ec2-184-169-190-197.us-west-1.compute.amazonaws.com. I see following error. After the error I see couch_db/updater kept crashing. I will wait until Junyi's quick looking at it from the xdcr side then figure out what is next.

Thanks,
Jin

[ns_server:info,2013-05-10T18:00:22.665,ns_1@ec2-184-169-190-197.us-west-1.compute.amazonaws.com:ns_config_log<0.562.0>:ns_config_log:handle_info:57]config change: rest_creds -> ********
[user:info,2013-05-10T18:00:22.718,ns_1@ec2-184-169-190-197.us-west-1.compute.amazonaws.com:<0.570.0>:ns_log:crash_consumption_loop:64]Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: WARNING: curl error: transfer closed with outstanding read data remaining from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
EOL on stdin. Exiting
[user:info,2013-05-10T18:00:32.169,ns_1@ec2-184-169-190-197.us-west-1.compute.amazonaws.com:<0.570.0>:ns_log:crash_consumption_loop:64]Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: 2013-05-10 18:00:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2013-05-10 18:00:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)

On May 13, 2013, at 10:41 AM, Ronnie Sun wrote:

Hi Jin, Junyi

On the ec2 cluster I used for cbrecovery, the disk write queue didn't flush for a very long time. (this symptom was observed twice)

It's an xdcr dest cluster, no other ops rather than xdcr incoming traffic.

Everything looks normal, not sure if it's related to the recent reb regression. Would you please take a look at it, thanks!

Ronnie

Destination:
http://ec2-184-169-190-197.us-west-1.compute.amazonaws.com:8091

And the source:
http://ec2-23-21-15-103.compute-1.amazonaws.com:8091

User: Administrator
Passwd: password

SSH:
user: root
passwd: couchbase

[MB-8348] Eviction of doc bodies on compaction should be configurable.

Fri, 24 May 2013 17:14:07 -0500

I've added a command line flag to the compactor, http://review.couchbase.org/#/c/26504/

The compactor is called in couch_db_updater.erl https://github.com/couchbase/couchdb/blob/789b84c1e7ff4f870cf88b5bdad194f1ead619b3/src/couchdb/couch_db_updater.erl#L885

Need a configuration option added to ns_server that I can use to decide whether to pass the flag.

[MB-8340] [system test][windows] memory overflow when bucket has a lot of expired items

Fri, 24 May 2013 17:03:21 -0500

Environment:
7 windows server 2008 r2 64bit with SSD and 8GB RAM

1:10.3.121.173
2:10.3.121.169
3:10.3.121.171
4:10.3.3.214
5:10.3.121.47
6:10.3.3.180
7:10.3.3.181

Cluster setup:
7 nodes cluster
1 default bucket (3GB with replica index enable)
1 sasl bucket (3GB with replica index disable)

Load item to both buckets until active resident ratio down to 70%
Run access phase more than one day with
default bucket: set 5% get 80% expired 5%
sasl bucket: set 5% get 5% delete 20% and expired 80%
After more than one day running, one node in cluster (10.3.121.171) become unstable with erlang crashed
See memory use is used over 3GB over mem quota for couchbase server as show in capture

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-807-rel.setup.exe.manifest.xml

Link to collect info of 6 nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_2/2013_05/7nodes_202-807_nodeDown_erlCrash_20130522-174845.tgz
Due to node 171 is down, I could not get the collect info file
erlang crash dump also attached

Cluster is in failed state now

Impact of this bug. Node 10.3.121.171 is in unstable state. One solution is failover that node, add another node in and rebalance out failed node.

.

[MB-8349] windows buildbot failed to build windows version 2.0.2-813

Fri, 24 May 2013 16:39:13 -0500

buildbot failed to build windows build 2.0.2-813 with error

"collecting /cygdrive/c/erl5.8.5.cb1/usr/lib/icons/hrl_icon.ico"
/usr/lib/ruby/1.8/fileutils.rb:500:in `mv': File exists - /opt/couchbase/lib (Errno::EEXIST)
from /usr/lib/ruby/1.8/fileutils.rb:1402:in `fu_each_src_dest'
from /usr/lib/ruby/1.8/fileutils.rb:1416:in `fu_each_src_dest0'
from /usr/lib/ruby/1.8/fileutils.rb:1400:in `fu_each_src_dest'
from /usr/lib/ruby/1.8/fileutils.rb:495:in `mv'
from ./server-win.rb:309
from ./server-win.rb:308:in `each'
from ./server-win.rb:308
Makefile:201: recipe for target `package-win' failed
make: *** [package-win] Error 1
program finished with exit code 2
elapsedTime=2988.406000

http://builds.hq.northscale.net:8010/builders/cs-win2008-x86-20-builder-202/builds/80

[MB-8306] Number of document mutations pending XDC replication is rapidly growing

Fri, 24 May 2013 15:53:15 -0500

Also time spent waiting for checkpointing is much higher.

Diags (internal IP address):
http://172.23.96.10:8080/view/xdcr/job/xperf-lnx-bi/25/artifact/

[MB-8324] (Data loss)some nodes have 0 active items after offline upgrade

Fri, 24 May 2013 15:42:53 -0500

1) Loaded data into a 9 node cluster at 181.
2) Stopped load, waited for disk-write-queue = 0.
3) shutdown all nodes simultaneously via /etc/init.d/couchbase-server stop
4) upgrade to 2.0.2-807.

Nodes come back up but with less items than before upgrade. 2 nodes have 0 active items.

Impact: apparent data loss
Cluster live ( 10.3.2.89:8091)

all node diags attached.
shutdown occured at timestamp: [2013-05-20 14:37...]
followed by upgrade.

[MB-7735] Memcached crash on a node on the destination cluster

Fri, 24 May 2013 15:41:03 -0500

Live clusters:
Source: http://10.6.2.37:8091/
Destination: http://10.6.2.89:8091/

On Source:
default: ~>70% Resident ratio
saslbucket: ~>40% Resident ratio

System uptime: >36 hours

On Destination:
2 nodes went down: 10.6.2.68, 10.6.2.69

- "ip seems to have changed" seen for both the nodes
- ns_1@10.6.2.68
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_node_disco,nodes_wanted]}}},
{trace,
[{gen_server,call,2},
{ns_orchestrator,needs_rebalance,0},
{ns_cluster_membership,is_balanced,0},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info,2},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] (repeated 15 times)

- ns_1@10.6.2.69
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{{timeout,
{gen_server,call,
[{'stats_reader-default',
'ns_1@10.6.2.69'},
{latest,minute,1}]}},
{gen_server,call,
[menelaus_web_alerts_srv,fetch_alert]}}},
{trace,
[{gen_server,call,2},
{diag_handler,diagnosing_timeouts,1},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info,2},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}]

- Memcached core on 10.6.2.68

The Back trace:

[root@pine-11803 data]# gdb /opt/couchbase/bin/memcached core.memcached.17636
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 17656]
[New Thread 17645]
[New Thread 17654]
[New Thread 17646]
[New Thread 17652]
[New Thread 17653]
[New Thread 17655]
[New Thread 17636]
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x00007fc8afec28a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.1-153.x86_64
(gdb) t a a bt

Thread 8 (Thread 0x7fc8b1376720 (LWP 17636)):
#0 0x00007fc8aff6a43d in write () from /lib64/libc.so.6
#1 0x00007fc8aff01033 in _IO_new_file_write () from /lib64/libc.so.6
#2 0x00007fc8aff00efa in _IO_new_file_xsputn () from /lib64/libc.so.6
#3 0x00007fc8afef692c in fputs () from /lib64/libc.so.6
#4 0x00007fc8aeb61143 in logger_log (severity=EXTENSION_LOG_WARNING, client_cookie=<value optimized out>, fmt=0x7fc8aab1bebb "Schedule cleanup of \"%s\"") at extensions/loggers/file_logger.c:275
#5 0x00007fc8aaad06a4 in TapConnMap::shutdownAllTapConnections (this=0x636e240) at src/tapconnmap.cc:366
#6 0x00007fc8aaa989d1 in EventuallyPersistentEngine::destroy (this=0x6374000, force=<value optimized out>) at src/ep_engine.cc:1401
#7 0x00007fc8aaa98ace in EvpDestroy (handle=<value optimized out>, force=false) at src/ep_engine.cc:130
#8 0x00007fc8adf52bb5 in bucket_shutdown_engine (key=<value optimized out>, nkey=<value optimized out>, val=0x63262a0, nval=<value optimized out>, args=<value optimized out>) at bucket_engine.c:1290
#9 0x00007fc8adf5966c in genhash_iter (h=0x632a000, iterfunc=0x7fc8adf52b80 <bucket_shutdown_engine>, arg=0x0) at genhash.c:275
#10 0x00007fc8adf53f46 in bucket_destroy (handle=0x7fc8ae15c640, force=<value optimized out>) at bucket_engine.c:1327
#11 0x0000000000409777 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7927

Thread 7 (Thread 0x7fc8a8627700 (LWP 17655)):
#0 0x00007fc8b022d7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fc8aaa76f28 in wait (this=0x63a22d0, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x63a22d0, d=...) at src/dispatcher.cc:336
#3 0x00007fc8aaa795ea in Dispatcher::run (this=0x636b880) at src/dispatcher.cc:173
#4 0x00007fc8aaa79eeb in launch_dispatcher_thread (arg=0x636b880) at src/dispatcher.cc:28
#5 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fc8a9a29700 (LWP 17653)):
#0 0x00007fc8b022d7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fc8aaa76f28 in wait (this=0x63a2120, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x63a2120, d=...) at src/dispatcher.cc:336
#3 0x00007fc8aaa795ea in Dispatcher::run (this=0x636ac40) at src/dispatcher.cc:173
#4 0x00007fc8aaa79eeb in launch_dispatcher_thread (arg=0x636ac40) at src/dispatcher.cc:28
#5 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fc8aa638700 (LWP 17652)):
#0 0x00007fc8aff3b97d in nanosleep () from /lib64/libc.so.6
#1 0x00007fc8aff70b34 in usleep () from /lib64/libc.so.6
#2 0x00007fc8aaab67f5 in updateStatsThread (arg=0x1ab44c0) at src/memory_tracker.cc:31
#3 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fc8aeb5d700 (LWP 17646)):
#0 0x00007fc8b022d7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fc8aeb614d6 in logger_thead_main (arg=0x1ab4040) at extensions/loggers/file_logger.c:368
#2 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fc8a9028700 (LWP 17654)):
#0 0x00007fc8b022d7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fc8aaa76f28 in wait (this=0x63a2090, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x63a2090, d=...) at src/dispatcher.cc:336
#3 0x00007fc8aaa795ea in Dispatcher::run (this=0x636aa80) at src/dispatcher.cc:173
#4 0x00007fc8aaa79eeb in launch_dispatcher_thread (arg=0x636aa80) at src/dispatcher.cc:28
#5 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fc8af772700 (LWP 17645)):
#0 0x00007fc8aff6a3dd in read () from /lib64/libc.so.6
#1 0x00007fc8aff01248 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007fc8aff02d4e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007fc8afef753a in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007fc8afef6399 in fgets () from /lib64/libc.so.6
#5 0x00007fc8af773939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#6 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fc8aff776dd in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 1 (Thread 0x7fc8a7c26700 (LWP 17656)):
#0 0x00007fc8afec28a5 in raise () from /lib64/libc.so.6
#1 0x00007fc8afec4085 in abort () from /lib64/libc.so.6
#2 0x0000000000404315 in release_cookie (cookie=<value optimized out>) at daemon/memcached.c:6707
#3 0x00007fc8adf55009 in bucket_engine_release_cookie (cookie=0x62c3b80) at bucket_engine.c:2565
#4 0x00007fc8aaa9461a in EventuallyPersistentEngine::releaseCookie (this=0x6374000, cookie=0x62c3b80) at src/ep_engine.cc:1230
#5 0x00007fc8aaabf376 in TapConnection::releaseReference (this=0x638a000, force=<value optimized out>) at src/tapconnection.cc:110
#6 0x00007fc8aaad2aeb in TapConnectionReaperCallback::callback(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /opt/couchbase/lib/memcached/ep.so
#7 0x00007fc8aaa795ea in Dispatcher::run (this=0x636b6c0) at src/dispatcher.cc:173
#8 0x00007fc8aaa79eeb in launch_dispatcher_thread (arg=0x636b6c0) at src/dispatcher.cc:28
#9 0x00007fc8b0229851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fc8aff776dd in clone () from /lib64/libc.so.6

[MB-8214] bgfetcher performance regression

Fri, 24 May 2013 15:32:40 -0500

* bgfetcher condition variable for woken task is not getting correctly (hasWokenTask).
* also need to immediately move any woken task to readyQueue instead of re-pushing to futureQueue

[MB-8343] In Memory-full scenarios, replication incorrectly reports 100 percent complete status, XDCR queue shows incorrect numbers, XDCR is non-functioning at this point.

Fri, 24 May 2013 15:28:39 -0500

Cluster1
1. Load 24M items on bucket0,
2, Setup unidirecrtional replications to cluster2, 3, 4
3. Run system over 10 hours.
4. Replicated data on Cluster3 and 4 show only 16M out of 24M items.

- Replication progress shows "100 percent complete" for all outbound replication - This is incorrect.
- Replication Queue is around 400-500k ( This is aggregated across all the replication streams) - This is incorrect, it should be at least 8+8 M items in outbound queue.
- Replication errors show " Post Condition error 500" - Can we have more detailed errors here? What is the best way to debug this error code/ message.

Remote Cluster 3, 4 stats
- Memory is nearly fully used up with Active resident ratio - 1 percent on both the clusters.
- Incoming XDCR stats show
- Zero setMeta ( which is expected)
- 5-10K get meta ops/sec

Added screenshot from Source Cluster - cluster1

and cluster3

Adding logs from the clusters.

[MB-8344] UI: Document viewer within view page spins indefinitely when trying to access an item that does not exist

Fri, 24 May 2013 14:24:30 -0500

See attached screenshot. I was able to reproduce this by clicking on an item from a view response that does not exist.

[MB-8266] erlang crashed with error: "no match of right hand side value {error, closed}...

Fri, 24 May 2013 14:10:02 -0500

Environment:
  4 windows vm with 4 core, 4GB RAM each

Couchbase server version 2.0.2-798

Run sanity test with rebalance is part of the test.
See erlang crash in log
Check to see if there is an erlang crash dump in cluster => no core dump.

Error in diags

=========================CRASH REPORT=========================
  crasher:
    initial call: erlang:apply/2
    pid: <0.13944.1>
    registered_name: []
    exception error: no match of right hand side value {error,closed}
      in function mc_binary:quick_stats_recv/3
      in call from mc_binary:quick_stats_loop/5
      in call from mc_binary:quick_stats/5
      in call from ns_memcached:do_handle_call/3
      in call from ns_memcached:worker_loop/3
    ancestors: ['ns_memcached-default','single_bucket_sup-default',
                  <0.13913.1>]
    messages: []
    links: [<0.13929.1>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 17711
    stack_size: 24
    reductions: 8661
  neighbours:

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-798-rel.setup.exe.manifest.xml