Couchbase

[MB-8316] Fix cbtransfer --help output

Sun, 19 May 2013 00:02:06 -0500

Need to Engineering to update descriptions for cbtransfer --h to match with documentation. We just edited these for 2.0.2 int he docs and made them readable/understandable.

Not sure what this release this should be in.

-h, --help Command help
-b BUCKET_SOURCE Single named bucket from source cluster to transfer
-B BUCKET_DESTINATION, --bucket-destination=BUCKET_DESTINATION Single named bucket on destination cluster which receives transfer. This allows you to transfer to a bucket with a different name as your source bucket. If you do not provide defaults to the same name as the bucket-source
-i ID, --id=ID Transfer only items that match a vbucketID
-k KEY, --key=KEY Transfer only items with keys that match a regexp
-n, --dry-run No actual transfer; just validate parameters, files, connectivity and configurations
-u USERNAME, --username=USERNAME REST username for source cluster or server node
-p PASSWORD, --password=PASSWORD REST password for cluster or server node
-t THREADS, --threads=THREADS Number of concurrent workers threads performing the transfer. Defaults to 4.
-v, --verbose Verbose logging; provide more verbosity
-x EXTRA, --extra=EXTRA Provide extra, uncommon config parameters
--single-node Transfer from a single server node in a source cluster. This single server node is a source node URL
--source-vbucket-state=SOURCE_VBUCKET_STATE Only transfer from source vbuckets in this state, such as 'active' (default) or 'replica'. Must be used with Couchbase cluster as source.
--destination-vbucket-state=DESTINATION_VBUCKET_STATE Only transfer to destination vbuckets in this state, such as 'active' (default) or 'replica'. Must be used with Couchbase cluster as destination.
--destination-operation=DESTINATION_OPERATION Perform this operation on transfer. "set" will override an existing document, 'add' will not override, 'get' will load all keys transferred from a source cluster into the caching layer at the destination.
The following are extra, specialized command options you use in this form cbtransfer -x [EXTRA OPTIONS]:
batch_max_bytes=400000 Transfer this # of bytes per batch.
batch_max_size=1000 Transfer this # of documents per batch
cbb_max_mb=100000 Split backup file on destination cluster if it exceeds MB
max_retry=10 Max number of sequential retries if transfer fails
nmv_retry=1 0 or 1, where 1 retries transfer after a NOT_MY_VBUCKET message. Default of 1.
recv_min_bytes=4096 Amount of bytes for every TCP/IP batch transferred
report=5 Number batches transferred before updating progress bar in console
report_full=2000 Number batches transferred before emitting progress information in console
try_xwm=1 As of 2.0.2, transfer documents with metadata. 1 is default. 0 should only be used if you transfer from 1.8.x to 1.8.x.
For 2.0.2
   <row>
   <entry>data_only=0</entry>
   <entry>For value 1, only transfer data from a backup file or cluster.</entry>
   </row>
   <row>
   <entry>design_doc_only=0</entry>
   <entry>For value 1, transfer design documents only from a backup file or cluster. Defaults to 0.</entry>
   </row>
   For 2.0.2

[MB-8315] Need Decision on cbtransfer, recv_min_bytes=4096

Sun, 19 May 2013 00:02:21 -0500

It is a batch side for TCP/IP used in the transfer. Not sure if it should be public/tunable or not......

Public for now.

[MB-8314] rebalance exited ns_vbucket_mover failed to initiate_indexing

Sun, 19 May 2013 00:00:16 -0500

Have a 5 node cluster that was upgraded from 181 -> 202 (build 805).
After upgrade attempted to swap out orchestrator(10.3.121.69) and add in a new 202 node (10.3.3.131)

Rebalance fails with ns_vbucket_mover000 on orchestrator reporting:
<0.11037.1> exited with {noproc,
{gen_server,call,
[{'janitor_agent-saslbucket',
'ns_1@10.3.3.131'},
{if_rebalance,<0.1.1>,initiate_indexing},
infinity]}}

On new node 202, couchdb went down and erlang crash dump was generated(attached).

Saw these errors from mccouch which may be why vbuckets couldn't be moved to this node:
Fri May 17 07:27:36.981472 PDT 3: (saslbucket) Trying to connect to mccouch: "127.0.0.1:11213"
Fri May 17 07:27:36.981615 PDT 3: (saslbucket) Connected to mccouch: "127.0.0.1:11213"
Fri May 17 07:27:37.019496 PDT 3: (saslbucket) Connection closed by mccouch
Fri May 17 07:27:37.019527 PDT 3: (saslbucket) Resetting connection to mccouch, lastReceivedCommand = notify_vbucket_update lastSentCommand = notify_vbucket_update currentCommand =unknown
Fri May 17 07:27:37.019595 PDT 3: (saslbucket) Trying to connect to mccouch: "127.0.0.1:11213"
Fri May 17 07:27:37.019730 PDT 3: (saslbucket) Connected to mccouch: "127.0.0.1:11213"
Fri May 17 07:27:37.021763 PDT 3: (saslbucket) Connection closed by mccouch
Fri May 17 07:27:37.021788 PDT 3: (saslbucket) Resetting connection to mccouch, lastReceivedCommand = select_bucket lastSentCommand = notify_vbucket_update currentCommand =unknown

=========================CRASH REPORT=========================
  crasher:
    initial call: mc_connection:init/1
    pid: <0.965.0>
    registered_name: []
    exception error: no case clause matching {error,system_limit}
      in function mc_connection:do_notify_vbucket_update/3
      in call from mc_connection:handle_message/9
      in call from mc_connection:read_full_message/2
      in call from mc_connection:run_loop/2
    ancestors: [mc_conn_sup,mc_sup,ns_server_sup,ns_server_cluster_sup,
                  <0.59.0>]
    messages: []
    links: [<0.641.0>,#Port<0.6784>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 1094838
  neighbours:

[error_logger:error,2013-05-17T7:27:36.622,ns_1@10.3.3.131:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
     Supervisor: {local,mc_conn_sup}
     Context: child_terminated
     Reason: {case_clause,{error,system_limit}}
     Offender: [{pid,<0.965.0>},
                  {name,mc_connection},
                  {mfargs,{mc_connection,start_link,undefined}},
                  {restart_type,temporary},
                  {shutdown,brutal_kill},
                  {child_type,worker}]

=========================CRASH REPORT=========================
  crasher:
    initial call: couch_file:spawn_reader/2
    pid: <0.652.0>
    registered_name: []
    exception exit: {problem_reopening_file,
                        {error,system_limit},
                        {set_close_after,infinity,<0.650.0>},
                        <0.652.0>,
                        "/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1",
                        10}
      in function couch_file:reader_loop/3
    ancestors: [<0.650.0>,couch_server,couch_primary_services,
                  couch_server_sup,cb_couch_sup,ns_server_cluster_sup,
                  <0.59.0>]
    messages: []
    links: [<0.650.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 377
    stack_size: 24
    reductions: 504
  neighbours:

[MB-8313] Table 7.4. Administration — Standard couchbase Tool Options should be updated with new options

Sun, 19 May 2013 00:00:56 -0500

http://www.couchbase.com/docs/couchbase-manual-2.0.pdf

Table 7.4. Administration — Standard couchbase Tool Options doesn't contains the full list of available options.

Actual info we can get from from './couchbase-cli -h":

CLUSTER:
  --cluster=HOST[:PORT] or -c HOST[:PORT]

OPTIONS:
  -u USERNAME, --user=USERNAME admin username of the cluster
  -p PASSWORD, --password=PASSWORD admin password of the cluster
  -o KIND, --output=KIND KIND is json or standard
  -d, --debug

server-add OPTIONS:
  --server-add=HOST[:PORT] server to be added
  --server-add-username=USERNAME admin username for the
                                    server to be added
  --server-add-password=PASSWORD admin password for the
                                    server to be added

server-readd OPTIONS:
  --server-add=HOST[:PORT] server to be added
  --server-add-username=USERNAME admin username for the
                                    server to be added
  --server-add-password=PASSWORD admin password for the
                                    server to be added

rebalance OPTIONS:
  --server-add* see server-add OPTIONS
  --server-remove=HOST[:PORT] the server to be removed

failover OPTIONS:
  --server-failover=HOST[:PORT] server to failover

cluster-* OPTIONS:
  --cluster-username=USER new admin username
  --cluster-password=PASSWORD new admin password
  --cluster-port=PORT new cluster REST/http port
  --cluster-ramsize=RAMSIZEMB per node ram quota in MB

node-init OPTIONS:
  --node-init-data-path=PATH per node path to store data
  --node-init-index-path=PATH per node path to store index

bucket-* OPTIONS:
  --bucket=BUCKETNAME bucket to act on
  --bucket-type=TYPE memcached or couchbase
  --bucket-port=PORT supports ASCII protocol and is auth-less
  --bucket-password=PASSWORD standard port, exclusive with bucket-port
  --bucket-ramsize=RAMSIZEMB ram quota in MB
  --bucket-replica=COUNT replication count
  --enable-flush=[0|1] enable/disable flush
  --enable-index-replica=[0|1] enable/disable index replicas
  --wait wait for bucket create to be complete before returning
  --force force to execute command without asking for confirmation
  --data-only compact datbase data only
  --view-only compact view data only

setting-compacttion OPTIONS:
  --compaction-db-percentage=PERCENTAGE at which point database compaction is triggered
  --compaction-db-size=SIZE[MB] at which point database compaction is triggered
  --compaction-view-percentage=PERCENTAGE at which point view compaction is triggered
  --compaction-view-size=SIZE[MB] at which point view compaction is triggered
  --compaction-period-from=HH:MM allow compaction time period from
  --compaction-period-to=HH:MM allow compaction time period to
  --enable-compaction-abort=[0|1] allow compaction abort when time expires
  --enable-compaction-parallel=[0|1] allow parallel compaction for database and view

setting-notification OPTIONS:
  --enable-notification=[0|1] allow notification

setting-alert OPTIONS:
  --enable-email-alert=[0|1] allow email alert
  --email-recipients=RECIPIENT email recipents, separate addresses with , or ;
  --email-sender=SENDER sender email address
  --email-user=USER email server username
  --email-password=PWD email server password
  --email-host=HOST email server host
  --email-port=PORT email server port
  --enable-email-encrypt=[0|1] email encrypt
  --alert-auto-failover-node node was auto failover
  --alert-auto-failover-max-reached maximum number of auto failover nodes was reached
  --alert-auto-failover-node-down node wasn't auto failover as other nodes are down at the same time
  --alert-auto-failover-cluster-small node wasn't auto fail over as cluster was too small
  --alert-ip-changed node ip address has changed unexpectedly
  --alert-disk-space disk space used for persistent storgage has reached at least 90% capacity
  --alert-meta-overhead metadata overhead is more than 50%
  --alert-meta-oom bucket memory on a node is entirely used for metadata
  --alert-write-failed writing data to disk for a specific bucket has failed

setting-autofailover OPTIONS:
  --enable-auto-failover=[0|1] allow auto failover
  --auto-failover-timeout=TIMEOUT (>=30) specify timeout that expires to trigger auto failover

setting-xdcr OPTIONS:
  --max-concurrent-reps=[32] maximum concurrent replications per bucket, 8 to 256.
  --checkpoint-interval=[1800] intervals between checkpoints, 60 to 14400 seconds.
  --worker-batch-size=[500] doc batch size, 500 to 10000.
  --doc-batch-size=[2048]KB document batching size, 10 to 100000 KB
  --failure-restart-interval=[30] interval for restarting failed xdcr, 1 to 300 seconds
  --optimistic-replication-threshold=[256] document body size threshold (bytes) to trigger optimistic replication

xdcr-setup OPTIONS:
  --create create a new xdcr configuration
  --edit modify existed xdcr configuration
  --delete delete existed xdcr configuration
  --xdcr-cluster-name=CLUSTERNAME cluster name
  --xdcr-hostname=HOSTNAME remote host name to connect to
  --xdcr-username=USERNAME remote cluster admin username
  --xdcr-password=PASSWORD remtoe cluster admin password

xdcr-replicate OPTIONS:
  --create create and start a new replication
  --delete stop and cancel a replication
  --xdcr-from-bucket=BUCKET local bucket name to replicate from
  --xdcr-clucter-name=CLUSTERNAME remote cluster to replicate to
  --xdcr-to-bucket=BUCKETNAME remote bucket to replicate to

[MB-8312] we don't have Table 7.3. and Table 7.4. (clouhbase-cli stuff)in online documentation

Fri, 17 May 2013 14:34:42 -0500

I see that they are presented in http://www.couchbase.com/docs/couchbase-manual-2.0.pdf
Table 7.3. Administration — couchbase Tool Commands
Table 7.4. Administration — Standard couchbase Tool Options

but missed in online documentation for '7.4. couchbase-cli Tool' in http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-cmdline-couchbase-cli.html

[MB-8311] Table 7.3. Administration — couchbase Tool Commands should be updated with new features

Fri, 17 May 2013 14:36:07 -0500

according to http://www.couchbase.com/docs/couchbase-manual-2.0.pdf

Table 7.3. Administration — couchbase Tool Commands
Command Description
server-list List all servers in a cluster
server-info Show details on one server
server-add Add one or more servers to the cluster
server-readd Re-add a server that was failed over to the cluster
rebalance Start a cluster rebalancingCommand-line Interface for Administration
169
Command Description
rebalance-stop Stop current cluster rebalancing
rebalance-status Show status of current cluster rebalancing
failover Failover one or more servers
cluster-init Set the username, password and port of the cluster
node-init Set node specific parameters
bucket-list List all buckets in a cluster
bucket-create Add a new bucket to the cluster
bucket-edit Modify an existing bucket
bucket-delete Delete an existing bucket
bucket-flush Flush a given bucket
help Show longer usage/help and examples

but the full list should be for 2.0.2 as:

./couchbase-cli
usage: couchbase-cli COMMAND CLUSTER [OPTIONS]

CLUSTER is --cluster=HOST[:PORT] or -c HOST[:PORT]

COMMANDs include
  server-list list all servers in a cluster
  server-info show details on one server
  server-add add one or more servers to the cluster
  server-readd readd a server that was failed over
  rebalance start a cluster rebalancing
  rebalance-stop stop current cluster rebalancing
  rebalance-status show status of current cluster rebalancing
  failover failover one or more servers
  cluster-init set the username,password and port of the cluster
  cluster-edit modify cluster settings
  node-init set node specific parameters
  bucket-list list all buckets in a cluster
  bucket-create add a new bucket to the cluster
  bucket-edit modify an existing bucket
  bucket-delete delete an existing bucket
  bucket-flush flush all data from disk for a given bucket
  bucket-compact compact database and index data
  setting-compaction set auto compaction settings
  setting-notification set notification settings
  setting-alert set email alert settings
  setting-autofailover set auto failover settings
  setting-xdcr set xdcr related settings
  xdcr-setup set up XDCR connection
  xdcr-replicate xdcr operations
  help show longer usage/help and examples

[MB-8310] Couchdb gerrit changes for master must use manifest 2.1-unstable.xml

Fri, 17 May 2013 06:28:32 -0500

2.1-stable.xml was abandoned, and any couchdb master gerrit change depends on couchstore revision more recent than the one listed in 2.1-stable.xml - this means the jobs couchdb-gerrit-views-master and couchdb-gerrit-views-pre-merge-master always fail.

[MB-8309] online_upgrade_swap_rebalance 1.8.1->2.0.2: Rebalance exited with reason {{change_filter_failed,{'EXIT',{{unexpected_reason,killed}

Fri, 17 May 2013 17:30:13 -0500

http://qa.hq.northscale.net/job/centos-32-2.0-upgrade-P1/4/consoleFull

./testrunner -i /tmp/centos-32-2.0-upgrade.ini upgrade_version=2.0.2-805-rel,get-cbcollect-info=True,GROUP=P1 -t newupgradetests.MultiNodesUpgradeTests.online_upgrade_swap_rebalance,initial_version=1.8.1-942-rel,standard_buckets=1,items=500000,max_verify=1000,GROUP=1_8;ONLINE;WINDOWS;P1

steps:
1)10.3.3.151, 10.3.3.152 with 1.8.1 build
10.3.3.153 with 2.0.2-805-rel

2)[2013-05-17 01:11:42,198] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.151&user=Administrator&knownNodes=ns_1%4010.3.3.152%2Cns_1%4010.3.3.151%2Cns_1%4010.3.3.153
...
[2013-05-17 01:26:15,908] - [task:340] INFO - rebalancing was completed with progress: 100.0% in 873.699863911 sec

3)[2013-05-17 01:26:37,009] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.152&user=Administrator&knownNodes=ns_1%4010.3.3.152%2Cns_1%4010.3.3.153
...

[2013-05-17 01:31:07,549] - [rest_client:1014] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2013-05-17 01:31:07,550] - [rest_client:1015] INFO - Latest logs from UI:
[2013-05-17 01:31:07,601] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 2, u'text': u"Rebalance exited with reason {{change_filter_failed,\n {'EXIT',\n {{unexpected_reason,killed},\n [{misc,executing_on_new_process,1},\n {ns_vbm_sup,local_change_vbucket_filter,4},\n {rpc,local_call,3},\n {ns_vbm_sup,change_vbucket_filter,4},\n {ns_vbm_sup,'-set_replicas/3-fun-2-',5},\n {lists,foldl,3},\n {ns_vbm_sup,set_replicas,3},\n {ns_vbm_sup,\n '-set_replicas_on_nodes/3-fun-1-',3}]}}},\n [{ns_vbm_sup,change_vbucket_filter,4},\n {ns_vbm_sup,'-set_replicas/3-fun-2-',5},\n {lists,foldl,3},\n {ns_vbm_sup,set_replicas,3},\n {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',\n 3},\n {lists,foreach,2},\n {janitor_agent,\n do_bulk_set_vbucket_state_old_style,4},\n {ns_vbucket_mover,\n update_replication_post_move,3}]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368779444497.0, u'type': u'info'}
[2013-05-17 01:31:07,602] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 43 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779418787.0, u'type': u'critical'}
[2013-05-17 01:31:07,603] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779358861.0, u'type': u'critical'}
[2013-05-17 01:31:07,603] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 179 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779358787.0, u'type': u'critical'}
[2013-05-17 01:31:07,604] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779299138.0, u'type': u'critical'}
[2013-05-17 01:31:07,604] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 154 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779298786.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779238798.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 132 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779238787.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779184396.0, u'type': u'critical'}
[2013-05-17 01:31:07,605] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Bucket "standard_bucket0" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368779184144.0, u'type': u'info'}

[MB-8308] Rebalance time regression in large scale DGM vperf tests

Fri, 17 May 2013 05:19:32 -0500

For instance, with aggressive auto-compaction setting it's > 2x slower.

Need to summarize and post all results.

[MB-8307] Litmus dashboard becomes unresponsive after fetching big set of results

Fri, 17 May 2013 14:27:12 -0500

I can see that it eats one of my cores for a couple of seconds.

Firefox tries to kill the script from time to time.

[MB-8306] Number of document mutations pending XDC replication is rapidly growing

Fri, 17 May 2013 15:54:16 -0500

Also time spent waiting for checkpointing is much higher.

It seems to be 2.0.2 regression.

Diags (internal IP address):
http://172.23.96.10:8080/view/xdcr/job/xperf-lnx-bi/25/artifact/

PDF report (internal IP address):
http://172.23.96.10:8080/view/graphs/job/graph-loop/159/artifact/xperf-mixed-bi.loop_2.0.1-170-rel-enterprise_2.0.2-802-rel-enterprise_DEST_May-16-2013_22%3A42%3A54.pdf

[MB-8305] 2.1 manifests have reference to membase-cli

Fri, 17 May 2013 00:16:05 -0500

membase/membase-cli instead of couchbase/couchbase-cli

For instance:
https://github.com/couchbase/manifest/blob/master/2.1-stable.xml

I'm not sure that we are using those manifest though.

[MB-8304] Grommit synchronization is not automated

Fri, 17 May 2013 00:11:26 -0500

Currently any project update requires manual checkout at build slaves.

Apart from being annoying it introduces bugs.

[MB-8303] [system test] Rebalance in hangs with some initial vbucket movement due to bucket state bouncing between not ready and active

Thu, 16 May 2013 18:57:57 -0500

Cluster ip is 172.23.105.23
1. create 8 nodes cluster, each node has 12G RAM, HHD
2. create 2 buckets default and saslbucket, with memory quota 6G and 4G
3. Run the KV use case for 2 days:
loading 35M items to each bucket, make resident ratio 70%~80%, access the data 15k ops/sec with 5% create, 5% delete, 5%expire, 5% update, 80 gets for several hours.
4. Then with the same work load, run some rebalance operations. When proceed to rebalance in 2 nodes, after several vbucket movement (17 vbucket for saslbucket), rebalance hangs.

When check the log, see a lot of bucket state change between not ready and active:

root@cola-s10305:/opt/couchbase/var/lib/couchbase/logs# tail -f error.1
<<"replication_building_1013_'ns_1@172.23.105.33'">>} took too long: 1196997 us
[ns_server:error,2013-05-16T15:13:17.111,ns_1@172.23.105.32:<0.11769.0>:ns_memcached:verify_report_long_call:294]call {stats,<<>>} took too long: 556521 us
[ns_server:error,2013-05-16T15:19:19.834,ns_1@172.23.105.32:<0.11769.0>:ns_memcached:verify_report_long_call:294]call {get_tap_docs_estimate,1013,
<<"replication_building_1013_'ns_1@172.23.105.33'">>} took too long: 725657 us
[ns_server:error,2013-05-16T15:22:18.618,ns_1@172.23.105.32:ns_memcached-saslbucket<0.11752.0>:ns_memcached:handle_info:671]handle_info(ensure_bucket,..) took too long: 661061 us
[ns_server:error,2013-05-16T15:23:35.127,ns_1@172.23.105.32:<0.11704.0>:ns_memcached:verify_report_long_call:294]call {stats,<<>>} took too long: 568954 us
[ns_server:error,2013-05-16T15:25:59.291,ns_1@172.23.105.32:<0.11705.0>:ns_memcached:verify_report_long_call:294]call topkeys took too long: 1217646 us
[ns_server:error,2013-05-16T15:30:17.553,ns_1@172.23.105.32:ns_doctor<0.9998.0>:ns_doctor:update_status:234]The following buckets became not ready on node 'ns_1@172.23.105.23': ["saslbucket"], those of them are active ["saslbucket"]
[ns_server:error,2013-05-16T15:30:37.540,ns_1@172.23.105.32:ns_doctor<0.9998.0>:ns_doctor:update_status:234]The following buckets became not ready on node 'ns_1@172.23.105.23': ["saslbucket"], those of them are active ["saslbucket"]

diag links:
https://s3.amazonaws.com/bugdb/jira/MB-8303/8nodes_202-800_rebalance_hang_20130516-143321.tgz

[MB-8302] create couchbase readme.txt file for Windows and Linux

Thu, 16 May 2013 17:06:12 -0500

per bug triage, we need couchbase installation readme.txt for Windows and Linux OS.

[MB-8301] XDCR core infrastructure change to enable replicate to remote memcached

Thu, 16 May 2013 16:24:57 -0500

XDCR core infrastructure change to enable replicate to remote memcached

[MB-8300] Implement ns_server side memcached API for new get_meta_batch and update_meta_batch

Thu, 16 May 2013 16:16:42 -0500

In MB-8213, ep_engine team will create two new batch operations, get_meta_batch and update_meta_batch, which will be built on new protocol. From ns_server side, we need to implement new memcached APIs correspondingly.

[MB-8299] remote_cluster_info module should return remote memcached access info

Thu, 16 May 2013 16:10:42 -0500

In 2.1, XDCR will provide an option to replicate to remote memcached instead of CAPI.

This requires XDCR core infrastructure to understand how to access memcached at remote cluster.

For example, in order to replicate remote bucket "default", vbucket 123, XDCR need to know

1) node (ip address) for this vbucket at remote cluster;
2) memcached port of the node where the vbucket lives;
3) credentials to access the memcached at that node

Module remote_cluster_info is a natural home for such information.

Today 1) has been already encoded in the vbucketmap maintained by remote_cluster_info, but 2) and 3) are not available. This task will expand remote_cluster_info module to include 2) and 3).

In particular, XDCR need an API to return 1), 2), 3) above, e.g.,

remote_cluster_info:fetch_remote_memcached_info(RemoteClusterId, Bucket, VBucket)

returns
{"10.3.114.2", 11998, "_admin", "_password"}

[MB-8298] remote_cluster_info module should return remote memcached access info

Thu, 16 May 2013 16:11:24 -0500

[MB-8297] Some key projects are still hosted at Membase GitHub account

Thu, 16 May 2013 13:16:10 -0500

memcached, libmemcached, grommit, buildbot-internal...

They are important components of build workflow. For instance, repo manifests have multiple references to these projects.

This is very confusing legacy, I believe we can avoid it.

[MB-8296] win32: rebalance failed after Port server memcached on node \'babysitter_of_ns_1@127.0.0.1\' exited with status 255

Thu, 16 May 2013 17:19:29 -0500

http://qa.hq.northscale.net/job/windows32_rebalance-kv/9/consoleFull

./testrunner -i /tmp/4-w-32.ini get-cbcollect-info=True -t rebalancetests.IncrementalRebalanceOut.test_load,replica=1,delete-ratio=0,expiry-ratio=0

password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.2.179%2Cns_1%4010.3.2.185%2Cns_1%4010.3.2.178%2Cns_1%4010.3.2.183
[2013-05-16 04:40:53,799] - [rest_client:929] INFO - rebalance operation started
[2013-05-16 04:40:53,991] - [rest_client:1031] INFO - rebalance percentage : 0 %
[2013-05-16 04:40:55,037] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:40:57,191] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:40:58,428] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:40:59,674] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:00,790] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:02,902] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:04,931] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:06,048] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:08,102] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:10,423] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
[2013-05-16 04:41:11,575] - [rest_client:1014] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2013-05-16 04:41:11,575] - [rest_client:1015] INFO - Latest logs from UI:
[2013-05-16 04:41:11,736] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.178', u'code': 2, u'text': u"Rebalance exited with reason {bulk_set_vbucket_state_failed,\n [{'ns_1@10.3.2.179',\n {'EXIT',\n {{badmatch,wrong_rebalancer_pid},\n [{janitor_agent,set_vbucket_state,7},\n {janitor_agent,\n '-bulk_set_vbucket_state/4-fun-0-',4},\n {misc,'-parallel_map/3-fun-0-',4}]}}}]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368704449014, u'type': u'info'}
[2013-05-16 04:41:11,736] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.178', u'code': 0, u'text': u"<0.8010.5> exited with {bulk_set_vbucket_state_failed,\n [{'ns_1@10.3.2.179',\n {'EXIT',\n {{badmatch,wrong_rebalancer_pid},\n [{janitor_agent,set_vbucket_state,7},\n {janitor_agent,\n '-bulk_set_vbucket_state/4-fun-0-',4},\n {misc,'-parallel_map/3-fun-0-',4}]}}}]}", u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368704446155, u'type': u'critical'}
[2013-05-16 04:41:11,736] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.179', u'code': 0, u'text': u'Port server memcached on node \'babysitter_of_ns_1@127.0.0.1\' exited with status 255. Restarting. Messages: Thu May 16 04:29:32.201380 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Schedule the backfill for vbucket 427\nThu May 16 04:29:32.201380 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nThu May 16 04:29:32.208215 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nThu May 16 04:29:32.213098 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 427\nThu May 16 04:29:32.222863 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Backfill is completed with VBuckets 427, \nThu May 16 04:29:32.222863 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 427\nThu May 16 04:29:32.267782 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.183\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nThu May 16 04:29:32.267782 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.183\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nThu May 16 04:29:32.315630 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:32.553896 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - disconnected, keep alive for 300 seconds\nThu May 16 04:29:32.565614 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Connection is closed by force\nThu May 16 04:29:32.693536 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nThu May 16 04:29:32.693536 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nThu May 16 04:29:32.693536 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - Sending TAP_VBUCKET_SET with vbucket 427 and state "pending"\nThu May 16 04:29:32.694512 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1608"\nThu May 16 04:29:32.697442 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.185\' - Clear the tap queues by force\nThu May 16 04:29:32.705254 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 774, cookie 044D50E0\nThu May 16 04:29:32.708183 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - VBucket <427> is going dead to complete vbucket takeover\nThu May 16 04:29:32.715995 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - Sending TAP_VBUCKET_SET with vbucket 427 and state "active"\nThu May 16 04:29:32.719901 Pacific Daylight Time 3: (bucket-0) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_427>\nThu May 16 04:29:32.720878 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - disconnected\nThu May 16 04:29:32.723807 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:rebalance_427"\nThu May 16 04:29:32.724784 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:rebalance_427 - Clear the tap queues by force\nThu May 16 04:29:32.742361 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.183\' - disconnected, keep alive for 300 seconds\nThu May 16 04:29:32.751149 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.183\' - Connection is closed by force\nThu May 16 04:29:32.817551 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1609 - disconnected\nThu May 16 04:29:32.983556 Pacific Daylight Time 3: (bucket-0) Deletion of vbucket 427 was completed.\nThu May 16 04:29:33.097807 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1612 - disconnected\nThu May 16 04:29:33.214010 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1610"\nThu May 16 04:29:33.214010 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1609"\nThu May 16 04:29:33.214010 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_building_427_\'ns_1@10.3.2.183\' - Clear the tap queues by force\nThu May 16 04:29:33.214010 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:33.214010 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1612"\nThu May 16 04:29:33.335096 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1611 - disconnected\nThu May 16 04:29:33.348767 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1611"\nThu May 16 04:29:33.401498 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 772, cookie 044D4960\nThu May 16 04:29:33.546020 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1614 - disconnected\nThu May 16 04:29:33.734485 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1613 - disconnected\nThu May 16 04:29:33.883889 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1613"\nThu May 16 04:29:33.883889 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1614"\nThu May 16 04:29:33.883889 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:33.900490 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1615 - disconnected\nThu May 16 04:29:34.011811 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1615"\nThu May 16 04:29:34.022552 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 771, cookie 044D4D20\nThu May 16 04:29:34.139732 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:34.308667 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1616 - disconnected\nThu May 16 04:29:34.412176 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1616"\nThu May 16 04:29:34.445377 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 770, cookie 044D5860\nThu May 16 04:29:34.524473 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1618 - disconnected\nThu May 16 04:29:34.547909 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1617 - disconnected\nThu May 16 04:29:34.793011 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1617"\nThu May 16 04:29:34.793011 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1618"\nThu May 16 04:29:34.793011 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:34.805705 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 769, cookie 044D5860\nThu May 16 04:29:34.942415 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1619 - disconnected\nThu May 16 04:29:34.976593 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1619"\nThu May 16 04:29:35.003935 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1620 - disconnected\nThu May 16 04:29:35.162128 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1621 - disconnected\nThu May 16 04:29:35.178728 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1620"\nThu May 16 04:29:35.178728 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1621"\nThu May 16 04:29:35.178728 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:35.595694 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:31:37.747055 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:31:37.747055 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=234 rev=2\nThu May 16 04:34:13.673599 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:13.673599 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=269 rev=4\nThu May 16 04:34:27.262573 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:27.262573 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=702 rev=3\nThu May 16 04:34:37.990402 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:37.990402 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=816 rev=4\nThu May 16 04:34:41.872966 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:41.872966 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=865 rev=4\nThu May 16 04:40:07.491773 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - disconnected, keep alive for 300 seconds\nThu May 16 04:40:07.538645 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1027 - disconnected\nThu May 16 04:40:07.594306 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1027"\nThu May 16 04:40:07.600165 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_4 - disconnected\nThu May 16 04:40:07.602118 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_18 - disconnected\nThu May 16 04:40:07.623601 Pacific Daylight Time 3: (bucket-0) Shutting down tap connections!\nThu May 16 04:40:07.623601 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_4"\nThu May 16 04:40:07.624577 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.183"\nThu May 16 04:40:07.625554 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.178"\nThu May 16 04:40:07.625554 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.183 - Clear the tap queues by force\nThu May 16 04:40:07.625554 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_18"\nThu May 16 04:40:07.626530 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.185"\nThu May 16 04:40:07.626530 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.178 - Clear the tap queues by force\nThu May 16 04:40:07.627507 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Clear the tap queues by force\nThu May 16 04:40:33.087881 Pacific Daylight Time 3: (bucket-0) Warning: failed to load the engine session stats due to IO exception "basic_ios::clear"\nThu May 16 04:40:33.087881 Pacific Daylight Time 3: (bucket-0) Failed to load mutation log, falling back to key dump\nThu May 16 04:40:33.087881 Pacific Daylight Time 3: Extension support isn\'t implemented in this version of bucket_engine\nThu May 16 04:40:33.088858 Pacific Daylight Time 3: (bucket-0) metadata loaded in 1236 usec\nThu May 16 04:40:33.089834 Pacific Daylight Time 3: (bucket-0) warmup completed in 1709 usec\nThu May 16 04:40:35.654123 Pacific Daylight Time 3: (bucket-0) Connection closed by mccouch\nThu May 16 04:40:35.654123 Pacific Daylight Time 3: (bucket-0) Resetting connection to mccouch, lastReceivedCommand = notify_vbucket_update lastSentCommand = notify_vbucket_update currentCommand =notify_vbucket_update\nMUTEX ERROR: Failed to acquire lock: Invalid argument\n\nThis application has requested the Runtime to terminate it in an unusual way.\nPlease contact the application\'s support team for more information.\nThu May 16 04:40:36.013475 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1622 - Reset vbucket 938 was completed succecssfully.\nThu May 16 04:40:36.444112 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1623 - Reset vbucket 682 was completed succecssfully.\nThu May 16 04:40:43.419251 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1624 - Reset vbucket 937 was completed succecssfully.', u'shortText': u'message', u'module': u'ns_log', u'tstamp': 1368704444183, u'type': u'info'}
[2013-05-16 04:41:11,737] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.179', u'code': 4, u'text': u"Control connection to memcached on 'ns_1@10.3.2.179' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368704444167, u'type': u'info'}
[2013-05-16 04:41:11,737] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.183', u'code': 0, u'text': u'Port server memcached on node \'babysitter_of_ns_1@127.0.0.1\' exited with status 255. Restarting. Messages: Thu May 16 04:29:15.442145 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3313 - disconnected\nThu May 16 04:29:15.624751 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 87, cookie 044041E0\nThu May 16 04:29:15.669670 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3317 - disconnected\nThu May 16 04:29:15.862040 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3316 - disconnected\nThu May 16 04:29:16.100306 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3320 - disconnected\nThu May 16 04:29:16.126672 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3313"\nThu May 16 04:29:16.127648 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3316"\nThu May 16 04:29:16.128625 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:16.133507 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3317"\nThu May 16 04:29:16.135460 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3320"\nThu May 16 04:29:16.220416 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 86, cookie 04405860\nThu May 16 04:29:16.252640 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3315 - disconnected\nThu May 16 04:29:17.141255 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3315"\nThu May 16 04:29:17.169574 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 433, cookie 04404B40\nThu May 16 04:29:17.269177 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:17.290660 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3319 - disconnected\nThu May 16 04:29:17.524043 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3319"\nThu May 16 04:29:17.652941 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3321 - disconnected\nThu May 16 04:29:17.821876 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3318 - disconnected\nThu May 16 04:29:18.312079 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3318"\nThu May 16 04:29:18.312079 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3321"\nThu May 16 04:29:18.312079 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:19.443842 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 432, cookie 04404B40\nThu May 16 04:29:19.913539 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 85, cookie 044052C0\nThu May 16 04:29:19.920374 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3322 - disconnected\nThu May 16 04:29:19.941857 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 431, cookie 04405680\nThu May 16 04:29:20.106886 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3323 - disconnected\nThu May 16 04:29:20.277773 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3325 - disconnected\nThu May 16 04:29:20.415460 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3322"\nThu May 16 04:29:20.416436 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3323"\nThu May 16 04:29:20.416436 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3325"\nThu May 16 04:29:20.416436 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0\nThu May 16 04:29:20.758211 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3324 - disconnected\nThu May 16 04:29:21.398795 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3326 - disconnected\nThu May 16 04:29:21.637061 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3324"\nThu May 16 04:29:21.637061 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3326"\nThu May 16 04:29:21.936847 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3327 - disconnected\nThu May 16 04:29:22.640903 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3327"\nThu May 16 04:29:23.123294 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3328 - disconnected\nThu May 16 04:29:23.646698 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3328"\nThu May 16 04:29:24.432781 Pacific Daylight Time 3: (bucket-0) Notified the completion of checkpoint persistence for vbucket 427, cookie 04405860\nThu May 16 04:29:24.720848 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3329 - disconnected\nThu May 16 04:29:25.651453 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3329"\nThu May 16 04:31:32.042777 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:31:32.042777 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=70 rev=3\nThu May 16 04:32:06.188053 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:32:06.188053 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=345 rev=3\nThu May 16 04:32:14.777347 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:32:14.777347 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=391 rev=3\nThu May 16 04:32:32.463715 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:32:32.463715 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=504 rev=3\nThu May 16 04:32:43.120259 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:32:43.120259 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=567 rev=3\nThu May 16 04:32:43.849705 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:32:43.849705 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=574 rev=4\nThu May 16 04:34:08.487842 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:08.487842 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=154 rev=4\nThu May 16 04:34:40.688906 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:40.689883 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=454 rev=4\nThu May 16 04:34:41.123449 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:41.123449 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=456 rev=4\nThu May 16 04:34:51.585670 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:51.586646 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=511 rev=5\nThu May 16 04:34:52.812154 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:52.812154 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=518 rev=4\nThu May 16 04:34:59.385952 Pacific Daylight Time 3: (bucket-0) Received error[86] from mccouch for unknown\nThu May 16 04:34:59.386928 Pacific Daylight Time 3: (bucket-0) Retry notify CouchDB of update, vbucket=550 rev=5\nThu May 16 04:39:57.487966 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - disconnected, keep alive for 300 seconds\nThu May 16 04:39:57.588545 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.179 - disconnected, keep alive for 300 seconds\nThu May 16 04:39:57.780916 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_2904 - disconnected\nThu May 16 04:39:57.783845 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.178 - disconnected, keep alive for 300 seconds\nThu May 16 04:39:57.788728 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_1541 - disconnected\nThu May 16 04:39:57.826811 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_131 - disconnected\nThu May 16 04:39:57.913720 Pacific Daylight Time 3: (bucket-0) Shutting down tap connections!\nThu May 16 04:39:57.913720 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.178"\nThu May 16 04:39:57.913720 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_131"\nThu May 16 04:39:57.913720 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.178 - Clear the tap queues by force\nThu May 16 04:39:57.914696 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_1541"\nThu May 16 04:39:57.914696 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.179"\nThu May 16 04:39:57.916649 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_2904"\nThu May 16 04:39:57.916649 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.179 - Clear the tap queues by force\nThu May 16 04:39:57.916649 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:replication_ns_1@10.3.2.185"\nThu May 16 04:39:57.919579 Pacific Daylight Time 3: (bucket-0) TAP (Producer) eq_tapq:replication_ns_1@10.3.2.185 - Clear the tap queues by force\nThu May 16 04:40:23.080161 Pacific Daylight Time 3: (bucket-0) Warning: failed to load the engine session stats due to IO exception "basic_ios::clear"\nThu May 16 04:40:23.080161 Pacific Daylight Time 3: (bucket-0) Failed to load mutation log, falling back to key dump\nThu May 16 04:40:23.080161 Pacific Daylight Time 3: Extension support isn\'t implemented in this version of bucket_engine\nThu May 16 04:40:23.081137 Pacific Daylight Time 3: (bucket-0) metadata loaded in 1162 usec\nThu May 16 04:40:23.082114 Pacific Daylight Time 3: (bucket-0) warmup completed in 1738 usec\nThu May 16 04:40:26.572125 Pacific Daylight Time 3: (bucket-0) Connection closed by mccouch\nThu May 16 04:40:26.572125 Pacific Daylight Time 3: (bucket-0) Resetting connection to mccouch, lastReceivedCommand = notify_vbucket_update lastSentCommand = notify_vbucket_update currentCommand =notify_vbucket_update\nMUTEX ERROR: Failed to acquire lock: Invalid argument\n\nThis application has requested the Runtime to terminate it in an unusual way.\nPlease contact the application\'s support team for more information.\nThu May 16 04:40:26.785978 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3330 - Reset vbucket 596 was completed succecssfully.\nThu May 16 04:40:37.310695 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3331 - Reset vbucket 595 was completed succecssfully.\nThu May 16 04:40:37.419087 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3331 - disconnected\nThu May 16 04:40:37.419087 Pacific Daylight Time 3: (bucket-0) TAP (Consumer) eq_tapq:anon_3330 - disconnected\nThu May 16 04:40:38.198334 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3330"\nThu May 16 04:40:38.198334 Pacific Daylight Time 3: (bucket-0) Schedule cleanup of "eq_tapq:anon_3331"', u'shortText': u'message', u'module': u'ns_log', u'tstamp': 1368704438866, u'type': u'info'}
[2013-05-16 04:41:11,738] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.183', u'code': 4, u'text': u"Control connection to memcached on 'ns_1@10.3.2.183' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368704438788, u'type': u'info'}
[2013-05-16 04:41:11,738] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.185', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.185\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368704436520, u'type': u'info'}
[2013-05-16 04:41:11,739] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.185', u'code': 3, u'text': u'Node ns_1@10.3.2.185 joined cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1368704435895, u'type': u'info'}
[2013-05-16 04:41:11,741] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.185', u'code': 4, u'text': u"Node 'ns_1@10.3.2.185' saw that node 'ns_1@10.3.2.183' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1368704435895, u'type': u'info'}
[2013-05-16 04:41:11,742] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.185', u'code': 4, u'text': u"Node 'ns_1@10.3.2.185' saw that node 'ns_1@10.3.2.179' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1368704435707, u'type': u'info'}
ERROR

[MB-8295] Dev views uses bsuperstar, but single vbucket

Thu, 16 May 2013 07:34:46 -0500

Dev views uses bsuperstar, but single vbucket

[MB-8294] Destination node with XDCR (heavy DGM) goes into pending.

Sat, 18 May 2013 01:57:03 -0500

Destination node with XDCR (heavy DGM) goes into pending.

Setup Information:

- Cluster Config : 4 nodes, OS: Centos 6.3, CPU : 6 Core, RAM : 16G, Disk : 500G
- XDCR Topology : Unidirectional ( Master(3) -> Slave(1))
- 4 buckets AbRegNums(2GB), MsgsCalls(2GB), RevAB(3GB), UserInfo(2GB)
- Data Loaded using Viber Workload with 6.7M(RR ~45%), 0.1M(RR 100%), 11M(RR 100%), 12M(RR ~30%) data on the 4 buckets
- Unidirectional XDCR setup to 1 node(master -> slave) for 3 buckets AbRegNums, RevAB, UserInfo

Destination node goes into pending state(screenshot attached). Both beam.smp and memcached are running on the node.

There are lot of crash reports like below but there are no views/design docs in the system.

[error_logger:error,2013-05-16T1:40:30.265,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
  crasher:
    initial call: set_view_update_daemon:init/1
    pid: <0.9511.707>
    registered_name: set_view_update_daemon
    exception exit: {noproc,
                        {gen_server,call,
                            ['capi_set_view_manager-UserInfo',
                             {foreach_doc,
                                 #Fun<capi_ddoc_replication_srv.2.102018441>},
                             infinity]}}
      in function gen_server:terminate/6
    ancestors: [ns_server_sup,ns_server_cluster_sup,<0.58.0>]
    messages: []
    links: [<0.298.0>,<0.9512.707>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 121393
    stack_size: 24
    reductions: 7537
  neighbours:

=========================CRASH REPORT=========================
  crasher:
    initial call: compaction_daemon:-spawn_bucket_compactor/3-fun-2-/0
    pid: <0.9508.707>
    registered_name: []
    exception exit: {noproc,
                        {gen_server,call,
                            ['capi_set_view_manager-RevAB',
                             {foreach_doc,
                                 #Fun<capi_ddoc_replication_srv.1.36030090>},
                             infinity]}}
      in function gen_server:call/3
      in call from capi_ddoc_replication_srv:foreach_live_ddoc_id/2
      in call from capi_ddoc_replication_srv:fetch_ddoc_ids/1
      in call from compaction_daemon:'-spawn_bucket_compactor/3-fun-2-'/4
    ancestors: [compaction_daemon,ns_server_sup,ns_server_cluster_sup,
                  <0.58.0>]
    messages: []
    links: [<0.412.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 28657
    stack_size: 24
    reductions: 30607
  neighbours:

The pending node is in the same state and can be used for investigation:
coconut-h20804.hq.couchbase.com

[MB-8293] Alert UI annoyingly flashes as metadata is used

Duplicate — Thu, 16 May 2013 03:29:06 -0500

As we go beyond 50% metadata, the UI will show alerts in the console. Problem is, the alerts display non-stop, with no way to disable them, making use of the UI very hard. One has to constantly close the alert box.

Scenario:
Load system with many items, such that metadata is consumed.

Expected behavior:
Alert can be acknowledged and won't be displayed again as long as we're in the alert area. Obviously, if we exit the alert range and hit it again, we'd expect to see the alert again.

Observed behavior:
While workload is running, the alert box is always popping up. Incessantly. It cannot be stopped. Yes, I know I'm above 50% metadata, go away. No, really, please go away. GO!

[MB-8292] Memory leak? on one of the buckets on destination cluster (single node) with source cluster on v_iber workload

Sat, 18 May 2013 02:00:53 -0500

- Viber workload on all 4 buckets, greater load on RevAB compared to the rest.
- After about half a day's run time:

- After a point, replication broke for RevAB as that bucket on destination has run out of memory.
- Resident ratios still at 100% for all buckets on source, and on destination 100% resident ratios on AbRegNums and UserInfo, but 0% on RevAB.
- Mem used is at 3G, with higher water mark at about 2.7G, no temp OOMs noticed however.
- cbstats on destination node:

ep_diskqueue_memory: 0
ep_mem_high_wat: 2738041651
ep_mem_low_wat: 2415919104
ep_mem_tracker_enabled: true
ep_meta_data_memory: 549618696
ep_mutation_mem_threshold: 95
ep_warmup_min_memory_threshold: 100
mem_used: 3060163976
vb_active_ht_memory: 50790400
vb_active_itm_memory: 610687928
vb_active_meta_data_memory: 549618696
vb_active_perc_mem_resident: 0
vb_active_queue_memory: 0
vb_pending_ht_memory: 0
vb_pending_itm_memory: 0
vb_pending_meta_data_memory: 0
vb_pending_perc_mem_resident: 0
vb_pending_queue_memory: 0

Attached cbcollect_info for source (10.3.4.27):
https://s3.amazonaws.com/bugdb/MB--/10_3_4_27.zip
, and destination (10.3.4.30):
https://s3.amazonaws.com/bugdb/MB--/10_3_4_30.zip

[MB-8291] [system test] warmup access log corrupt during warmup in windows

Fri, 17 May 2013 17:21:06 -0500

Environment:
  9 nodes windows server 2008 R2 64-bit
  Each node has 8GB RAM, 40+ GB SSD

Cluster:
  Create 7 nodes cluster from nodes below
1:10.3.121.173
2:10.3.121.169
3:10.3.121.171
4:10.3.3.214
5:10.3.121.47
6:10.3.3.180
7:10.3.3.181

8:10.3.3.182
9:10.3.121.243

  Create 2 buckets:
    default: 3GB, 1 replica, replica index enable
    saslbucket: 3GB, 1 replica, replica index disable

Run tests kv only using specification from http://hub.internal.couchbase.com/confluence/display/QA/views+%28and+now+with+XDCR%29+tests

At warm up phase (stop and restart couchbase server on node 10.3.121.243), monitor the warmup process
Check the wamup status of this node and see access_log is corrupted as below

$ /cygdrive/c/Program\ Files/Couchbase/Server/bin/cbstats.exe localhost:11210 raw warmup
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 28569
ep_warmup_estimated_key_count: 3828057
ep_warmup_estimated_value_count: 3828057
ep_warmup_item_expired: 0
ep_warmup_key_count: 3828057
ep_warmup_keys_time: 45952020
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 131417617
ep_warmup_value_count: 3804872

Administrator@WIN-5IC1JB7LHUB ~
$ /cygdrive/c/Program\ Files/Couchbase/Server/bin/cbstats.exe localhost:11210 raw warmup -b saslbucket -p password
ep_warmup: enabled
ep_warmup_access_log: corrupt
ep_warmup_dups: 0
ep_warmup_estimate_time: 40933
ep_warmup_estimated_key_count: 23217866
ep_warmup_estimated_value_count: 23217866
ep_warmup_item_expired: 0
ep_warmup_key_count: 23217866
ep_warmup_keys_time: 243091809
ep_warmup_min_item_threshold: 100
ep_warmup_min_memory_threshold: 100
ep_warmup_oom: 0
ep_warmup_state: done
ep_warmup_thread: complete
ep_warmup_time: 243243953
ep_warmup_value_count: 1

Link to manifest file http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-802-rel.setup.exe.manifest.xml

Link to cbcollect_info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_2/2013_05/9ndoes_202-802_access-log-corrupt_at_warmup_20130515-125236.tgz

[MB-8290] cbcollectinfo failed to get logs in windows

Wed, 15 May 2013 20:09:02 -0500

Install couchbase server 2.0.2-803 on one node windows server 2008 r2 64bit

After install, the couchbase server web console is down. Filed bug ~~MB-8287~~
Run cbcollect_info to get logs for bug ~~MB-8287~~. Failed to collect logs with error

C:\>".\Program Files\Couchbase\Server\bin\cbcollect_info" node_21.zip
System information (systeminfo) - OK
Computer system (wmic computersystem) - OK
Computer OS (wmic os) - OK
Service list (wmic service where state="running" GET caption, name, state) - OK
Process list (wmic process) - OK
Process usage (tasklist /V /fo list) - OK
Swap settings (wmic pagefile) - OK
Disk partition (wmic partition) - OK
Disk volumes (wmic volume) - OK
Network configuration (ipconfig /all) - OK
Taking sample 2 after 10.000000 seconds -
OK
Network status (netstat -an) - OK
Network routing table (netstat -rn) - OK
Arp cache (arp -a) - OK
Network Interface Controller (wmic nic) - OK
Network Adapter (wmic nicconfig) - OK
Active network connection (wmic netuse) - OK
Protocols (wmic netprotocol) - OK
Cache memory (wmic memcache) - OK
Physical memory (wmic memphysical) - OK
Physical memory chip info (wmic memorychip) - OK
Local storage devices (wmic logicaldisk) - OK
Checking for server guts in C:\Program Files\Couchbase\Server\var\lib\couchbase\initargs...
Failed to read ns_config: {'EXIT',
                              {badarg,
                                  [{erlang,binary_to_term,[<<>>]},
                                   {'dump-guts__escript__1368__640921__899000',
                                       read_config,1},
                                   {'dump-guts__escript__1368__640921__899000',
                                       main_with_static_config,1},
                                   {'dump-guts__escript__1368__640921__899000',
                                       main,1},
                                   {escript,run,2},
                                   {escript,start,1},
                                   {init,start_it,1},
                                   {init,start_em,1}]}}
Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs...
Failed to read initargs:
{error,enoent}
Checking for server guts in C:\Users\Administrator/Library/Application Support/Couchbase/var/lib/cou
chbase/initargs...
Failed to read initargs:
{error,enoent}�
                                                                                    ��

[MB-8287] couhbase web console is down although erlang process is running

Wed, 15 May 2013 15:50:13 -0500

Environment:
4 nodes windows server 2008 r2 64bit (each node has 4 core cpu annd 4GB RAM)
Run sanity test on build 2.0.2-803.
After uninstall old build and install build 2.0.2-803, connection to couchbase server of one node failed.
I go to that node (10.3.2.21) and see erl process is running.
Stop couchbase server.
Restart couchbase server
erlang process is running but web console is not up
Check firewall, all firewall are turn off

The node is in failed state now (10.3.2.21 with Administrator/Membase123)

[MB-8286] Explicit error message if encoded value has a length not representable with 24 bits during index builds and updates

Wed, 15 May 2013 09:08:47 -0500

Explicit error message if encoded value has a length not representable with 24 bits during index builds and updates

[MB-8285] DevGuide: metadata reported as 150 bytes needs to be revised

Wed, 15 May 2013 09:04:08 -0500

Found a few places where the metadata size is still reading 150 bytes and should be reduced to 60 for 2.0.1 and 54 for 2.0.2...maybe we can setup some sort of linking or monitoring as this will continue to change in future versions and customers are stumbling across the old values

http://www.couchbase.com/docs/couchbase-devguide-2.0/cb-store-operations.html
http://www.couchbase.com/docs/couchbase-devguide-2.0/cb-store-operations.html
http://www.couchbase.com/docs/couchbase-devguide-2.0/couchbase-keys.html
\http://www.couchbase.com/docs/couchbase-devguide-2.0/cb-store-operations.html

[MB-8284] Rebalance fails with reason replicator_died

Fri, 17 May 2013 18:33:48 -0500

Rebalance failures are seen in Centos64 sanity job:
http://qa.hq.northscale.net/job/CouchbaseServer-SanityTest-4Node-Centos64/73/consoleFull

** Reason for termination ==
** {exited,
       {'EXIT',<0.28006.6>,
           {replicator_died,
               {'EXIT',<17674.28481.2>,{badmatch,{error,closed}}}}}}

[ns_server:info,2013-05-15T3:24:33.837,ns_1@10.3.3.32:janitor_agent-default<0.23324.6>:janitor_agent:handle_info:761]Undoing temporary vbucket states caused by rebalance
[user:info,2013-05-15T3:24:33.837,ns_1@10.3.3.32:<0.30432.2>:ns_orchestrator:handle_info:403]Rebalance exited with reason {exited,
                              {'EXIT',<0.28006.6>,
                               {replicator_died,
                                {'EXIT',<17674.28481.2>,
                                 {badmatch,{error,closed}}}}}}

ns_server:error,2013-05-15T3:24:33.584,ns_1@10.3.3.32:<0.28006.6>:ns_replicas_builder:build_replicas_main:90]Got premature exit from one of ebucketmigrators: {'EXIT',<17674.28481.2>,
                                                  {badmatch,{error,closed}}}
[error_logger:error,2013-05-15T3:24:33.585,ns_1@10.3.3.32:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
  crasher:
    initial call: ebucketmigrator_srv:init/1
    pid: <17674.28481.2>
    registered_name: []
    exception error: no match of right hand side value {error,closed}
      in function mc_client_binary:cmd_binary_vocal_recv/5
      in call from mc_client_binary:set_vbucket/3
      in call from ebucketmigrator_srv:'-init/1-lc$^0/1-0-'/3
      in call from ebucketmigrator_srv:init/1
    ancestors: [<0.28006.6>,<0.28000.6>,<0.27823.6>,<0.26685.6>]
    messages: []
    links: [#Port<17674.80060>,<0.28006.6>,#Port<17674.80059>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 24
    reductions: 1725
  neighbours:

Attaching collect_info

[MB-8283] Typos from Japanese Translator

Tue, 14 May 2013 23:28:45 -0500

I should also mention that Japanese translation team found a number of
typos in the original English documentation.
It is tough to submit as issues to JIRA one by one, so I've created a
sheet in the Google Spreadsheet to share with you.
Please find the "Typos" sheet in this document:
https://docs.google.com/a/atware.co.jp/spreadsheet/ccc?key=0Ak1vmI0MjYfZdDYzM2FqYXJWSU5ueTNvTlRHaVZ5aGc&usp=sharing

We put original documentation URL, the sentence and suggestion for correction.
Please check it and update its 'status' column or put your comments if needed.

Thank you!
Koji Kawamura

[MB-8282] -b bucket selection option is not filtering other buckets for cbhealthchecker

Thu, 16 May 2013 14:40:53 -0500

1. create 2 buckets, saslbucket, and default.
2. Start the front end workload.
3. When the system is in dgm state, run cbhealthchecker just for default bucket

[jenkins@centos-54-x64-302 ~]$ /opt/couchbase/bin/cbhealthchecker -c 172.23.105.23:8091 -u Administrator -p password -b default
bucket: default
  node: 172.23.105.29 11210
  node: 172.23.105.23 11210
  node: 172.23.105.30 11210
  node: 172.23.105.31 11210
  node: 172.23.105.25 11210
  node: 172.23.105.28 11210
  node: 172.23.105.32 11210
  node: 172.23.105.26 11210
................................
Traceback (most recent call last):
  File "/opt/couchbase/lib/python/cbhealthchecker", line 134, in ?
    main()
  File "/opt/couchbase/lib/python/cbhealthchecker", line 121, in main
    scale = retriever.collect_data(bucket, cluster, user, password, inputfile, statsfile, scale, opts, outputdir)
  File "/opt/couchbase/lib/python/collector.py", line 316, in collect_data
    self.get_ns_stats(bucketlist, server, port, user, password, bucketname, scale, opts)
  File "/opt/couchbase/lib/python/collector.py", line 279, in get_ns_stats
    if stats_buffer.bucket_info[bucket_name]["bucketType"] == 'memcached':
KeyError: u'saslbucket'

-b option is not working properly. When we iterate stats_buffer.bucket_info, we are still looking for sasl bucket

[MB-8281] other buckets storage size shows negative number

Wed, 15 May 2013 13:37:15 -0500

Environment:
4 ubuntu 10.04 64bit, 4 core, 4GB RAM

Couchbase server 2.0.2-802

Run test:
./testrunner -i ../../ini/4-c-up.ini -t failovertests.FailoverTests.test_failover_normal,replica=2,load_ratio=1

Open default bucket info, see other buckets value in negative
Reload browser several times, other buckes still shows negative value

Link to manifest file http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-802-rel.rpm.manifest.xml

Link to collect info file of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_2/2013_05/4nodes_202-802_bucket_info_negative_value_20130514-153022.tgz

[MB-8280] cbrecovery should check if tap batch requests are processed correctly before moving to next batch

Thu, 16 May 2013 16:15:36 -0500

All recovered msgs are pumped into destination cluster in batch mode. If due to slow traffic or poor server performance, batch msg processing may fail.

Recovery tool should make sure every batch is processed correctly before moving to next one. Acknowledgement request for the last msg in batch may be an option.

[MB-8279] System Test : Error replicating vbucket XXX {http_request_failed, "POST", {error,{code,500}}

Tue, 14 May 2013 17:46:28 -0500

1. Setup a 3 node cluster with 4 buckets.
2. Setup another destination cluster with 3 buckets.
3. Setup 3 replications from source to destination bucket.

Bucket RevAB shows xdcr replication errors.
Error replicating vbucket 758: {http_request_failed, "POST", "http://Administrator:*****@10.3.4.30:8092/RevAB%2f758%3b12b42cb2c0aa11f587ddc265ac174f81/_bulk_docs", {error,{code,500}}

Results.
- Replication not catching up, 0.5M items still to be replicated, no more sets on destination node.
- The replication status shows 100 percent complete - this is incorrect. There are 0.5M to be replicated to destination.

Adding logs.

[MB-8278] Need to provide Tool Tips in Web Console

Tue, 14 May 2013 15:38:57 -0500

Hi,

I was talking to Maria about 2.1 futures and right now we have all of these field descriptions from Web Console in the Web Console chapter of documentation. Example here: http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-web-console-data-buckets-individual.html#couchbase-admin-web-console-data-buckets-tapqueues

Where this information really belong is as tool tips in the UI itself. People are not going to go into the docs to find out what a field means. That is awkward and makes them leave the software. It is much easier to roll over the item and get context-sensitive information about the item in the UI.

I think having the amount of web console doc in documentation is a legacy of not having a UI designer/developer.

It is much easier and a standard industry practice to:

- only document how to manage something in a UI
- not document wizards since the process of how to use a wizard is already in the UI
- provide tool tips for UI elements (what it means, what it does) and make sure they are section 508 compliant.

If we can have this for 2.1, that would be great.

Thanks,

Karen

[MB-8277] Front Matter for Release Notes

Tue, 14 May 2013 23:36:12 -0500

As we did for 2.0.1, I need the intro/front-matter for release notes. Currently:

Couchbase Server 2.0.2 is second maintenance release for
Couchbase Server 2.0. This release contains a number of enhancements
including:

[MB-8276] Run Japanese Translations

Tue, 14 May 2013 14:47:14 -0500

Run Japanese Translations

[MB-8275] installation document in 2.0 in rpm shows old printout after installation

Wed, 15 May 2013 17:34:41 -0500

The information in http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-getting-started-install-redhat.html
showing the old print out of 1.8.1 version

[root@cen-1907 ~]# rpm -i couchbase-server-enterprise_x86_64_1.8.1-937-rel.rpm
Starting couchbase-server[ OK ]

You have successfully installed Couchbase Server.
Please browse to http://cen-1907:8091/ to configure your server.
Please refer to http://couchbase.com for additional resources.

Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091 and from 21100 to 21299.

By using this software you agree to the End User License Agreement.
See /opt/couchbase/LICENSE.txt.

From 2.0 and later, it should show requirement of hardware as in the following printout
[root@cen-1907 ~]# rpm -i couchbase-server-enterprise_x86_64_2.0.x-xx-rel.rpm
Minimum RAM required : 4 GB
System RAM configured : 8174464 kB

Minimum number of processors required : 4 cores
Number of processors on the system : 4 cores

Starting couchbase-server[ OK ]

You have successfully installed Couchbase Server.
Please browse to http://cen-1907:8091/ to configure your server.
Please refer to http://couchbase.com for additional resources.

Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091, 8092 and from 21100 to 21299.

By using this software you agree to the End User License Agreement.
See /opt/couchbase/LICENSE.txt.

It may be out of date in other package.

[MB-8274] items not draining seems... ep-engine is deadlocked

Wed, 15 May 2013 17:33:56 -0500

during xdcr longevity test memcached kept restarting on a node
ui reports: Control connection to memcached on 'ns_1@172.23.105.57' disconnected: {badmatch,
{error,
couldnt_connect_to_memcached}}

(guinep-s1050.sc.couchbase.com)

everytime I do a gdb backtrace I see threads 16,17,18 in the lock() method.

Thread 18 (Thread 0x7f1212849700 (LWP 21113)):
#0 0x00007f121a5b4054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f121a5af388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f121a5af257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f1214436f7a in Mutex::acquire (this=0x641e0f0) at src/mutex.cc:79
#4 0x00007f121447d566 in lock (this=0x641e000) at ./src/locks.hh:48
#5 LockHolder (this=0x641e000) at ./src/locks.hh:26
#6 CouchNotifier::selectBucket (this=0x641e000) at src/couch-kvstore/couch-notifier.cc:723
#7 0x00007f121447dbcf in CouchNotifier::processInput (this=0x641e000) at src/couch-kvstore/couch-notifier.cc:606
#8 0x00007f121447e475 in waitOnce (this=0x641e000, vbs=..., file_version=1, header_offset=212992, cb=...) at src/couch-kvstore/couch-notifier.cc:675
#9 CouchNotifier::notify_update (this=0x641e000, vbs=..., file_version=1, header_offset=212992, cb=...) at src/couch-kvstore/couch-notifier.cc:755
#10 0x00007f1214475cb8 in notify_headerpos_update (this=0x643fb00, vbid=760, rev=1, docs=0x323bd880, docinfos=0x323bda40, docCount=56) at ./src/couch-kvstore/couch-notifier.hh:144
#11 CouchKVStore::saveDocs (this=0x643fb00, vbid=760, rev=1, docs=0x323bd880, docinfos=0x323bda40, docCount=56) at src/couch-kvstore/couch-kvstore.cc:1498
#12 0x00007f121447628b in CouchKVStore::commit2couchstore (this=0x643fb00) at src/couch-kvstore/couch-kvstore.cc:1410
#13 0x00007f121447647a in CouchKVStore::commit (this=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:806
#14 0x00007f1214401f56 in EventuallyPersistentStore::flushVBucket (this=0x6395c00, vbid=760) at src/ep.cc:1919
#15 0x00007f121442aeb9 in doFlush (this=0x520b7a0, tid=1098) at src/flusher.cc:222
#16 Flusher::step (this=0x520b7a0, tid=1098) at src/flusher.cc:152
#17 0x00007f121443ac10 in ExecutorThread::run (this=0x52bfba0) at src/scheduler.cc:148
#18 0x00007f121443b32d in launch_executor_thread (arg=0x52bfba0) at src/scheduler.cc:34
#19 0x00007f121a5ad851 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f121a2fb90d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f1211e48700 (LWP 21114)):
#0 0x00007f121a5b4054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f121a5af388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f121a5af257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f1214436f7a in Mutex::acquire (this=0x641e0f0) at src/mutex.cc:79
#4 0x00007f121447e2c3 in lock (this=0x641e000, vbs=..., file_version=1, header_offset=217088, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x641e000, vbs=..., file_version=1, header_offset=217088, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x641e000, vbs=..., file_version=1, header_offset=217088, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00007f1214475cb8 in notify_headerpos_update (this=0x64d4c00, vbid=693, rev=1, docs=0x93f4480, docinfos=0x94186c0, docCount=72) at ./src/couch-kvstore/couch-notifier.hh:144
#8 CouchKVStore::saveDocs (this=0x64d4c00, vbid=693, rev=1, docs=0x93f4480, docinfos=0x94186c0, docCount=72) at src/couch-kvstore/couch-kvstore.cc:1498
#9 0x00007f121447628b in CouchKVStore::commit2couchstore (this=0x64d4c00) at src/couch-kvstore/couch-kvstore.cc:1410
#10 0x00007f121447647a in CouchKVStore::commit (this=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:806
#11 0x00007f1214401f56 in EventuallyPersistentStore::flushVBucket (this=0x6395c00, vbid=693) at src/ep.cc:1919
#12 0x00007f121442aeb9 in doFlush (this=0x520b680, tid=1095) at src/flusher.cc:222
#13 Flusher::step (this=0x520b680, tid=1095) at src/flusher.cc:152
#14 0x00007f121443ac10 in ExecutorThread::run (this=0x52bfa00) at src/scheduler.cc:148
#15 0x00007f121443b32d in launch_executor_thread (arg=0x52bfa00) at src/scheduler.cc:34
#16 0x00007f121a5ad851 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f121a2fb90d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f1211447700 (LWP 21115)):
#0 0x00007f121a5b4054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f121a5af388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f121a5af257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f1214436f7a in Mutex::acquire (this=0x641e0f0) at src/mutex.cc:79
#4 0x00007f121447e2c3 in lock (this=0x641e000, vbs=..., file_version=1, header_offset=212992, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x641e000, vbs=..., file_version=1, header_offset=212992, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x641e000, vbs=..., file_version=1, header_offset=212992, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00007f1214475cb8 in notify_headerpos_update (this=0x64d4600, vbid=750, rev=1, docs=0x321a4480, docinfos=0x321a4fc0, docCount=65) at ./src/couch-kvstore/couch-notifier.hh:144
#8 CouchKVStore::saveDocs (this=0x64d4600, vbid=750, rev=1, docs=0x321a4480, docinfos=0x321a4fc0, docCount=65) at src/couch-kvstore/couch-kvstore.cc:1498
#9 0x00007f121447628b in CouchKVStore::commit2couchstore (this=0x64d4600) at src/couch-kvstore/couch-kvstore.cc:1410
#10 0x00007f121447647a in CouchKVStore::commit (this=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:806
#11 0x00007f1214401f56 in EventuallyPersistentStore::flushVBucket (this=0x6395c00, vbid=750) at src/ep.cc:1919
#12 0x00007f121442aeb9 in doFlush (this=0x520b560, tid=1096) at src/flusher.cc:222
#13 Flusher::step (this=0x520b560, tid=1096) at src/flusher.cc:152
#14 0x00007f121443ac10 in ExecutorThread::run (this=0x52bf860) at src/scheduler.cc:148
#15 0x00007f121443b32d in launch_executor_thread (arg=0x52bf860) at src/scheduler.cc:34
#16 0x00007f121a5ad851 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f121a2fb90d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f1210a46700 (LWP 21116)):
#0 0x00007f121a5b4054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f121a5af388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f121a5af257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f1214436f7a in Mutex::acquire (this=0x528f0f0) at src/mutex.cc:79
#4 0x00007f121447d566 in lock (this=0x528f000) at ./src/locks.hh:48
#5 LockHolder (this=0x528f000) at ./src/locks.hh:26
#6 CouchNotifier::selectBucket (this=0x528f000) at src/couch-kvstore/couch-notifier.cc:723
#7 0x00007f121447dbcf in CouchNotifier::processInput (this=0x528f000) at src/couch-kvstore/couch-notifier.cc:606
#8 0x00007f121447e475 in waitOnce (this=0x528f000, vbs=..., file_version=2, header_offset=1798144, cb=...) at src/couch-kvstore/couch-notifier.cc:675
#9 CouchNotifier::notify_update (this=0x528f000, vbs=..., file_version=2, header_offset=1798144, cb=...) at src/couch-kvstore/couch-notifier.cc:755
#10 0x00007f1214475cb8 in notify_headerpos_update (this=0x5295b00, vbid=1023, rev=2, docs=0x1a897c00, docinfos=0x1a896a80, docCount=107) at ./src/couch-kvstore/couch-notifier.hh:144
#11 CouchKVStore::saveDocs (this=0x5295b00, vbid=1023, rev=2, docs=0x1a897c00, docinfos=0x1a896a80, docCount=107) at src/couch-kvstore/couch-kvstore.cc:1498
#12 0x00007f121447628b in CouchKVStore::commit2couchstore (this=0x5295b00) at src/couch-kvstore/couch-kvstore.cc:1410
#13 0x00007f121447647a in CouchKVStore::commit (this=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:806
#14 0x00007f1214401f56 in EventuallyPersistentStore::flushVBucket (this=0x527e000, vbid=1023) at src/ep.cc:1919
#15 0x00007f121442aeb9 in doFlush (this=0x520a360, tid=12) at src/flusher.cc:222
#16 Flusher::step (this=0x520a360, tid=12) at src/flusher.cc:152
#17 0x00007f121443ac10 in ExecutorThread::run (this=0x52e6ea0) at src/scheduler.cc:148
#18 0x00007f121443b32d in launch_executor_thread (arg=0x52e6ea0) at src/scheduler.cc:34
#19 0x00007f121a5ad851 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f121a2fb90d in clone () from /lib64/libc.so.6

there are also errors in memcached.logs about "Too many connections"

I have full backtrace attached and machine is currently live.

[MB-8273] System test : observed slow swap-rebalance in 2.0.2 ( 15 hours+)

Thu, 16 May 2013 19:23:49 -0500

1. Setup a 6 node cluster with 2 buckets.
2. Each bucket has 1 ddoc and 2 views.
3. Load 60M items on both buckets, data-size 512 bytes consistent
4. Start mutations on the cluster for 2 hours.
5. Wait for initial indexing to complete.
6. Swap rebalance 1-1 node.

- Seeing very high swap ~ 5G on most of the nodes.
- Further inspection of process using swap shows memcached is using most of swap space.
- The memory stats show very high fragmentation
- Views are highly fragmented ( 91 percent) across the cluster.

]# pgrep memcached
9551
# grep --color VmSwap /proc/9551/status
VmSwap: 5219956 kB

[root@orange-11601 ~]# /opt/couchbase/bin/cbstats localhost:11210 -b default allocator
NOTE: SMALL MEMORY MODEL IS IN USE, PERFORMANCE MAY SUFFER.
------------------------------------------------
MALLOC: 14060977616 (13409.6 MiB) Bytes in use by application
MALLOC: + 263405568 ( 251.2 MiB) Bytes in page heap freelist
MALLOC: + 5191139360 ( 4950.7 MiB) Bytes in central cache freelist
MALLOC: + 0 ( 0.0 MiB) Bytes in transfer cache freelist
MALLOC: + 2662928 ( 2.5 MiB) Bytes in thread cache freelists
MALLOC: + 137932952 ( 131.5 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 19656118424 (18745.5 MiB) Actual memory used (physical + swap)
MALLOC: + 126885888 ( 121.0 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 19783004312 (18866.5 MiB) Virtual address space used
MALLOC:
MALLOC: 2348103 Spans in use
MALLOC: 20 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
------------------------------------------------
Size class breakdown
------------------------------------------------
class 1 [ 8 bytes ] : 838 objs; 0.0 MiB; 0.0 cum MiB
class 2 [ 16 bytes ] : 2856430 objs; 43.6 MiB; 43.6 cum MiB
class 3 [ 32 bytes ] : 684331 objs; 20.9 MiB; 64.5 cum MiB
class 4 [ 48 bytes ] : 10987 objs; 0.5 MiB; 65.0 cum MiB
class 5 [ 64 bytes ] : 21645 objs; 1.3 MiB; 66.3 cum MiB
class 6 [ 80 bytes ] : 5347446 objs; 408.0 MiB; 474.3 cum MiB
class 7 [ 96 bytes ] : 905 objs; 0.1 MiB; 474.4 cum MiB
class 8 [ 112 bytes ] : 130 objs; 0.0 MiB; 474.4 cum MiB
class 9 [ 128 bytes ] : 337 objs; 0.0 MiB; 474.4 cum MiB
class 10 [ 144 bytes ] : 224 objs; 0.0 MiB; 474.4 cum MiB
class 11 [ 160 bytes ] : 260 objs; 0.0 MiB; 474.5 cum MiB
class 12 [ 176 bytes ] : 43 objs; 0.0 MiB; 474.5 cum MiB
class 13 [ 192 bytes ] : 83 objs; 0.0 MiB; 474.5 cum MiB
class 14 [ 208 bytes ] : 64 objs; 0.0 MiB; 474.5 cum MiB
class 15 [ 224 bytes ] : 70 objs; 0.0 MiB; 474.5 cum MiB
class 16 [ 240 bytes ] : 66 objs; 0.0 MiB; 474.6 cum MiB
class 17 [ 256 bytes ] : 127 objs; 0.0 MiB; 474.6 cum MiB
class 18 [ 288 bytes ] : 41 objs; 0.0 MiB; 474.6 cum MiB
class 19 [ 320 bytes ] : 76 objs; 0.0 MiB; 474.6 cum MiB
class 20 [ 352 bytes ] : 119 objs; 0.0 MiB; 474.7 cum MiB
class 21 [ 384 bytes ] : 57 objs; 0.0 MiB; 474.7 cum MiB
class 22 [ 416 bytes ] : 28 objs; 0.0 MiB; 474.7 cum MiB
class 23 [ 448 bytes ] : 52 objs; 0.0 MiB; 474.7 cum MiB
class 24 [ 480 bytes ] : 36 objs; 0.0 MiB; 474.7 cum MiB
class 25 [ 512 bytes ] : 907 objs; 0.4 MiB; 475.2 cum MiB
class 26 [ 576 bytes ] : 165 objs; 0.1 MiB; 475.3 cum MiB
class 27 [ 640 bytes ] : 18 objs; 0.0 MiB; 475.3 cum MiB
class 28 [ 704 bytes ] : 60 objs; 0.0 MiB; 475.3 cum MiB
class 29 [ 768 bytes ] : 4872097 objs; 3568.4 MiB; 4043.7 cum MiB
class 30 [ 896 bytes ] : 42 objs; 0.0 MiB; 4043.8 cum MiB
class 31 [ 1024 bytes ] : 96 objs; 0.1 MiB; 4043.9 cum MiB
class 32 [ 1152 bytes ] : 50 objs; 0.1 MiB; 4043.9 cum MiB
class 33 [ 1280 bytes ] : 38 objs; 0.0 MiB; 4044.0 cum MiB
class 34 [ 1408 bytes ] : 33 objs; 0.0 MiB; 4044.0 cum MiB
class 35 [ 1536 bytes ] : 20 objs; 0.0 MiB; 4044.0 cum MiB
class 36 [ 1792 bytes ] : 78 objs; 0.1 MiB; 4044.2 cum MiB
class 37 [ 2048 bytes ] : 122 objs; 0.2 MiB; 4044.4 cum MiB
class 38 [ 2304 bytes ] : 12 objs; 0.0 MiB; 4044.4 cum MiB
class 39 [ 2560 bytes ] : 5 objs; 0.0 MiB; 4044.5 cum MiB
class 40 [ 2816 bytes ] : 16 objs; 0.0 MiB; 4044.5 cum MiB
class 42 [ 3328 bytes ] : 52 objs; 0.2 MiB; 4044.7 cum MiB
class 43 [ 4096 bytes ] : 29 objs; 0.1 MiB; 4044.8 cum MiB
class 47 [ 6656 bytes ] : 64 objs; 0.4 MiB; 4045.2 cum MiB
class 48 [ 8192 bytes ] : 14 objs; 0.1 MiB; 4045.3 cum MiB
class 49 [ 9216 bytes ] : 27 objs; 0.2 MiB; 4045.5 cum MiB
class 53 [ 16384 bytes ] : 7 objs; 0.1 MiB; 4045.6 cum MiB
class 54 [ 20480 bytes ] : 1 objs; 0.0 MiB; 4045.7 cum MiB
class 57 [ 32768 bytes ] : 13 objs; 0.4 MiB; 4046.1 cum MiB
class 61 [ 65536 bytes ] : 1 objs; 0.1 MiB; 4046.1 cum MiB
class 85 [ 262144 bytes ] : 1 objs; 0.2 MiB; 4046.4 cum MiB
------------------------------------------------
PageHeap: 26 sizes; 251.2 MiB free; 121.0 MiB unmapped
------------------------------------------------
     1 pages * 27211 spans ~ 212.6 MiB; 212.6 MiB cum; unmapped: 5.4 MiB; 5.4 MiB cum
     2 pages * 3062 spans ~ 47.8 MiB; 260.4 MiB cum; unmapped: 12.3 MiB; 17.7 MiB cum
     3 pages * 1018 spans ~ 23.9 MiB; 284.3 MiB cum; unmapped: 18.4 MiB; 36.1 MiB cum
     4 pages * 721 spans ~ 22.5 MiB; 306.8 MiB cum; unmapped: 20.1 MiB; 56.2 MiB cum
     5 pages * 175 spans ~ 6.8 MiB; 313.7 MiB cum; unmapped: 6.8 MiB; 63.0 MiB cum
     6 pages * 113 spans ~ 5.3 MiB; 319.0 MiB cum; unmapped: 5.3 MiB; 68.3 MiB cum
     7 pages * 50 spans ~ 2.7 MiB; 321.7 MiB cum; unmapped: 2.7 MiB; 71.0 MiB cum
     8 pages * 101 spans ~ 6.3 MiB; 328.0 MiB cum; unmapped: 6.1 MiB; 77.1 MiB cum
     9 pages * 23 spans ~ 1.6 MiB; 329.6 MiB cum; unmapped: 1.6 MiB; 78.7 MiB cum
    10 pages * 18 spans ~ 1.4 MiB; 331.0 MiB cum; unmapped: 1.4 MiB; 80.1 MiB cum
    11 pages * 10 spans ~ 0.9 MiB; 331.9 MiB cum; unmapped: 0.9 MiB; 80.9 MiB cum
    12 pages * 11 spans ~ 1.0 MiB; 332.9 MiB cum; unmapped: 1.0 MiB; 82.0 MiB cum
    13 pages * 1 spans ~ 0.1 MiB; 333.0 MiB cum; unmapped: 0.1 MiB; 82.1 MiB cum
    14 pages * 2 spans ~ 0.2 MiB; 333.2 MiB cum; unmapped: 0.2 MiB; 82.3 MiB cum
    15 pages * 3 spans ~ 0.4 MiB; 333.6 MiB cum; unmapped: 0.4 MiB; 82.6 MiB cum
    16 pages * 3 spans ~ 0.4 MiB; 334.0 MiB cum; unmapped: 0.2 MiB; 82.9 MiB cum
    17 pages * 1 spans ~ 0.1 MiB; 334.1 MiB cum; unmapped: 0.0 MiB; 82.9 MiB cum
    20 pages * 1 spans ~ 0.2 MiB; 334.2 MiB cum; unmapped: 0.2 MiB; 83.0 MiB cum
    21 pages * 1 spans ~ 0.2 MiB; 334.4 MiB cum; unmapped: 0.2 MiB; 83.2 MiB cum
    29 pages * 1 spans ~ 0.2 MiB; 334.6 MiB cum; unmapped: 0.2 MiB; 83.4 MiB cum
    49 pages * 18 spans ~ 6.9 MiB; 341.5 MiB cum; unmapped: 6.9 MiB; 90.3 MiB cum
    58 pages * 1 spans ~ 0.5 MiB; 342.0 MiB cum; unmapped: 0.5 MiB; 90.8 MiB cum
    75 pages * 1 spans ~ 0.6 MiB; 342.6 MiB cum; unmapped: 0.6 MiB; 91.4 MiB cum
    98 pages * 8 spans ~ 6.1 MiB; 348.7 MiB cum; unmapped: 6.1 MiB; 97.5 MiB cum
   100 pages * 1 spans ~ 0.8 MiB; 349.5 MiB cum; unmapped: 0.8 MiB; 98.3 MiB cum
   122 pages * 1 spans ~ 1.0 MiB; 350.4 MiB cum; unmapped: 1.0 MiB; 99.2 MiB cum
>255 large * 13 spans ~ 21.8 MiB; 372.2 MiB cum; unmapped: 21.8 MiB; 121.0 MiB cum

[root@orange-11601 ~]#
[root@orange-11601 ~]# /opt/couchbase/bin/cbstats localhost:11210 -b default memory
bytes: 8468018888
ep_kv_size: 8190890236
ep_max_data_size: 13212057600
ep_mem_high_wat: 11230248960
ep_mem_low_wat: 9909043200
ep_mem_tracker_enabled: true
ep_oom_errors: 0
ep_overhead: 128884714
ep_tmp_oom_errors: 0
ep_value_size: 6628874716
mem_used: 8468018888
tcmalloc_current_thread_cache_bytes: 2581856
tcmalloc_max_thread_cache_bytes: 4194304
tcmalloc_unmapped_bytes: 126844928
total_allocated_bytes: 14068670960
total_fragmentation_bytes: 5313338896
total_free_bytes: 263061504
total_heap_bytes: 19645071360

added screenshot of the cluster

[MB-8272] there is README file from Erlang distro (on Windows) that should be removed

Thu, 16 May 2013 17:11:12 -0500

In couchbase 2.0.2-xx for mac and windows, there is a README file at root directory. I don't see this README file in rpm and deb package after install.
Is it what we intend to do for rpm and deb package or the bug?

[MB-8271] Document usage examples of vbuckettool

Tue, 14 May 2013 03:28:16 -0500

http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-cmdline-vbuckettool.html

There are a few other command-line tools that need similar attention.

[MB-8270] node down with error net_kernal_terminated

Fri, 17 May 2013 17:39:20 -0500

Environment:
  4 windows server r2 64-bit with 4 core, 4GB RAM for each server

Couchbase:
  couchbase server version 2.0.2-801

Run sanity test on windows. When investigate log file for bug ~~MB-8231~~, I see error saying node down with reason "net_kernel_terminated"

2013-05-13 17:59:28: (agent_config.c.705) ERROR: bad JSON configuration from http://127.0.0.1:8091/pools/default/saslBucketsStreaming: Number of vBuckets must be a power of two > 0 and <= 65536 ({
"name": "default",
"nodeLocator": "vbucket",
"saslPassword": "",
"nodes": [{
"hostname": "127.0.0.1:8091",
"ports": {
"direct": 11210,
"proxy": 11211
}
}],
"vBucketServerMap": {
"hashAlgorithm": "CRC",
"numReplicas": 1,
"serverList": ["127.0.0.1:11210"],
"vBucketMap": []
}
})
EOL on stdin. Exiting
[menelaus:info,2013-05-13T18:02:35.839,ns_1@127.0.0.1:<0.20352.3>:menelaus_web_buckets:handle_bucket_delete:345]Deleted bucket "default"

[ns_server:debug,2013-05-13T18:02:36.980,ns_1@127.0.0.1:ns_config_log<0.284.0>:ns_config_log:log_common:111]config change:
memory_quota ->
2184
[ns_server:debug,2013-05-13T18:02:36.980,ns_1@127.0.0.1:ns_config_rep<0.308.0>:ns_config_rep:do_push_keys:317]Replicating some config keys ([memory_quota]..)
[cluster:debug,2013-05-13T18:02:38.136,ns_1@127.0.0.1:ns_cluster<0.273.0>:ns_cluster:handle_call:135]handling add_node("10.3.2.142", 8091, ..)

[cluster:info,2013-05-13T18:02:38.136,ns_1@127.0.0.1:ns_cluster<0.273.0>:ns_cluster:do_change_address:315]Decided to change address to "10.3.2.143"

[user:warn,2013-05-13T18:02:38.136,nonode@nohost:ns_node_disco<0.301.0>:ns_node_disco:handle_info:168]Node nonode@nohost saw that node 'ns_1@127.0.0.1' went down. Details: [{nodedown_reason,
                                                                        net_kernel_terminated}]
[ns_server:info,2013-05-13T18:02:38.136,nonode@nohost:dist_manager<0.263.0>:dist_manager:handle_call:249]Adjusted IP to "10.3.2.143"
[ns_server:info,2013-05-13T18:02:38.136,nonode@nohost:dist_manager<0.263.0>:dist_manager:bringup:227]Attempting to bring up net_kernel with name 'ns_1@10.3.2.143'
[error_logger:info,2013-05-13T18:02:38.136,nonode@nohost:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================PROGRESS REPORT=========================
          supervisor: {local,net_sup}
             started: [{pid,<0.4435.4>},
                       {name,erl_epmd},
                       {mfargs,{erl_epmd,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,2000},
                       {child_type,worker}]

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-801-rel.setup.exe.manifest.xml

Link to collect info of all node https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_2/2013_05/4nodes-202-801_reb_hang_20130513-185157.tgz

[MB-8269] rebalance hang but log page shows it is completed

Thu, 16 May 2013 20:26:34 -0500

Environment:
windows server 2008 R2 64bit

Build: couchbase server 2.0.2-801

Run rebalance test
./testrunner -i fournode.ini -t failovertests.FailoverTests.test_failover_normal,replica=2,load_ratio=1

rebalance hang with zero percent progress but log page shows it is completed

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-801-rel.setup.exe.manifest.xml

Link to cbcollect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_2/2013_05/4nodes-202-801_reb_hang_20130513-185157.tgz

[MB-8268] flag value not retained after items restored through cbrecovery

Tue, 14 May 2013 23:40:20 -0500

Flag value of items don't match after they're recovered through cbrecovery.

- Set the flag value at 100 during the loading phase
- During verification:
Exception: Bad result for flag value: 1677721600 != the value we set: 100

- Set the flag value at 2 during the loading phase
- During verification:
Exception: Bad result for flag value: 33554432 != the value we set: 2

[MB-8267] cbrecovery doesn't keep flag values intact for recovered msgs

Mon, 13 May 2013 23:10:26 -0500

cbrecovery doesn't keep flag values intact for recovered msgs

[MB-8266] erlang crashed with error: "no match of right hand side value {error, closed}...

Fri, 17 May 2013 19:39:43 -0500

Environment:
  4 windows vm with 4 core, 4GB RAM each

Couchbase server version 2.0.2-798

Run sanity test with rebalance is part of the test.
See erlang crash in log
Check to see if there is an erlang crash dump in cluster => no core dump.

Error in diags

=========================CRASH REPORT=========================
  crasher:
    initial call: erlang:apply/2
    pid: <0.13944.1>
    registered_name: []
    exception error: no match of right hand side value {error,closed}
      in function mc_binary:quick_stats_recv/3
      in call from mc_binary:quick_stats_loop/5
      in call from mc_binary:quick_stats/5
      in call from ns_memcached:do_handle_call/3
      in call from ns_memcached:worker_loop/3
    ancestors: ['ns_memcached-default','single_bucket_sup-default',
                  <0.13913.1>]
    messages: []
    links: [<0.13929.1>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 17711
    stack_size: 24
    reductions: 8661
  neighbours:

Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.2-798-rel.setup.exe.manifest.xml

[MB-8265] need redirect the page to 2.0 manual if user types direct url address for 2.0.1 manual

Thu, 16 May 2013 19:57:18 -0500

We have manual page at http://www.couchbase.com/docs/couchbase-manual-2.0/

If users want to look for manual of 2.0.1 version and type in directly in url as below
http://www.couchbase.com/docs/couchbase-manual-2.0.1/

The page shows error. I should redirect to 2.0 manual page

[MB-8264] there is not couchbase homepage link in manual page of 2.0

Tue, 14 May 2013 14:13:51 -0500

there is not couchbase homepage link in manual page of 2.0

[MB-8263] [system test] Erlang crash during data access phase with Mike's toy build

Tue, 14 May 2013 18:33:19 -0500

cluster ip is 10.5.2.30

1.3 node cluster, create 2 buckets, default 2.5G quota and saslbucket 1.3G
2.load items into buckets to make it dgm. Around 80% resident ratio
3.After initial loading, do a lot of "gets", around 80%. 20% cache miss maximum. Total ops per sec is 3k ops/sec
During data access phase, nodes went down in "Pend" state. bucket is not accessible from UI.

Find erlang_crash.dump on 10.5.2.31 under /opt/couchbase/var/lib/couchbase.

Link to the core_dump file: https://s3.amazonaws.com/bugdb/jira/MB-8263/erl_crash.dump.05-10-2013-19:05:36.7640

[MB-8262] raise a warning if more than 10 views are created for a bucket

Mon, 13 May 2013 19:18:27 -0500

raise a warning if more than 10 views are created for a bucket

[MB-8261] Checking for purged deletions and update the view accordingly

Dependency — Mon, 13 May 2013 19:12:25 -0500

Checking for purged deletions and update the view accordingly

[MB-8260] Password encryption for SASL buckets over the wire from Client to Server

Mon, 13 May 2013 19:16:58 -0500

Currently, SASL passwords are sent in plain text over the wire when client requests data from the server.
This feature is requested to encrypt the passwords used for SASL buckets

[MB-8259] Disk not flush on xdcr dest node

Fri, 17 May 2013 13:39:20 -0500

On the ec2 cluster I used for cbrecovery, the disk write queue didn't flush for a very long time. (this symptom was observed twice)

It's an xdcr dest cluster, no other ops rather than xdcr incoming traffic.

Fired as blocker, since it blocks the cbrecovery testing. The clusters are left intact as they are for developer to diagnose, please email back to me when the problem got resolved then I could continue the test. Thanks!

Ronnie

Destination:
http://ec2-184-169-190-197.us-west-1.compute.amazonaws.com:8091

And the source:
http://ec2-23-21-15-103.compute-1.amazonaws.com:8091

User: Administrator
Passwd: password

SSH:
user: root
passwd: couchbase

Now attach the original email thread with diagnosis from Jin and Junyi:

This should be a bug. There are a lot of crash in ns_server/baby_sitter and couchdb/writer process. Couchdb is unable to spawn writer process for some reasons.

I am not 100% sure which caused which but It should be unrelated to XDCR.

Please file a 2.0.2 bug and assign to Filipe to take first look. Thanks.

Junyi

On May 13, 2013, at 10:56 AM, Jin Lim wrote:

Logs from the ec2-184-169-190-197.us-west-1.compute.amazonaws.com. I see following error. After the error I see couch_db/updater kept crashing. I will wait until Junyi's quick looking at it from the xdcr side then figure out what is next.

Thanks,
Jin

[ns_server:info,2013-05-10T18:00:22.665,ns_1@ec2-184-169-190-197.us-west-1.compute.amazonaws.com:ns_config_log<0.562.0>:ns_config_log:handle_info:57]config change: rest_creds -> ********
[user:info,2013-05-10T18:00:22.718,ns_1@ec2-184-169-190-197.us-west-1.compute.amazonaws.com:<0.570.0>:ns_log:crash_consumption_loop:64]Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: WARNING: curl error: transfer closed with outstanding read data remaining from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
WARNING: curl error: couldn't connect to host from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
ERROR: could not contact REST server(s): http://127.0.0.1:8091/pools/default/saslBucketsStreaming
EOL on stdin. Exiting
[user:info,2013-05-10T18:00:32.169,ns_1@ec2-184-169-190-197.us-west-1.compute.amazonaws.com:<0.570.0>:ns_log:crash_consumption_loop:64]Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: 2013-05-10 18:00:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2013-05-10 18:00:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)

On May 13, 2013, at 10:41 AM, Ronnie Sun wrote:

Hi Jin, Junyi

On the ec2 cluster I used for cbrecovery, the disk write queue didn't flush for a very long time. (this symptom was observed twice)

It's an xdcr dest cluster, no other ops rather than xdcr incoming traffic.

Everything looks normal, not sure if it's related to the recent reb regression. Would you please take a look at it, thanks!

Ronnie

Destination:
http://ec2-184-169-190-197.us-west-1.compute.amazonaws.com:8091

And the source:
http://ec2-23-21-15-103.compute-1.amazonaws.com:8091

User: Administrator
Passwd: password

SSH:
user: root
passwd: couchbase

[MB-8258] Complete Metadata ejection when item becomes non-resident

Mon, 13 May 2013 19:19:57 -0500

Memory overhead from an item's metadata would be huge in case of a very large number of items. Moving the metadata to the value blob allows us to evict the metadata with the value together and fetch it from disk on demand.

[MB-8257] [system test] Frequent memcached disconnected due to timeout error during long term loading

Thu, 16 May 2013 19:44:36 -0500

Cluster ip is 172.23.105.23
1. create 8 nodes cluster, each node has 12G RAM, HHD
2. create 2 buckets default and saslbucket, with memory quota 6G and 4G
3. Run the KV use case for 1 day:
loading 35M items to each bucket, make both buckets into 70% resident ratio, access the data 4k ops/sec with 5% create, 5% delete, 5%expire, 5% update, 80% gets for several hours. Then with the same work load, run some rebalance and failover operations. Works good.
4 Continue the same workload with 15k ops/sec for another 3 days, then detect memcached disconnect and restart from time to time among those cluster nodes in saslbucket. Saslbucket is around 66M items with 30% resident ratio.

Bucket "saslbucket" loaded on node 'ns_1@172.23.105.30' in 0 seconds. ns_memcached001 ns_1@172.23.105.30 12:43:56 - Mon May 13, 2013
Control connection to memcached on 'ns_1@172.23.105.30' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
cmd_binary_vocal_recv,
5},
{mc_client_binary,
select_bucket,
2},
{ns_memcached,
ensure_bucket,
2},
{ns_memcached,
handle_info,
2},
{gen_server,
handle_msg,
5},
{ns_memcached,
init,
1},
{gen_server,
init_it,
6},
{proc_lib,
init_p_do_apply,
3}]} ns_memcached004 ns_1@172.23.105.30 12:43:56 - Mon May 13, 2013

[MB-8256] snapshotStats crash during shutdown bucket

Mon, 13 May 2013 20:14:54 -0500

scheduled snapshotStats task is picked up and being executed during the bucket destruction. This unfortunately cause a crash for invalid reference to a being-deallocated internal structure (epstat, configuration, etc).

[MB-8255] Update Sizing Calculator for 2.0.2

Tue, 14 May 2013 15:26:34 -0500

Will need help on Javascript portion.

Schedules meeting with Anil to get new stats for HTML file.

[MB-8254] windows buildbot failed to build windows version 2.0.2-800

Duplicate — Mon, 13 May 2013 14:25:00 -0500

http://builds.hq.northscale.net:8010/builders/cs-win2008-x64-20-builder-202/builds/110

*** searching for required modules ***
Makefile:201: recipe for target `package-win' failed
Traceback (most recent call last):
  File "dist_setup.py", line 9, in <module>
    zipfile=None)
  File "C:\Python27\lib\distutils\core.py", line 152, in setup
    dist.run_commands()
  File "C:\Python27\lib\distutils\dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:\Python27\lib\site-packages\py2exe\build_exe.py", line 243, in run
    self._run()
  File "C:\Python27\lib\site-packages\py2exe\build_exe.py", line 296, in _run
    self.find_needed_modules(mf, required_files, required_modules)
  File "C:\Python27\lib\site-packages\py2exe\build_exe.py", line 1297, in find_needed_modules
    mf.import_hook(mod)
  File "C:\Python27\lib\site-packages\py2exe\mf.py", line 719, in import_hook
    return Base.import_hook(self,name,caller,fromlist,level)
  File "C:\Python27\lib\site-packages\py2exe\mf.py", line 136, in import_hook
    q, tail = self.find_head_package(parent, name)
  File "C:\Python27\lib\site-packages\py2exe\mf.py", line 204, in find_head_package
    raise ImportError, "No module named " + qname
ImportError: No module named Cheetah
./util.rb:69:in `hard_sh': ERROR: cmd failed: /cygdrive/c/Python27/python dist_setup.py py2exe (RuntimeError)
ERROR: with result: false, 256
from /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1010:in `call'
from /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1010:in `sh'
from /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1094:in `sh'
from ./util.rb:66:in `hard_sh'
from ./server-win.rb:228:in `py2exe'
from ./server-win.rb:332
from ./server-win.rb:329:in `each'
from ./server-win.rb:329
from ./server-win.rb:328:in `each'
from ./server-win.rb:328
make: *** [package-win] Error 1
program finished with exit code 2
elapsedTime=3146.141000

[MB-8253] windows 2.0.2 builds failing due to Cheetah not being installed correctly

Duplicate — Mon, 13 May 2013 18:37:37 -0500

The Cheetah python module is missing from windows build machines, causing the build to fail with:

/cygdrive/c/Python27/python dist_setup.py py2exe
running py2exe
creating C:\cygwin\opt\couchbase\bin\install\build
creating C:\cygwin\opt\couchbase\bin\install\build\bdist.win-amd64
creating C:\cygwin\opt\couchbase\bin\install\build\bdist.win-amd64\winexe
creating C:\cygwin\opt\couchbase\bin\install\build\bdist.win-amd64\winexe\collect-2.7
creating C:\cygwin\opt\couchbase\bin\install\build\bdist.win-amd64\winexe\bundle-2.7
creating C:\cygwin\opt\couchbase\bin\install\build\bdist.win-amd64\winexe\temp
creating C:\cygwin\opt\couchbase\bin\install\dist
*** searching for required modules ***
Makefile:201: recipe for target `package-win' failed
Traceback (most recent call last):
. . .
ImportError: No module named Cheetah
./util.rb:69:in `hard_sh': ERROR: cmd failed: /cygdrive/c/Python27/python dist_setup.py py2exe (RuntimeError)
ERROR: with result: false, 256
from /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1010:in `call'
from /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1010:in `sh'
from /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1094:in `sh'
from ./util.rb:66:in `hard_sh'
from ./server-win.rb:228:in `py2exe'
from ./server-win.rb:332
from ./server-win.rb:329:in `each'
from ./server-win.rb:329
from ./server-win.rb:328:in `each'
from ./server-win.rb:328
make: *** [package-win] Error 1
program finished with exit code 2

[MB-8252] Docs: How to work with data across different client languages

Mon, 13 May 2013 10:56:05 -0500

There are a few cases where the customer wants to use the same data from different languages and it would be helpful to provide some documentation on how to do this. There are certain things like serialization, etc to take into account.

[MB-8251] Docs: how to handle "document is too large" exception across various SDK's

Mon, 13 May 2013 10:54:59 -0500

A specific request for dealing with the exception that comes back when a document is too large...it probably differs across all client libraries

[MB-8250] Doc: Need description and best practices on how to add a field/column to all documents within a bucket

Mon, 13 May 2013 10:54:26 -0500

The two strategies are:
-Update the application to "work" with both types of schemas and add the field to documents as they are accessed
-Create a view to find documents that don't have the new field, and then iterrate over every document to do it.

Want a description on what's being accomplished, why, how, etc including examples in different client libraries.

[MB-8249] Docs: link in dev guide doesn't go anywhere

Mon, 13 May 2013 11:18:57 -0500

This page: http://www.couchbase.com/docs/couchbase-devguide-2.0/get-and-lock.html

Has a link to "memcached protocol" which doesn't go anywhere: http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-architecture-apis-memcached-protocol.html

[MB-8248] GetAndLock doesn't prevent a regular "get" from succeeding

Mon, 13 May 2013 11:19:30 -0500

In the dev guide (as well as maybe other places?): http://www.couchbase.com/docs/couchbase-devguide-2.0/get-and-lock.html

The text at the beginning says that getandlock will prevent the retrieval of an item...that will only be the case if getandlock is also used to retrieve it, but a normal get will still succeed.

[MB-8247] Provide documentation and examples on using map-reduce views versus elastic search

Mon, 13 May 2013 09:45:01 -0500

Coming up a lot is the request for "when do I use views and when do I use elastic search"

[MB-8246] [system test] Rebalance exited with reason timeout waiting for backfill determination

Fri, 17 May 2013 17:28:55 -0500

Cluster ip is 172.23.105.23
1. create 8 nodes cluster, each node has 12G RAM, HHD
2. create 2 buckets default and saslbucket, with memory quota 6G and 4G
3. Run the KV use case for 1 day:
loading 35M items to each bucket, access the data 4k ops/sec with 5% create, 5% delete, 5%expire, 5% update, 80 gets for several hours. Then with the same work load, run some rebalance and failover operations. Works good.
4 Continue the workload for another day, then try to rebalance in one node, rebalance exit with time out.

Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.7961.56>,
{{badmatch,
[{'EXIT',
{timeout,
{gen_server,call,
[<12869.26294.2>,had_backfill,30000]}}}]},
[{ns_single_vbucket_mover,
'-wait_backfill_determination/1-fun-1-',
1}]}}}
ns_orchestrator002 ns_1@172.23.105.23 14:36:20 - Fri May 10, 2013
<0.7953.56> exited with {unexpected_exit,
{'EXIT',<0.7961.56>,
{{badmatch,
[{'EXIT',
{timeout,
{gen_server,call,
[<12869.26294.2>,had_backfill,30000]}}}]},
[{ns_single_vbucket_mover,
'-wait_backfill_determination/1-fun-1-',1}]}}}

The link for diags is https://s3.amazonaws.com/bugdb/jira/MB-8246/ns-diag-20130510162640.txt.zip

[MB-8245] During rpm upgrade, encountered error while copying opt/couchbase/var/lib/couchbase/config/config.dat (already exists)

Wed, 15 May 2013 17:49:19 -0500

Steps:
-linux centos64
-upgraded (rpm -Uvh) from 2.0.2 build 786 to 795.
-did not shutdown service.
-Encountered error. It failed to copy /opt/couchbase/var/lib/couchbase/config/config.dat
see below.

-rw-r--r-- 1 root root 127886701 May 4 00:26 couchbase-server-enterprise_x86_64_2.0.2-786-rel.rpm
drwxr-xr-x 3 root root 4096 May 6 17:46 mm
-rw-r--r-- 1 root root 128296356 May 10 09:54 couchbase-server-enterprise_x86_64_2.0.2-795-rel.rpm
[root@cen-1910 ~]# rpm -Uvh couchbase-server-enterprise_x86_64_2.0.2-795-rel.rpm
Preparing... ########################################### [100%]
Stopping couchbase-server
Minimum RAM required : 4 GB
System RAM configured : 8174464 kB

Minimum number of processors required : 4 cores
Number of processors on the system : 4 cores

   1:couchbase-server ########################################### [100%]
Upgrading couchbase-server ...
  /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes
Automatic mode: running without interactive questions or confirmations.
Analysing...
Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat
Target node: ns_1@10.3.2.49
Upgrading from 2.0

Couchbase should not be running.
  Please use: /etc/init.d/couchbase-server stop

Database dir: /opt/couchbase/var/lib/couchbase/data

Buckets to upgrade: beer-sample,gamesim-sample,default

Checking disk space available for buckets in directory:
  /opt/couchbase/var/lib/couchbase/data
  Free disk bucket space wanted: 0.0
  Free disk bucket space available: 15459573760
  Free disk space factor: 2.0
  Ok.

Analysis complete.

Copying /opt/couchbase/var/lib/couchbase/config/config.dat
    cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat
Error: /opt/couchbase/bin/install/../../var/lib/couchbase/ip already exists while copying /opt/couchbase/var/lib/couchbase/ip.rpmsave
Starting couchbase-server[ OK ]

You have successfully installed Couchbase Server.
Please browse to http://cen-1910:8091/ to configure your server.
Please refer to http://couchbase.com for additional resources.

Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091, 8092 and from 21100 to 21299.

By using this software you agree to the End User License Agreement.
See /opt/couchbase/LICENSE.txt.

[root@cen-1910 ~]# rpm -qa|grep couchbase
couchbase-server-2.0.2-795
[root@cen-1910 ~]#

[MB-8244] add or enable dispatcher stats for each multi-readers/writers thread

Wed, 15 May 2013 11:56:36 -0500

add or enable dispatcher stats for each multi-readers/writers thread

[MB-8243] memcached crashed in Configuration::getCouchBucket

Fri, 17 May 2013 16:46:58 -0500

I see crash from http://qa.hq.northscale.net/job/CouchbaseServer-SanityTest-4Node-
Centos64/63/consoleFull run on build 2.0.2-795-rel

seems it's the test:
./testrunner -i fournode.ini -t viewquerytests.ViewQueryTests.test_employee_dataset_all_queries,limit=1000,docs-per-day=2,wait_persistence=true

but this job doesn't generate collect info after failures, and I have logs that have been obtained much later

gdb /opt/couchbase/bin/memcached core.memcached.7524
GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 7705]
[New Thread 8584]
[New Thread 8583]
[New Thread 8582]
[New Thread 7709]
[New Thread 7708]
[New Thread 7707]
[New Thread 7706]
[New Thread 7704]
[New Thread 7703]
[New Thread 7702]
[New Thread 7701]
[New Thread 7539]
[New Thread 7538]
[New Thread 7537]
[New Thread 7536]
[New Thread 7535]
[New Thread 7533]
[New Thread 7532]
[New Thread 7524]

warning: .dynamic section for "/usr/lib64/libstdc++.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff9dbe8000
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x0000003866c30285 in raise () from /lib64/libc.so.6
(gdb) t aa bt
A syntax error in expression, near `bt'.
(gdb) t a a bt

Thread 20 (Thread 0x2af656d41220 (LWP 7524)):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6000, tv=<value optimized out>) at epoll.c:404
#2 0x00002af6568b2e44 in event_base_loop (base=0x222b6000, flags=<value optimized out>) at event.c:1558
#3 0x00000000004097d6 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7926

Thread 19 (Thread 7532):
#0 0x0000003866cc545b in read () from /lib64/libc.so.6
#1 0x0000003866c6b677 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x0000003866c6c03e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x0000003866c61124 in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x0000003866c5ffc9 in fgets () from /lib64/libc.so.6
#5 0x00002af656d42939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#6 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#7 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 18 (Thread 7533):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0x1da4e040) at extensions/loggers/file_logger.c:368
#2 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#3 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 17 (Thread 7535):
#0 0x00002af656b17f6f in tcmalloc::CentralFreeList::Populate() () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#1 0x00002af656b17c93 in tcmalloc::CentralFreeList::FetchFromSpansSafe() () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#2 0x00002af656b17bcc in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#3 0x00002af656b1ccd5 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#4 0x00002af656b12054 in tcmalloc::ThreadCache::Allocate(unsigned long, unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#5 0x00002af656b10b60 in (anonymous namespace)::do_malloc(unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#6 0x00002af656b10892 in (anonymous namespace)::do_malloc_or_cpp_alloc(unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#7 0x00002af656b10bec in (anonymous namespace)::do_calloc(unsigned long, unsigned long) () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#8 0x00002af656b24764 in tc_calloc () from /opt/couchbase/lib/libtcmalloc_minimal.so.4
#9 0x00002aaaaaef94b9 in HashTable (this=<value optimized out>, i=918, newState=vbucket_state_replica, st=..., checkpointConfig=..., checkpointId=1, initState=vbucket_state_dead)
    at src/stored-value.hh:814
#10 VBucket::VBucket (this=<value optimized out>, i=918, newState=vbucket_state_replica, st=..., checkpointConfig=..., checkpointId=1, initState=vbucket_state_dead) at src/vbucket.hh:126
#11 0x00002aaaaaefb932 in EventuallyPersistentStore::setVBucketState (this=0x2a424400, vbid=918, to=vbucket_state_replica) at src/ep.cc:795
#12 0x00002aaaaaf1e11d in setVBucketState (h=0x2230e900, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.h:508
#13 setVBucket (h=0x2230e900, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.cc:738
#14 processUnknownCommand (h=0x2230e900, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.cc:884
#15 0x00002aaaaaf1e99c in EvpUnknownCommand (handle=<value optimized out>, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at src/ep_engine.cc:1021
#16 0x00002aaaaacc4dd4 in bucket_unknown_command (handle=<value optimized out>, cookie=0x222858c0, request=0x22276800, response=0x4075e0 <binary_response_handler>) at bucket_engine.c:2475
#17 0x0000000000411b4e in process_bin_unknown_packet (c=0x222858c0) at daemon/memcached.c:2882
#18 process_bin_packet (c=0x222858c0) at daemon/memcached.c:3170
#19 complete_nread_binary (c=0x222858c0) at daemon/memcached.c:3744
#20 complete_nread (c=0x222858c0) at daemon/memcached.c:3826
#21 conn_nread (c=0x222858c0) at daemon/memcached.c:5679
#22 0x0000000000405ec5 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x222858c0) at daemon/memcached.c:5942
#23 0x00002af6568b2f3c in event_process_active_single_queue (base=0x222b6500, flags=<value optimized out>) at event.c:1308
#24 event_process_active (base=0x222b6500, flags=<value optimized out>) at event.c:1375
#25 event_base_loop (base=0x222b6500, flags=<value optimized out>) at event.c:1572
#26 0x0000000000414604 in worker_libevent (arg=0x1da51900) at daemon/thread.c:301
#27 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#28 0x0000003866cd325d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 16 (Thread 7536):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6280, tv=<value optimized out>) at epoll.c:404
#2 0x00002af6568b2e44 in event_base_loop (base=0x222b6280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0x1da519f8) at daemon/thread.c:301
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 15 (Thread 7537):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6c80, tv=<value optimized out>) at epoll.c:404
#2 0x00002af6568b2e44 in event_base_loop (base=0x222b6c80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0x1da51af0) at daemon/thread.c:301
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 14 (Thread 7538):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6a00, tv=<value optimized out>) at epoll.c:404
#2 0x00002af6568b2e44 in event_base_loop (base=0x222b6a00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0x1da51be8) at daemon/thread.c:301
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 13 (Thread 7539):
#0 0x0000003866cd3648 in epoll_wait () from /lib64/libc.so.6
#1 0x00002af6568c7576 in epoll_dispatch (base=0x222b6780, tv=<value optimized out>) at epoll.c:404
#2 0x00002af6568b2e44 in event_base_loop (base=0x222b6780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0x1da51ce0) at daemon/thread.c:301
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 12 (Thread 7701):
#0 0x0000003866c99221 in nanosleep () from /lib64/libc.so.6
#1 0x0000003866cccba4 in usleep () from /lib64/libc.so.6
#2 0x00002aaaaaf35125 in updateStatsThread (arg=0x1da4e4c0) at src/memory_tracker.cc:31
#3 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 11 (Thread 7702):
#0 0x000000386740d594 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003867408e8a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003867408d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002aaaaaf3601a in Mutex::acquire (this=0x223310f0) at src/mutex.cc:79
#4 0x00002aaaaaf7d303 in lock (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26595800, vbucketId=936, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#8 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26595800, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#9 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#10 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
---Type <return> to continue, or q <return> to quit---
#11 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22361ba0) at src/scheduler.cc:153
#12 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22361ba0) at src/scheduler.cc:34
#13 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#14 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 10 (Thread 7703):
#0 0x000000386740d594 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003867408e8a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003867408d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002aaaaaf3601a in Mutex::acquire (this=0x223310f0) at src/mutex.cc:79
#4 0x00002aaaaaf7d303 in lock (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26595200, vbucketId=937, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#8 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26595200, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#9 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#10 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#11 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22361a00) at src/scheduler.cc:153
#12 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22361a00) at src/scheduler.cc:34
#13 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#14 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 9 (Thread 7704):
#0 0x000000386740d594 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003867408e8a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003867408d4c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002aaaaaf3601a in Mutex::acquire (this=0x223310f0) at src/mutex.cc:79
#4 0x00002aaaaaf7d303 in lock (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26594c00, vbucketId=934, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#8 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26594c00, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#9 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#10 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#11 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22361860) at src/scheduler.cc:153
#12 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22361860) at src/scheduler.cc:34
#13 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#14 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 8 (Thread 7706):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf39631 in wait (this=0x22388d00) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x22388d00) at src/scheduler.cc:139
#3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388d00) at src/scheduler.cc:34
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 7 (Thread 7707):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf39631 in wait (this=0x22388b60) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x22388b60) at src/scheduler.cc:139
#3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388b60) at src/scheduler.cc:34
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 6 (Thread 7708):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf39631 in wait (this=0x223889c0) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x223889c0) at src/scheduler.cc:139
#3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x223889c0) at src/scheduler.cc:34
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 5 (Thread 7709):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf39631 in wait (this=0x22388820) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x22388820) at src/scheduler.cc:139
#3 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388820) at src/scheduler.cc:34
#4 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 4 (Thread 8582):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf0fe0f in wait (this=0x2230e900) at src/syncobject.hh:57
#2 wait (this=0x2230e900) at src/syncobject.hh:73
#3 wait (this=0x2230e900) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x2230e900) at src/ep_engine.cc:3379
#5 0x00002aaaaaf0fef3 in EvpNotifyPendingConns (arg=0x2230e900) at src/ep_engine.cc:1153
#6 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#7 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 3 (Thread 8583):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef3688 in wait (this=0x222aa1b0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x222aa1b0, d=...) at src/dispatcher.cc:342
#3 0x00002aaaaaef61ea in Dispatcher::run (this=0x22313c00) at src/dispatcher.cc:184
#4 0x00002aaaaaef69ad in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
#5 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 2 (Thread 8584):
#0 0x000000386740b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef3688 in wait (this=0x222ab0e0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x222ab0e0, d=...) at src/dispatcher.cc:342
#3 0x00002aaaaaef61ea in Dispatcher::run (this=0x22313500) at src/dispatcher.cc:184
#4 0x00002aaaaaef69ad in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
#5 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003866cd325d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x49688940 (LWP 7705)):
#0 0x0000003866c30285 in raise () from /lib64/libc.so.6
#1 0x0000003866c31d30 in abort () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#2 0x00002aaaaaf36070 in Mutex::acquire (this=0x2230e828) at src/mutex.cc:83
#3 0x00002aaaaaf83fe8 in lock (this=0x2230e828, key="couch_bucket") at src/locks.hh:48
#4 LockHolder (this=0x2230e828, key="couch_bucket") at src/locks.hh:26
#5 Configuration::getString (this=0x2230e828, key="couch_bucket") at src/configuration.cc:38
#6 0x00002aaaaaf8e5eb in Configuration::getCouchBucket (this=0x2230e828) at src/generated_configuration.cc:71
#7 0x00002aaaaaf7c59e in CouchNotifier::selectBucket (this=0x22331000) at src/couch-kvstore/couch-notifier.cc:721
#8 0x00002aaaaaf7cc0f in CouchNotifier::processInput (this=0x22331000) at src/couch-kvstore/couch-notifier.cc:606
#9 0x00002aaaaaf7c199 in maybeProcessInput (this=0x22331000, rh=0x22385540) at src/couch-kvstore/couch-notifier.cc:546
#10 CouchNotifier::sendCommand (this=0x22331000, rh=0x22385540) at src/couch-kvstore/couch-notifier.cc:439
#11 0x00002aaaaaf7d478 in CouchNotifier::notify_update (this=0x22331000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:774
#12 0x00002aaaaaf73183 in CouchKVStore::setVBucketState (this=0x26594600, vbucketId=935, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#13 0x00002aaaaaf74089 in CouchKVStore::snapshotVBuckets (this=0x26594600, vbstates=Traceback (most recent call last):
  File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
    nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#14 0x00002aaaaaefc2b3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2a424400, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#15 0x00002aaaaaf54bef in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#16 0x00002aaaaaf397a0 in ExecutorThread::run (this=0x22388ea0) at src/scheduler.cc:153
#17 0x00002aaaaaf39ebd in launch_executor_thread (arg=0x22388ea0) at src/scheduler.cc:34
#18 0x000000386740677d in start_thread () from /lib64/libpthread.so.0
#19 0x0000003866cd325d in clone () from /lib64/libc.so.6

core file:
root(couchbase)@10.3.3.30:/tmp/core.memcached.7524

[MB-8242] Lazy computed outbound XDCR stats per replication are incorrect

Thu, 16 May 2013 19:53:11 -0500

1. Setup the Longevity cluster, West Master to East Master and West Master to West3 cluster on bucket0.
2. Load items,
3. Wait for replication to catch up for both the outbound replication streams.

Notice that"Percentage complete" stats are exactly identical (incorrect) for the 2 replications.
( See attached screenshot)

1. Percentage Complete - Although it is 100 percent replicated, the replication2 shows the same number as the replication1 which is still replicating

Cluster here : http://guinep-s10501:8091

[MB-8241] Refactor set_view code

Fri, 17 May 2013 06:36:44 -0500

Refactor the MapReduce specific code into its own module. The goal is to have major code sharing between the MapReduce and that Spatial Views.

[MB-8240] memslap vbucketkeygen vbuckettool is not working with MAC for latest build

Fri, 17 May 2013 18:03:55 -0500

Chishengs-MacBook-Pro:tools chisheng$ ./memslap -h
dyld: Library not loaded: /opt/couchbase/lib/libmemcached.6.dylib
  Referenced from: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/./memslap
  Reason: image not found
Trace/BPT trap: 5

Chishengs-MacBook-Pro:tools chisheng$ ./vbuckettool -h
dyld: Library not loaded: /opt/couchbase/lib/libvbucket.1.dylib
  Referenced from: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/./vbuckettool
  Reason: image not found
Trace/BPT trap: 5

Chishengs-MacBook-Pro:tools chisheng$ ./vbucketkeygen -h
dyld: Library not loaded: /opt/couchbase/lib/libvbucket.1.dylib
  Referenced from: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/./vbucketkeygen
  Reason: image not found
Trace/BPT trap: 5

[MB-8239] cbcollect_info only collects couchbase.log and misses other logs

Fri, 17 May 2013 17:45:40 -0500

Chishengs-MacBook-Pro:bin chisheng$ ./cbcollect_info
info.zipuname (uname -a) - OK
Directory structure membase - previous versions (ls -lR /opt/membase /var/membase /var/opt/membase /etc/opt/membase) - Exit code 1
sysctl settings (sysctl -a) - OK
Process list snapshot (top -l 1) - OK
Disk activity (iostat 1 10) - OK
Process list (ps -Aww -o user,pid,lwp,ppid,nlwp,pcpu,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - Exit code 1
Network configuration (ifconfig -a) - OK
Taking sample 2 after 10.000000 seconds -
OK
Network status (netstat -an) - OK
Network routing table (netstat -rn) - OK
Arp cache (arp -na) - OK
Filesystem (df -ha) - OK
System activity reporter (sar 1 10) - OK
System paging activity (vmstat 1 10) - Exit code 127
System uptime (uptime) - OK
couchbase user definition (getent passwd couchbase) - Exit code 127
couchbase user limits (su couchbase -c "ulimit -a") - skipped (needs root privs)
membase user definition (getent passwd membase) - Exit code 127
couchbase user limits (su couchbase -c "ulimit -a") - skipped (needs root privs)
membase user limits (su membase -c "ulimit -a") - skipped (needs root privs)
Interrupt status (intrstat 1 10) - Exit code 127
Processor status (mpstat 1 10) - Exit code 127
System log (cat /var/adm/messages) - Exit code 1
Kernel log buffer (dmesg) - Exit code 1
Checking for server guts in /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/var/lib/couchbase/initargs...
./erl: line 28: /lib/erlang/erts-5.8.*/bin/erlexec: No such file or directory
./erl: line 28: exec: /lib/erlang/erts-5.8.*/bin/erlexec: cannot execute: No such file or directory
Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs...
./erl: line 28: /lib/erlang/erts-5.8.*/bin/erlexec: No such file or directory
./erl: line 28: exec: /lib/erlang/erts-5.8.*/bin/erlexec: cannot execute: No such file or directory
Checking for server guts in /Users/chisheng/Library/Application Support/Couchbase/var/lib/couchbase/initargs...
./erl: line 28: /lib/erlang/erts-5.8.*/bin/erlexec: No such file or directory
./erl: line 28: exec: /lib/erlang/erts-5.8.*/bin/erlexec: cannot execute: No such file or directory

[MB-8238] NO menu bar to access menu commands after installation and click the Couchbase icon on MAC

Fri, 17 May 2013 18:03:39 -0500

1. Intall Couchbase server on MAC Lion OS
2. Click Couchbase Icon no access menu poped out

For previous build like 755, we have this access menu. The screen shot is from 755

[MB-8237] Server Error "{what, {noproc, {gen_server,call, ['capi_set_view_manager-saslbucket'"

Tue, 14 May 2013 16:05:40 -0500

Seeing the server errors on "capi_set_view_manager" during indexing + rebalancing on the cluster.

Server error during processing: ["web request failed",
{path,
"/pools/default/buckets/saslbucket/statsDirectory"},
{type,exit},
{what,
{noproc,
{gen_server,call,
['capi_set_view_manager-saslbucket',
{foreach_doc,
#Fun<capi_ddoc_replication_srv.1.36030090>},
infinity]}}},
{trace,
[{gen_server,call,3},
{capi_ddoc_replication_srv,
foreach_live_ddoc_id,2},
{capi_ddoc_replication_srv,fetch_ddoc_ids,
1},
{menelaus_stats,
couchbase_view_stats_descriptions,1},
{menelaus_stats,membase_stats_description,
1},
{menelaus_stats,serve_stats_directory,3},
{menelaus_web_buckets,
checking_bucket_access,4},

Logs at https://s3.amazonaws.com/bugdb/jira/bug-ns-server/bug.tar

[MB-8236] Server error on {global,ns_rebalance_observer},

Tue, 14 May 2013 19:30:45 -0500

Seeing server errors during rebalance on build 789.

1. Setup a 2 node cluster.
2. Create 2 ddocs
3. Continous front end load a+ indexing
4. Start rebalance.

Server error during processing: ["web request failed",
{path,"/pools/default/tasks"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[{global,ns_rebalance_observer},
get_detailed_progress,10000]}}},
{trace,
[{gen_server,call,3},
{ns_rebalance_observer,
get_detailed_progress,0},
{ns_doctor,get_detailed_progress,0},
{ns_doctor,do_build_tasks_list,4},
{menelaus_web,handle_tasks,2},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] (repeated 4 times)

Logs at https://s3.amazonaws.com/bugdb/jira/bug-ns-server/bug.tar

[MB-8235] error restarting couch_server: {{read_loop_died, {problem_reopening_file,

Fri, 17 May 2013 17:24:57 -0500

Junyi > For some reasons, the CouchDB updater crashed during XDCR, and caused cascading results that babysitting proc restart the CouchDB multiple times, and then cause XDCR replicator crashed due to the inconsistent instance startup time (Source database out of sync).

Test topology:
source:bucket0 <- bidirection -> dest1:bucket0
source:bucket0 -> dest2:bucket0
                             dest3:bucket0
                             dest4:bucket0
we have 4 outbound streams from source and 1 inbound.

there is a 20k frontend load on bucket0 with
get:70%,delete:10%,update:10%,set:10%,expire:5%

inbound load from destination is from a 4k load with
get:90%,delete:2%,update:5%,set:5%,expire:5%

data is loaded till about 70% dgm.
No views, no rebalancing.

In Couchdb Logs we see:
[couchdb:error,2013-05-09T7:44:46.318,ns_1@172.23.105.55:couch_server<0.8331.208>:couch_log:error:42]Unexpected message, restarting couch_server: {'EXIT',<0.16961.208>,
                                              {{read_loop_died,
                                                {problem_reopening_file,
                                                 {error,system_limit},
                                                 {set_close_after,infinity,
                                                  <0.16958.208>},
                                                 <0.16959.208>,
                                                 "/opt/couchbase/var/lib/couchbase/data/bucket1/345.couch.2",
                                                 10}},
                                               {gen_server,call,
                                                [<0.16958.208>,snapshot_reads,
                                                 infinity]}}}

In xdcr there are sync errors suggesting we increase max_dbs, but seems we are already hitting a limit as couchdb is restarting:

Replication `a1c985cbafac10e773b130f01d1ba85c/bucket0/bucket0` ...failed: Source database out of sync. Try to increase max_dbs_open at the source's server.

Attaching logs here from time of crash. full logs were copied up and left here:

172.23.105.55:/0509_couchdb/ (use rsa key)

[MB-8232] [Doc'd] Multi-reader/writer

Thu, 16 May 2013 16:38:49 -0500

[Doc'd] Multi-reader/writer

[MB-8231] Rebalance hangs with progress zero percent

Wed, 15 May 2013 17:50:34 -0500

Manifest info:
<manifest>
<remote name="couchbase" fetch="git://github.com/couchbase/"/>
<remote name="membase" fetch="git://github.com/membase/"/>
<remote name="apache" fetch="git://github.com/apache/"/>
<remote name="erlang" fetch="git://github.com/erlang/"/>
<default remote="couchbase" revision="master"/>
<project name="tlm" path="tlm" revision="9f8a97b773c2b97cd63893a84a2fef2562c8860f">
<copyfile src="Makefile.top" dest="Makefile"/>
</project>
<project name="bucket_engine" path="bucket_engine" revision="2a797a8d97f421587cce728f2e6aa2cd42c8fa26"/>
<project name="ep-engine" path="ep-engine" revision="7fa9438353fa783d72e3e7f28e32703ab8922733"/>
<project name="libconflate" path="libconflate" revision="c0d3e26a51f25a2b020713559cb344d43ce0b06c"/>
<project name="libmemcached" path="libmemcached" revision="ca739a890349ac36dc79447e37da7caa9ae819f5" remote="membase"/>
<project name="libvbucket" path="libvbucket" revision="408057ec55da3862ab8d75b1ed25d2848afd640f"/>
<project name="couchbase-cli" path="couchbase-cli" revision="87dcaa935efb0eac4e75d529ab7e3c81b4439e61" remote="couchbase"/>
<project name="memcached" path="memcached" revision="b6ceb46fc26ac6f1d6be7a5866d6c6c0f6e6d32a" remote="membase"/>
<project name="moxi" path="moxi" revision="4b391021af7a453bf88716724d2c644916ebd969"/>
<project name="ns_server" path="ns_server" revision="d77b8c4d9eb27fbd60778ca299cec29bca749e4c"/>
<project name="portsigar" path="portsigar" revision="159b6179ea8a3c2075ee9eb2afa6f91c98c0fda6"/>
<project name="sigar" path="sigar" revision="63a3cd1b316d2d4aa6dd31ce8fc66101b983e0b0"/>
<project name="couchbase-examples" path="couchbase-examples" revision="cd9c8600589a1996c1ba6dbea9ac171b937d3379"/>
<project name="couchbase-python-client" path="couchbase-python-client" revision="d443169c0694fca1be67d8f6934a8c50f0175ee7"/>
<project name="couchdb" path="couchdb" revision="586e4bb73b92db4362192616370c4e3edb8c34a0"/>
<project name="couchdbx-app" path="couchdbx-app" revision="833f5f2491c1d307d4d1ea291fd446a740df83f0"/>
<project name="couchstore" path="couchstore" revision="abc2af1310ca375697e08aad4fa78e5e5d61adcf"/>
<project name="geocouch" path="geocouch" revision="ed9ad43aa361df0829262fef811b5236331b44c8"/>
<project name="testrunner" path="testrunner" revision="2ba2259a62e609b2bbaffc179be2d6c6be6ca61a"/>
<project name="healthchecker" path="healthchecker" revision="72dab0d4f293e80644b38321f001b42846701890"/>
<project name="otp" path="otp" revision="b6dc1a844eab061d0a7153d46e7e68296f15a504" remote="erlang"/>
<project name="icu4c" path="icu4c" revision="26359393672c378f41f2103a8699c4357c894be7" remote="couchbase"/>
<project name="snappy" path="snappy" revision="5681dde156e9d07adbeeab79666c9a9d7a10ec95" remote="couchbase"/>
<project name="v8" path="v8" revision="447decb75060a106131ab4de934bcc374648e7f2" remote="couchbase"/>
<project name="gperftools" path="gperftools" revision="44a584d1de8c89addfb4f1d0522bdbbbed83ba48" remote="couchbase"/>
<project name="pysqlite" path="pysqlite" revision="0ff6e32ea05037fddef1eb41a648f2a2141009ea" remote="couchbase"/>
</manifest>

Rebalance 1 -> 4 hangs sometimes with status stuck at 0% and below logs repeating:

     {move_state,985,
         ['ns_1@10.3.3.100',undefined],
         ['ns_1@10.3.3.98','ns_1@10.3.3.96'],
         [{replica_building_stats,'ns_1@10.3.3.98',1,1,<<>>},
          {replica_building_stats,'ns_1@10.3.3.96',1,1,<<>>}]},
     {move_state,986,
         ['ns_1@10.3.3.100',undefined],
         ['ns_1@10.3.3.98','ns_1@10.3.3.96'],
         [{replica_building_stats,'ns_1@10.3.3.98',1,1,<<>>},
          {replica_building_stats,'ns_1@10.3.3.96',1,1,<<>>}]},
     {move_state,987,
         ['ns_1@10.3.3.100',undefined],
         ['ns_1@10.3.3.98','ns_1@10.3.3.96'],
         [{replica_building_stats,'ns_1@10.3.3.98',1,1,<<>>},
          {replica_building_stats,'ns_1@10.3.3.96',1,1,<<>>}]},
     {move_state,988,
         ['ns_1@10.3.3.100',undefined],
         ['ns_1@10.3.3.98','ns_1@10.3.3.96'],
         [{replica_building_stats,'ns_1@10.3.3.98',1,1,<<>>},
          {replica_building_stats,'ns_1@10.3.3.96',1,1,<<>>}]},

Reproducer test:
./testrunner -i ../ini/vm-4nodes-sanity.ini -t warmupcluster.WarmUpClusterTest.test_warmUpCluster,num_of_docs=1000

Attaching collect info

[MB-8230] Rebalance exited with reason {badarg,[{erlang,'++', [{'EXIT', {{janitor_agent_servant_died,

Mon, 13 May 2013 23:06:55 -0500

Rebalance fails with the below test
./testrunner -i ../ini/vm-4nodes-sanity.ini -t failovertests.FailoverTests.test_failover_normal,replica=2,load_ratio=1

Below crash report is seen in the logs:

=========================CRASH REPORT=========================
  crasher:
    initial call: ns_vbucket_mover:init/1
    pid: <0.1175.16>
    registered_name: []
    exception exit: {badarg,
                     [{erlang,'++',
                       [{'EXIT',
                         {{janitor_agent_servant_died,
                           {{badmatch,{error,closed}},
                            {gen_server,call,
                             ['ns_memcached-default',
                              {get_mass_tap_docs_estimate,
                               [86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,
                                101,102,103,104,105,106,107,108,109,110,111,
                                112,113,114,115,116,117,118,119,120,121,122,
                                123,124,125,126,127,128,129,130,131,132,133,
                                134,135,136,137,138,139,140,141,142,143,144,
                                145,146,147,148,149,150,151,152,153,154,155,
                                156,157,158,159,160,161,162,163,164,165,166,
                                167,168,169,170,342,343,344,345,346,347,348,
                                349,350,351,352,353,354,355,356,357,358,359,
                                360,361,362,363,364,365,366,367,368,369,370,
                                371,372,373,374,375,376,377,378,379,380,381,
                                382,383,384,385,386,387,388,389,390,391,392,
                                393,394,395,396,397,398,399,400,401,402,403,
                                404,405,406,407,408,409,410,411,412,413,414,

Attaching the diags

[MB-8229] [Doc'd] Couchbase UI shows a node down when Rest API is used to rename a node in a multiple node cluster

Thu, 16 May 2013 15:28:58 -0500

When we reference a node from hostname to IP in a multiple node cluster, using REST API, then UI of another node shows that first node is down.

In Detail:

Steps followed:

1. Take a 2-node cluster referred with hostame (lets say node A and node B).
2. Rename node A from hostname to its corresponding IP, using the rest API - node/controller/rename.

Result: Node A gets renamed to its corresponding IP but the UI of other node(i.e. node B) shows that node A is down.

Attaching cbcollectinfo from both the nodes. Also uploading the screen-shot of UI. Please see.

[MB-8228] healthchecker launches exceptions when reading data

Tue, 14 May 2013 21:18:57 -0500

root@cola-s10302:/opt/couchbase/bin# ./cbhealthchecker -c localhost:8091 -u Administrator -p password
ERROR: unable to bucket-ddocs; please check your username (-u) and password (-p); (404) Object Not Found Traceback (most recent call last):
  File "/opt/couchbase/lib/python/collector.py", line 169, in number_bucketddocs
    docs = buckets.Buckets().runCmd('bucket-ddocs', server, port, user, password, opts_tmp)
  File "/opt/couchbase/lib/python/timeout.py", line 67, in wrapper
    return function(*args, **kargs)
  File "/opt/couchbase/lib/python/buckets.py", line 162, in runCmd
    self.user, self.password, opts)
  File "/opt/couchbase/lib/python/restclient.py", line 162, in restCmd
    return self.handleResponse(method, response, opts)
  File "/opt/couchbase/lib/python/restclient.py", line 70, in handleResponse
    output_json = json.loads(response.read())
  File "/opt/couchbase/lib/python/simplejson/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/opt/couchbase/lib/python/simplejson/decoder.py", line 335, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/couchbase/lib/python/simplejson/decoder.py", line 353, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

[MB-8227] Broken link in remote cluster section

Wed, 8 May 2013 20:41:58 -0500

Found this in XDCR testing, if you click the IP (e.g., 172.23.105.58) on remote cluster section (see screenshot attached), the link does not contain the port thus users are unable to go to the remote cluster UI via the link

The incorrect link is
http://172.23.105.58/index.html#sec=analytics&statsHostname=172.23.105.58

The correct should be something like
http://172.23.105.58:8091/index.html#sec=analytics&statsHostname=172.23.105.58

Ketaki, if you think it is a bug, please reassign to the UI folks. Thanks.

[MB-8226] Implement ns_server side memcached API for new get_meta_batch and update_meta_batch

Thu, 16 May 2013 16:15:27 -0500

[MB-8225] OSX installer has changes in master branch that are not in 2.0.2 branch

Mon, 13 May 2013 11:14:34 -0500

There are changes in couchdbx-app project that are in the master branch which are not in the 2.0.2 branch. Some of those changes mention "2.0.2" in their title....

commit 590f96de205abc336160367926e77fc68c613c42
Author: Phil Labee <philip@couchbase.com>
Date: Tue Apr 23 09:50:14 2013 -0700

    ~~MB-7973~~: cbanalyze-core is in bin, not bin/tools

    Change-Id: I3bab5a45ee48b2beb288450d0a0571837090604e
    Reviewed-on: http://review.couchbase.org/25857
    Reviewed-by: Philip Labee <philip@couchbase.com>
    Tested-by: Philip Labee <philip@couchbase.com>

commit 543ae0bdff82c3ff6ce250656178a0a03053c97e
Merge: cfb7228 cf709ac
Author: Phil Labee <philip@couchbase.com>
Date: Tue Apr 23 09:42:20 2013 -0700

    <- 2.0.2: ~~MB-8129~~: cbworkloadgen is in bin, not bin/tools

    Change-Id: I4770d2c8422a3d6534907f39ee92784ee87e40bc

[MB-8224] Fix race condition in vkey stats scheduling/checking to see if we have a bgfetch to do

Mon, 13 May 2013 23:01:59 -0500

Fix race condition in vkey stats scheduling/checking to see if we have a bgfetch to do

[MB-8222] single node goes offline during a node add, subsequent node add works okay

Tue, 14 May 2013 23:33:57 -0500

During some recent testing with 2.0.1, when building the cluster, the single-node cluster went offline during a node-add. By offline, I mean the list of server nodes showed the only node in the cluster as "down" but the web console was working just fine.

There was also this message in the logs:
Node nonode@nohost saw that node 'ns_1@127.0.0.1' went down.

Which is kind of surprising.

The node in question is centosa, and it was centosb which was being added. These are in the attached logs. A subsequent attempt to add node centosb was just fine. Later, the host cdh3 was added. I've just included that host for completeness.

[MB-8221] Updates to Upgrade Process for 2.0.2

Thu, 16 May 2013 17:21:57 -0500

Review session on 2.0.2 upgrade process: see what overhaul required to existing chapter.

[MB-8220] cbrecovery should have a detail final report

Thu, 16 May 2013 13:22:33 -0500

Cbrecovery should produce a final report with the following metrics:

- num of items been recovered.
- num of vbuckets been recovered.
- num of bytes been recovered.
- (maybe) runtime.
- (maybe) num of items left not been recovered? [not implemented]

[MB-8219] Find ways of keeping logs more relevant, perhaps compression?

Wed, 8 May 2013 14:56:26 -0500

One of the challenges we run into often in support is that when a customer has a problem, but the time we actually get logs from the systems, some or all of them have already rolled over and so the problem or context is lost.

This is especially true for stats information which can be incredibly useful, but typically only lasts a few days within the log files.

On easy proposal to help this would be to compress the log files as they are rolled over which would allow us to store much more history within the gigabyte of disk space that we enforce for ourselves.

[MB-8218] windows buildbot failed to build windows version 2.0.2-789

Mon, 13 May 2013 22:57:44 -0500

Buildbot failed to build windows binany

http://builds.hq.northscale.net:8010/builders/cs-win2008-x64-20-builder-202/builds/98

[MB-8216] Per-character lookup in documents section of UI causes performance and user experience issues

Wed, 8 May 2013 11:59:41 -0500

With enough documents in a particular bucket (40k at least at the customer), the search for a given document in the "documents" screen causes some web browsers to hang at the press of a single key.

Would it be possible to stop it from trying to lookup the documents on each key stroke?

Or is this just a problem with a particular browser (see attachment)

[MB-8215] [windows] firewalled node is seen as healthy

Fri, 17 May 2013 14:35:52 -0500

.29 node was firewalled
Domain Profile Settings:
----------------------------------------------------------------------
State ON
Firewall Policy BlockInbound,AllowOutbound
LocalFirewallRules N/A (GPO-store only)
LocalConSecRules N/A (GPO-store only)
InboundUserNotification Disable
RemoteManagement Disable
UnicastResponseToMulticast Enable

Logging:
LogAllowedConnections Disable
LogDroppedConnections Disable
FileName %systemroot%\system32\LogFiles\Firewall\pfirewall.log
MaxFileSize 4096

Private Profile Settings:
----------------------------------------------------------------------
State ON
Firewall Policy BlockInbound,AllowOutbound
LocalFirewallRules N/A (GPO-store only)
LocalConSecRules N/A (GPO-store only)
InboundUserNotification Disable
RemoteManagement Disable
UnicastResponseToMulticast Enable

Logging:
LogAllowedConnections Disable
LogDroppedConnections Disable
FileName %systemroot%\system32\LogFiles\Firewall\pfirewall.log
MaxFileSize 4096

Public Profile Settings:
----------------------------------------------------------------------
State ON
Firewall Policy BlockInbound,AllowOutbound
LocalFirewallRules N/A (GPO-store only)
LocalConSecRules N/A (GPO-store only)
InboundUserNotification Disable
RemoteManagement Disable
UnicastResponseToMulticast Enable

Logging:
LogAllowedConnections Disable
LogDroppedConnections Disable
FileName %systemroot%\system32\LogFiles\Firewall\pfirewall.log
MaxFileSize 4096

Ok.

[MB-8214] bgfetcher performance regression

Sun, 19 May 2013 00:10:40 -0500

* bgfetcher condition variable for woken task is not getting correctly (hasWokenTask).
* also need to immediately move any woken task to readyQueue instead of re-pushing to futureQueue

[MB-8213] Implement batch getMeta and setMeta/delMeta operations in ep_engine

Thu, 16 May 2013 16:19:04 -0500

In 2.1, XDCR will be able to replicate documents to remote memcached directly.

In order to allow batch replication, we need at least two new memcached operations, get_meta_batch, and update_meta_batch.

1. get_meta_batch: by this new command ep_engine should receive a list of keys and return a list of metadata for these keys.

2. update_meta_batch: this new command will take a list of (ops, key, metadata) as input and execute them in the same order as the keys ordered in the list. The ops can be either "deleteWithMeta" or "setWithMeta".

For these two new operations, we need to design new protocol (request and response) for ep_engine.

On the ns_server side, we also need to create memcached API for these two new operations. This will be tracked separately.

The first step is to determine the protocol so xdcr and ep_engine team can work in parallel.

[MB-8211] making lots of streaming connections in general, or lots of streaming connections in a short period of time causes unexpected resource usage and/or crashing owing to OOM

Wed, 8 May 2013 12:51:10 -0500

Going into version 2.0, we'd integrated changes that make handling of many streaming connections much less expensive in terms of memory, but there is some evidence that creating a lot of them at the same time or creating a large number of streaming connections could cause instability and crashing in ns_server.

Note that in the case of PHP deployments, we've found the number of worker processes to be 256, 512, or 5000 processes in some deployments. That is per server.

Then, deployments can be 50 or more systems.

Thus, we could have 12,000+ or 25,000+ or 250,000 processes which may all try to connect to a cluster.

This issue is to track possible solutions to this kind of thing occurring.

Possible solutions could be to drop connections or to only allow a number of connections tested to work in our minimum system configuration.

Note that at the client library side we have a method of sharing the configuration between multiple processes. However, it is not currently a complete solution and may never be because:
- It is not on by default as it requires the configuration of a file path.
- In some cases, during a rebalance, we can still have a rush of processes trying to get a configuration.
- We do not currently have a good solution for updating the configuration quickly if it's a memcached type bucket, as there is no NOT_MY_VBUCKET response.

Even if we can solve all of those issues at the client library side, a bug or an very large number of client systems can still cause problems, so we will need ns_server to be reliable even in the face of lots of requests.

[MB-8210] Update the incorrect information on the Automated Index Updates

Tue, 7 May 2013 18:20:49 -0500

We need to fix this page http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-views-operation-autoupdate.html ?

Relevant information about how it actually works.

"Every updateInterval milliseconds it checks if index file is more than updateMinChanges behind .couch files (which is itself behind in-memory source of truth, potentially for tens of seconds). And true, it triggers view update."

[MB-8209] cbrecovery will never run for vbucket 0

Thu, 9 May 2013 20:42:47 -0500

In cbrecovery:

134 if vbucket:
135 params = urllib.urlencode({"vbucket": vbucket})

Thus when vbucket 0 is missing, it will never run through the code, giving us this error:

root@ubu-0107:~# /opt/couchbase/bin/cbrecovery http://Administrator:password@10.1.3.72:8091 http://Administrator:password@10.3.3.141:8091 -b default -B default
2013-05-07 14:06:11,430: MainThread Missing vbuckets to be recovered:[{"node": "ns_1@10.3.4.25", "vbuckets": [0]}]
[ ] 0.1% (100/100000 msgs)
bucket: default, msgs transferred...
: total | last | per sec
batch : 4 | 4 | 27.2
byte : 25600 | 25600 | 174003.9
msg : 100 | 100 | 679.7
transfer data only. bucket design docs will be skipped.
done
2013-05-07 14:06:11,678: mt Error:recovery_uuid query parameter is missing

The change would be:

134 if vbucket is not None:
135 params = urllib.urlencode({"vbucket": vbucket})

[MB-8208] Replace XDCR intro with XDCR whitepaper intro

Fri, 10 May 2013 18:18:19 -0500

Replace XDCR intro with XDCR whitepaper intro

[MB-8207] moxi does not allow a noop before authentication on binary protocol

Wed, 8 May 2013 15:16:05 -0500

We had added a feature to spymemcached based on a Couchbase user's request to detect hung processes. This tries to complete a noop before doing auth.

It's fine against memcached/ep-engine in all cases, and it appears to be fine for ascii (where there is no authentication and it fails back to the version command), but moxi does not seem to allow auth after the noop. This may be because it's expecting the first command to wire it up to a downstream for the "gateway" moxi?

We're going to search for a workaround, but I wanted to make sure this issue was known.

See also:
https://code.google.com/p/spymemcached/issues/detail?id=272&thanks=272&ts=1364702110
and
https://github.com/mumoshu/play2-memcached/issues/17

[MB-8206] memcached crashing/restarting with Assertion `metadata.size == 16' failed.

Wed, 15 May 2013 00:33:52 -0500

Encountered this issue during an xdcr test on 2.0.2 build 780.

Created 3 buckets on local/remote site.
Started unidirection replication from default bucket on local site to remote.
I waited about 5 minutes after xdcr pairing to start loading data. Then I checked on the test about 30 minutes into the access phase I notices nodes were constantly restarting with these messages in UI logs:

Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 134. Restarting.

and from babysitter.1:

[ns_server:info,2013-05-06T16:36:18.200,babysitter_of_ns_1@127.0.0.1:<0.134.0>:ns_port_server:log:168]memcached<0.134.0>: memcached: src/c
ouch-kvstore/couch-kvstore.cc:1331: static int CouchKVStore::recordDbDump(Db*, DocInfo*, void*): Assertion `metadata.size == 16' failed.

[ns_server:info,2013-05-06T16:36:18.307,babysitter_of_ns_1@127.0.0.1:<0.133.0>:supervisor_cushion:handle_info:58]Cushion managed superviso
r for memcached failed: {abnormal,134}
[ns_server:debug,2013-05-06T16:36:18.308,babysitter_of_ns_1@127.0.0.1:<0.135.0>:supervisor_cushion:init:39]starting ns_port_server with de
lay of 5000
[error_logger:error,2013-05-06T16:36:18.307,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generi
c server <0.134.0> terminating
** Last message in was {#Port<0.3039>,{exit_status,134}}
** When Server state == {state,#Port<0.3039>,memcached,
                               {["memcached: src/couch-kvstore/couch-kvstore.cc:1331: static int CouchKVStore::recordDbDump(Db*, DocInfo*,
void*): Assertion `metadata.size == 16' failed.",
                                 "Mon May 6 16:36:15.660505 PDT 3: (saslbucket) metadata loaded in 2819 ms",
                                 "Mon May 6 16:36:13.707220 PDT 3: (default) warmup completed in 804 ms",
                                 "Mon May 6 16:36:13.694935 PDT 3: (default) metadata loaded in 792 ms",
                                 "Mon May 6 16:36:13.018266 PDT 3: (default) Failed to load mutation log, falling back to key dump",
                                 "Mon May 6 16:36:12.942281 PDT 3: Extension support isn't implemented in this version of bucket_engine",
                                 "Mon May 6 16:36:12.936368 PDT 3: (saslbucket) Failed to load mutation log, falling back to key dump",
                                 "Mon May 6 16:36:12.886616 PDT 3: (default) Connected to mccouch: \"127.0.0.1:11213\"",
                                 "Mon May 6 16:36:12.885946 PDT 3: (default) Trying to connect to mccouch: \"127.0.0.1:11213\"",
                                 "Mon May 6 16:36:12.879580 PDT 3: Extension support isn't implemented in this version of bucket_engine",
                                 "Mon May 6 16:36:12.844079 PDT 3: (saslbucket1) Failed to load mutation log, falling back to key dump",
                                 "Mon May 6 16:36:12.834687 PDT 3: (saslbucket) Connected to mccouch: \"127.0.0.1:11213\"",
                                 "Mon May 6 16:36:12.834366 PDT 3: (saslbucket) Trying to connect to mccouch: \"127.0.0.1:11213\"",
                                 "Mon May 6 16:36:12.810127 PDT 3: Extension support isn't implemented in this version of bucket_engine",
                                 "Mon May 6 16:36:12.767939 PDT 3: (saslbucket1) Connected to mccouch: \"127.0.0.1:11213\"",
                                 "Mon May 6 16:36:12.767549 PDT 3: (saslbucket1) Trying to connect to mccouch: \"127.0.0.1:11213\"",
                                 empty],

Seems the cluster never stabilizes but memcached keeps restarting.
Logs from suspected host attached(172.23.105.45 ), remaining logs pending.

[MB-8205] couchbase-cli broken in build 786, getting: ImportError: No module named json

Fri, 10 May 2013 18:52:55 -0500

python 2.4
CentOS release 5.8 (Final)
2.0.2 Build 786

0). stopped cb service: /etc/init.d/couchbase-server stop
1). removed previous installation of couchbase: rpm -e couchbase-server-2.0.2-779
2). installed new rpm: rpm -i couchbase-server-enterprise_x86_64_2.0.2-786-rel.rpm
3). invoked: couchbase-cli

[root@cen-1910 bin]# pwd
/opt/couchbase/bin

[root@cen-1910 bin]# ./couchbase-cli
Traceback (most recent call last):
  File "/opt/couchbase/lib/python/couchbase-cli", line 13, in ?
    import node
  File "/opt/couchbase/lib/python/node.py", line 8, in ?
    import util_cli as util
  File "/opt/couchbase/lib/python/util_cli.py", line 4, in ?
    import json
ImportError: No module named json
[root@cen-1910 bin]# rpm -qa|grep couchbase
couchbase-server-2.0.2-786
[root@cen-1910 bin]# uname -a
Linux cen-1910 2.6.18-308.el5 #1 SMP Tue Feb 21 20:06:06 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@cen-1910 bin]# pwd
/opt/couchbase/bin
[root@cen-1910 bin]# ./couchbase-cli server-list -c 10.1.3.106:8091 -u Administrator -p couchbase
Traceback (most recent call last):
  File "/opt/couchbase/lib/python/couchbase-cli", line 13, in ?
    import node
  File "/opt/couchbase/lib/python/node.py", line 8, in ?
    import util_cli as util
  File "/opt/couchbase/lib/python/util_cli.py", line 4, in ?
    import json
ImportError: No module named json
[root@cen-1910 bin]#

[MB-8204] Cbrecovery's unnecessary options, some that cause imminent data loss

Thu, 16 May 2013 13:15:41 -0500

The following options need to be removed for cbrecovery:

-k KEY, --key=KEY allow only items with keys that match a regexp
--vbucket-list=VBUCKET_LIST
transfer items from specified vbuckets only

Desc:
- vbucket-list is being calculated internally anyway.
- key could cause data loss anyway, consider scenario:

e.g.

Vbucket 1020 is missing, supposed to contain key-1, and key-2
Say we run cbrecovery with key option on just key-2, so what cbrecovery would do is just load key-2, and mark status of vbucket 1020 as done. So even in the subsequent runs, cbrecovery would fail to detect data loss on vbucket 1020. -- We wouldn't want this.

[MB-8203] [DOC] Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 139. Restarting. Messages: Invalid configuration: Shards must bea factor of max vbuckets

Mon, 6 May 2013 20:25:06 -0500

when I played with cocuhbase-cli( bucket createion, node/ckuster init/edit)

I was constantly getting errors on the server:

Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 139. Restarting. Messages: Mon May 6 09:32:10.957626 PDT 3: (AAA) Invalid configuration: Shards must bea factor of max vbuckets ns_log000 ns_1@127.0.0.1 19:32:11 - Mon May 6, 2013

so. I can't create buckets on node

I can't exactly provide command line/steps that led to this server state

[MB-8202] cluster-init/edit doesn't provide status(success or not)

Fri, 10 May 2013 14:02:48 -0500

/opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost -u Administrator -p password --cluster-init-username=Administrator --cluster-init-password=password --cluster-init-port=8091 --cluster-init-ramsize=300
SUCCESS: init localhost

root@localhost ~]# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost -u Administrator -p password --cluster-init-username=Administrator --cluster-init-username=Administrator1 --cluster-init-password=password1 --cluster-init-port=8099
[root@localhost ~]# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-init-username=Administrator --cluster-init-password=password --cluster-init-port=8091
[root@localhost ~]# /opt/couchbase/bin/couchbase-cli cluster-edit --cluster=localhost -u Administrator -p password --cluster-init-username=Administrator --cluster-init-username=Administrator1 --cluster-init-password=password1 --cluster-init-port=8099
[root@localhost ~]#

desirable to obtain the status that the settings have been changed

apparently it's regression because before we got 'SUCCESS: init localhost'

[MB-8201] docsTotal changed during rebalance

Mon, 13 May 2013 19:33:18 -0500

docsTotal decreased during rebalance. It was 1512, and became 1509:

2013-05-06 16:40:03 | INFO | MainProcess | MainThread | [rebalance_progress._check_stats] Checked stat: {u'172.27.33.10': {u'activeVBucketsLeft': 0, u'docsTotal': 1512, u'docsTransferred': 1438, u'replicaVBucketsLeft': 0}, u'172.27.33.11': {u'activeVBucketsLeft': 176, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 24}, u'172.27.33.12': {u'activeVBucketsLeft': 176, u'docsTotal': 2524, u'docsTransferred': 1647, u'replicaVBucketsLeft': 24}, 'bucket': u'default', 'buckets_count': 1}
2013-05-06 16:40:03 | INFO | MainProcess | MainThread | [rebalance_progress._check_stats] Checked stat: {u'172.27.33.10': {u'activeVBucketsLeft': 0, u'docsTotal': 1512, u'docsTransferred': 1438, u'replicaVBucketsLeft': 0}, u'172.27.33.11': {u'activeVBucketsLeft': 176, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 24}, u'172.27.33.12': {u'activeVBucketsLeft': 176, u'docsTotal': 2524, u'docsTransferred': 1647, u'replicaVBucketsLeft': 24}, 'bucket': u'default', 'buckets_count': 1}
2013-05-06 16:40:04 | INFO | MainProcess | MainThread | [rebalance_progress._check_stats] Checked stat: {u'172.27.33.10': {u'activeVBucketsLeft': 0, u'docsTotal': 1509, u'docsTransferred': 1447, u'replicaVBucketsLeft': 0}, u'172.27.33.11': {u'activeVBucketsLeft': 173, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 21}, u'172.27.33.12': {u'activeVBucketsLeft': 173, u'docsTotal': 2524, u'docsTransferred': 1668, u'replicaVBucketsLeft': 21}, 'bucket': u'default', 'buckets_count': 1}

[MB-8200] Rebalance exited with reason stop

Mon, 6 May 2013 18:04:57 -0500

3 buckets:
1 default bucket, 2 buckets with dedicated ports

rebalance failed:
Rebalance exited with reason stop

attaching logs

[MB-8199] [2.0.2 - RN + Docs] many requests to views causes resource leak, crash

Fri, 17 May 2013 14:28:51 -0500

In response to many view requests against the scatter/gather view merger, a node can allocate so many resources that it will fail to recover.

In one case, this did cause many timeouts in the log leading to max_restart_intensity:
[error_logger:error,2013-04-25T15:23:26.047,ns_1@10.128.16.171:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
     Supervisor: {local,ns_node_disco_sup}
     Context: shutdown
     Reason: reached_max_restart_intensity
     Offender: [{pid,<0.17237.774>},
                  {name,ns_config_rep},
                  {mfargs,{ns_config_rep,start_link,[]}},
                  {restart_type,permanent},
                  {shutdown,1000},
                  {child_type,worker}]

[MB-8198] window installer - show wrong result of system RAM configured

Tue, 7 May 2013 02:39:00 -0500

window installer - show wrong result of system RAM configured

[MB-8197] The REST API /settings/viewUpdateDaemon does not raise any exception with the user enter invalid login

Fri, 3 May 2013 04:58:59 -0500

The command documented here:
http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-views-operation-autoupdate.html

are correct but the following command does not return anything (error message, 401, ...) when a user calls the following command with an invalid username password:

curl http://Administrator:passwords@192.168.0.34:8091/settings/viewUpdateDaemon

We should return a 401 error

[MB-8196] [windows] Rebalance exited with reason {badarg,\n [{erlang,'++',\n [{'EXIT',\n {{badmatch,<17634.10455.4>},\n [{janitor_agent,handle_call_via_se

Tue, 7 May 2013 00:54:31 -0500

-t newmemcapable.GetrTests.getr_test,rebalance=2,GROUP=P0,descr=#simple getr replica_count-1 expiration and flags - 0 docs_ops-create cluster ops
rebalancing in 1->6 nodes

[2013-05-03 01:14:38,800] - [rest_client:991] ERROR - {u'node': u'ns_1@10.1.3.82', u'code': 2, u'text': u"Rebalance exited with reason {badarg,\n [{erlang,'++',\n [{'EXIT',\n {{{badmatch,<17634.10455.4>},\n [{janitor_agent,handle_call_via_servant,4},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n [{'janitor_agent-default',\n 'ns_1@10.1.3.86'},\n {get_mass_tap_docs_estimate,\n [103,104,105,106,107,108,109,110,111,\n 112,113,114,115,116,117,118,119,120,\n 121,122,123,124,125,126,127,128,129,\n 130,131,132,133,134,135,136,274,275,\n 276,277,278,279,280,281,282,283,284,\n 285,286,287,288,289,290,291,292,293,\n 294,295,296,297,298,299,300,301,302,\n 303,304,305,306,307,445,446,447,448,\n 449,450,451,452,453,454,455,456,457,\n 458,459,460,461,462,463,464,465,466,\n 467,468,469,470,471,472,473,474,475,\n 476,477,478,616,617,618,619,620,621,\n 622,623,624,625,626,627,628,629,630,\n 631,632,633,634,635,636,637,638,639,\n 640,641,642,643,644,645,646,647,648,\n 649,684,685,686,687,688,689,690,691,\n 692,693,694,695,696,697,698,699,700,\n 701,702,703,704,705,706,707,708,709,\n 710,711,712,713,714,715,716,717,718,\n 719,720,721,722,723,724,725,726,727,\n 728,729,730,731,732,733,734,735,736,\n 737,738,739,740,741,742,743,744,745,\n 746,747,748,749,750,751,752,753,754,\n 755,756,757,758,759,760,761,762,763,\n 764,765,766,767,768,769,770,771,772,\n 773,774,775,776,777,778,779,780,781,\n 782,783,784,785,786,787,788,789,790,\n 791,792,793,794,795,796,797,798,799,\n 800,801,802,803,804,805,806,807,808,\n 809,810,811,812,813,814,815,816,817,\n 818,819,820,821,822,823,824,825,826,\n 827,828,829,830,831,832,833,834,835,\n 836,837,838,839,840,841,842,843,844,\n 845,846,847,848,849,850,851,852,853,\n 990,991,992,993,994,995,996,997,998,\n 999,1000,1001,1002,1003,1004,1005,1006,\n 1007,1008,1009,1010,1011,1012,1013,\n 1014,1015,1016,1017,1018,1019,1020,\n 1021,1022,1023]},\n infinity]}}},\n {'EXIT',\n {{{badmatch,<17636.899.0>},\n [{janitor_agent,handle_call_via_servant,4},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n [{'janitor_agent-default',\n 'ns_1@10.1.3.87'},\n

attaching logs

[MB-8195] Current visual design is inconsistent and unpretty. Need new global visual (re)design.

Thu, 2 May 2013 17:12:31 -0500

SUBJ.

Because our UI has grown for multiple releases there are tons of inconsistencies and just plain bugs. We tried twice to fix CSS without touching look but failed. We should do both CSS/HTML cleanup and visual cleanup as one process. Starting with some consistent and clean visual design.

Discussed this with Dipti some time ago and in general AFAIR she was ok. But initially assigned to her just in case.

Also intention of this work is to have some UI guidelines and some defined common style. E.g. how buttons should look (disabled, normal, ...).

[MB-8194] we should put bucket uuid into couchApiBase urls we advertise so that view requests cannot race with nodes leaving cluster

Thu, 2 May 2013 20:29:35 -0500

we should put bucket uuid into couchApiBase urls we advertise so that view requests cannot race with nodes leaving cluster

[MB-8193] [system test] Memcached segfault during rebalance with workload running causes rebalance failure

Tue, 14 May 2013 15:24:41 -0500

15G RAM 500G regular disc machine
cluster_ip: 172.23.105.23

Create a 8 node cluster, create default bucket with 6G RAM, 1 replica, sasl bucket with 5G RAM, 1 replica

Load items to both buckets into dgm state, around 70%.

Access the data for 2 hours, with workload:
default bucket: create:5,update:5,get:80,delete:5,expire:5,cache_miss:5, 10K ops/sec
saslbucket: create:5,update:5,get:20,delete:10,expire:60,cache_miss:5., 10K ops/sec

Then rebalance in one node with workload continue
Find core.memcached under /data path.
Saw a lot of errors like this:

Control connection to memcached on 'ns_1@172.23.105.23' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
cmd_binary_vocal_recv,
5},
{mc_client_binary,
select_bucket,
2},
{ns_memcached,
ensure_bucket,
2},
{ns_memcached,
handle_info,
2},
{gen_server,
handle_msg,
5},
{ns_memcached,
init,
1},
{gen_server,
init_it,
6},
{proc_lib,
init_p_do_apply,
3}]} (repeated 1 times)

Rebalance failed in a short time

This also happened in Windows the bug number is ~~MB-8192~~
The stack trace of the core.memcached.xxxx on some node will be added later