[MB-11954] wizard default bucket creation step fails silently if default bucket already exists Created: 13/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Aleksey Kondratenko Assignee: Aleksey Kondratenko
Resolution: Fixed Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Yes

 Description   
SUBJ.

Steps:

1) on blank and uninitialized cluster create bucket default

2) try to set up cluster via wizard

3) observe that it fails on "CREATE DEFAULT BUCKET" step

Extracting this out of MB-9750 that conflated this issue with something else that was originally reported for osex

 Comments   
Comment by Aleksey Kondratenko [ 14/Aug/14 ]
http://review.couchbase.org/40598
Comment by Wayne Siu [ 29/Aug/14 ]
Reopening to add a label to the ticket.




[MB-12099] Erlang crashes if xdcr_trace attempts to log invalid utf8 Created: 29/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Aleksey Kondratenko Assignee: Aliaksey Artamonau
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to MB-12097 [Utf-16 XDCR] Data loss - No items re... Resolved
Triage: Untriaged
Is this a Regression?: Yes

 Description   
See MB-12097 where attempt to capi-replicate invalid utf8 key causes find_missing to crash which causes some crash within ale.


 Comments   
Comment by Aleksey Kondratenko [ 29/Aug/14 ]
It looks like change of unicode:characters_to_binary to simple iolist_to_binary helps.
Comment by Aleksey Kondratenko [ 29/Aug/14 ]
http://review.couchbase.org/41096




[MB-12005] vbucket-seqno stats getting timed out during Views DGM test Created: 19/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket, view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Meenakshi Goel Assignee: Meenakshi Goel
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1166-rel

Triage: Triaged
Operating System: Ubuntu 64-bit
Is this a Regression?: Yes

 Description   
Test to Reproduce:
./testrunner -i yourfile.ini -t view.createdeleteview.CreateDeleteViewTests.pending_removal_with_ddoc_ops,ddoc_ops=update,test_with_view=True,num_ddocs=3,num_views_per_ddoc=3,items=200000,nodes_out=1,active
_resident_threshold=10,dgm_run=True,eviction_policy=fullEviction,skip_cleanup=true

Steps to Reproduce:
1. Setup a 5-node cluster
2. Rebalance in all nodes
3. Load bucket to achieve dgm 10%
4. Failover 1 node
5. Create Views and perform ddoc update operations
6. Test exits with error during ddoc validation

014-08-18 04:00:37 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://10.3.5.90:8092/default/_design/ddoc_test1/_view/views0?stale=false&connection_timeout=60000&full_set=true
2014-08-18 04:15:37 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.3.5.90:8092/default/_design/ddoc_test1/_view/views0?stale=false&connection_timeout=60000&full_set=true error timed out
2014-08-18 04:15:37 | ERROR | MainProcess | Cluster_Thread | [task.execute] Unexpected Exception Caught
ERROR
[('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 56, 'step', 'self.execute(task_manager)'), ('lib/tasks/task.py', 1525, 'execute', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
Mon Aug 18 04:15:37 2014
[('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/case.py', 391, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/case.py', 327, 'run', 'testMethod()'), ('pytests/view/createdeleteview.py', 626, 'pending_removal_with_ddoc_ops', 'self._verify_ddoc_data_all_buckets()'), ('pytests/view/createdeleteview.py', 274, '_verify_ddoc_data_all_buckets', 'result = self.cluster.query_view(self.master, ddoc_name, view.name, query, num_items, bucket)'), ('lib/couchbase/cluster.py', 464, 'query_view', 'return _task.result(timeout)'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')]
2014-08-18 04:15:37 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED

======================================================================
ERROR: pending_removal_with_ddoc_ops (view.createdeleteview.CreateDeleteViewTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/view/createdeleteview.py", line 626, in pending_removal_with_ddoc_ops
    self._verify_ddoc_data_all_buckets()
  File "pytests/view/createdeleteview.py", line 274, in _verify_ddoc_data_all_buckets
    result = self.cluster.query_view(self.master, ddoc_name, view.name, query, num_items, bucket)
  File "lib/couchbase/cluster.py", line 464, in query_view
    return _task.result(timeout)
  File "lib/tasks/future.py", line 160, in result
    return self.__get_result()
  File "lib/tasks/future.py", line 112, in __get_result
    raise self._exception
ServerUnavailableException: unable to reach the host @ 10.3.5.90

Logs:
[couchdb:error,2014-08-18T4:39:41.963,ns_1@10.3.5.90:<0.12717.6>:couch_log:error:44]Set view `default`, replica group `_design/ddoc_test0`, doc loader error
error: {timeout,{gen_server,call,
                                 [<0.12658.6>,
                                  {add_stream,706,0,0,5637,6},
                                  60000]}}
stacktrace: [{gen_server,call,3,[{file,"gen_server.erl"},{line,188}]},
             {couch_dcp_client,enum_docs_since,8,
                 [{file,
                      "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_dcp/src/couch_dcp_client.erl"},
                  {line,246}]},
             {couch_set_view_updater,'-load_changes/8-fun-2-',12,
                 [{file,
                      "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"},
                  {line,516}]},
             {lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
             {couch_set_view_updater,load_changes,8,
                 [{file,
                      "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"},
                  {line,589}]},
             {couch_set_view_updater,'-update/8-fun-3-',14,
                 [{file,
                      "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"},
                  {line,281}]}]

[couchdb:error,2014-08-18T6:46:34.997,ns_1@10.3.5.90:<0.12648.6>:couch_log:error:44]dcp client (<0.12660.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...
[couchdb:error,2014-08-18T6:46:39.608,ns_1@10.3.5.90:<0.21856.6>:couch_log:error:44]dcp client (<0.21861.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...
[couchdb:error,2014-08-18T6:46:42.611,ns_1@10.3.5.90:<0.21856.6>:couch_log:error:44]dcp client (<0.21861.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...

*Observed stacktraces and crashes in logs. Uploading logs.

Live Cluster:
1:10.3.5.90
2:10.3.5.91
3:10.3.5.92
4:10.3.5.93
5:10.3.4.75

 Comments   
Comment by Meenakshi Goel [ 19/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12005/f806d72b/10.3.5.90-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12005/8f3bae64/10.3.5.91-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12005/4fc16d8a/10.3.5.92-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12005/16116a7e/10.3.4.75-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12005/a3e503f6/10.3.5.93-diag.zip
Comment by Sarath Lakshman [ 19/Aug/14 ]
I believe we tried a toy build with separate connection for stats before in MB-11706. So I think I don't have much information about this problem
Comment by Nimish Gupta [ 22/Aug/14 ]
I have a toy build with change for separate connection (http://latestbuilds.hq.couchbase.com/couchbase-server-community_ubunt12-3.0.0-toy-nimish-x86_64_3.0.0-704-toy.deb). Meenakshi, please run the test with this build.
Comment by Meenakshi Goel [ 22/Aug/14 ]
Started test with the toy build.
Comment by Nimish Gupta [ 25/Aug/14 ]
I don't see any crash in couchdb after using the separate connection. The crashes are in ns_server logs, looks due to rebalance and failover which were in earlier logs also. The separate connection has reduced the number of stats timeout messages. Previously there were 714 stats timeout message which now has reduced to 113.
Comment by Sriram Melkote [ 25/Aug/14 ]
Wayne, please help us obtain a toy build with mctimings compiled.
Comment by Cihan Biyikoglu [ 26/Aug/14 ]
Hi team, could you explain why this is critical? trying to shut the gates for 3.0 and want to make sure we keep only things that have high impact on the release.
Comment by Ketaki Gangal [ 27/Aug/14 ]
This causes a number of view-test timeouts. While these can be worked around by adjusting test timeouts, imo it is not the right way for the code/query timings to work.
As mentioned earlier, there are a large number of stat calls being made here.

This should manifest in performance tests already, but with functional tests, there is a large increase in query-runtime due to the above.


Comment by Meenakshi Goel [ 27/Aug/14 ]
Started test with toy build http://qa.sc.couchbase.com/job/ubuntu_x64--65_02--view_query_extended-P1/171/console
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
not a blocker for RC2 - continue research and we can consider the fix for GA if it is low impact.
Comment by Meenakshi Goel [ 29/Aug/14 ]
I am unable to reproduce this issue with 3.0.0-1199-rel and with RC1(3.0.0-1174-rel) as test doesn't progresses after starting the View Query.
It has been observed that slave from which test is launched gets out of Memory as running python process consumes all the available memory due to which it exits when there is no memory left.
Test has been tried on multiple slaves and multiple clusters, Also tried with less number of items upto 1.5M but seeing same behaviour.

http://qa.sc.couchbase.com/job/ubuntu_x64--65_03--view_dgm_tests-P1/119/consoleFull
Comment by Wayne Siu [ 29/Aug/14 ]
please re-open if we can reproduce the issue.




[MB-12097] [Utf-16 XDCR] Data loss - No items replicated to destination Created: 29/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Sangharsh Agarwal Assignee: Aleksey Kondratenko
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: CentOs5 (64 bit)

Attachments: PNG File Screen Shot 2014-08-29 at 5.00.49 PM.png    
Issue Links:
Relates to
relates to MB-12099 Erlang crashes if xdcr_trace attempts... Resolved
Triage: Triaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: [Source]
10.3.2.109 : https://s3.amazonaws.com/bugdb/jira/MB-12097/2e776f21/10.3.2.109-8292014-433-couch.tar.gz
10.3.2.109 : https://s3.amazonaws.com/bugdb/jira/MB-12097/6eff2cf5/10.3.2.109-8292014-429-diag.zip
10.3.2.109 : https://s3.amazonaws.com/bugdb/jira/MB-12097/a4c3ccf0/10.3.2.109-diag.txt.gz

10.3.4.175 : https://s3.amazonaws.com/bugdb/jira/MB-12097/27df791c/10.3.4.175-diag.txt.gz
10.3.4.175 : https://s3.amazonaws.com/bugdb/jira/MB-12097/3db3fbfd/10.3.4.175-8292014-433-couch.tar.gz
10.3.4.175 : https://s3.amazonaws.com/bugdb/jira/MB-12097/70f9482c/10.3.4.175-8292014-428-diag.zip

[Erlang Crashed]
erlang : https://s3.amazonaws.com/bugdb/jira/MB-12097/1838e7ea/erlang-10.3.4.175-0.log
erlang : https://s3.amazonaws.com/bugdb/jira/MB-12097/f90a602e/erlang-10.3.2.109-0.log

[Destination]
10.3.4.176 : https://s3.amazonaws.com/bugdb/jira/MB-12097/2c2138ce/10.3.4.176-8292014-432-diag.zip
10.3.4.176 : https://s3.amazonaws.com/bugdb/jira/MB-12097/7a3deb96/10.3.4.176-diag.txt.gz
10.3.4.176 : https://s3.amazonaws.com/bugdb/jira/MB-12097/fd6123e8/10.3.4.176-8292014-433-couch.tar.gz
10.3.2.161 : https://s3.amazonaws.com/bugdb/jira/MB-12097/0468eed0/10.3.2.161-8292014-433-couch.tar.gz
10.3.2.161 : https://s3.amazonaws.com/bugdb/jira/MB-12097/158395ba/10.3.2.161-diag.txt.gz
10.3.2.161 : https://s3.amazonaws.com/bugdb/jira/MB-12097/f1c6caf0/10.3.2.161-8292014-431-diag.zip

Is this a Regression?: Yes

 Description   
[Test]
./testrunner -i INI/xdcr.4.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=True,get-coredumps=True -t cornercases.negativetests.NegativeTests2.test_utf_16_keys_with_xdcr,rdirection=unidirection,ctopology=chain,items=100

[Test Logs]
https://friendpaste.com/12jJEznsAjjuxZR6ZAF8Fn

[Live Cluster]


[Source]
10.3.2.109
10.3.4.175

[Destination]
10.3.2.161
10.3.4.176

1. Setup Uni-XDCR CAPI mode XDCR from Source -> Destination. Bucket: default
2. Load 100 Items encoded with UTF-16.
3. No items transferred to Destination.


test is passed in 2.5.1-1083


Keys Not Replicated: ��loadOne0, ��loadOne1, ��loadOne2, ..... ��loadOne99

Screen Shot attached for the documents on Source side. Cluster is Live for debugging.

 Comments   
Comment by Sangharsh Agarwal [ 29/Aug/14 ]
Erlang crash also happened on Source cluster, crash is also attached.
Comment by Aleksey Kondratenko [ 29/Aug/14 ]
There is _exact_ same issue on 2.5.1.

And all of that is specific to capi mode and only for non-optimistically replicated docs.

3.0.0 xdcr_trace code however additional issue with logging that causes troubles for entire erlang VM. _That_ is worth fixing.
Comment by Aleksey Kondratenko [ 29/Aug/14 ]
Also for future reference. specific thing that this test is doing is UTF16 BOM chars. Which are not valid utf8 and cause issues like this:

[error_logger:error,2014-08-29T12:23:30.039,n_0@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
  crasher:
    initial call: xdc_vbucket_rep:init/1
    pid: <0.2520.0>
    registered_name: []
    exception exit: {{nocatch,
                         {invalid_json,
                             {{error,
                                  {182,
                                   "lexical error: invalid bytes in UTF8 string.\n"}},
                              <<"{\"error\":\"bad_request\",\"reason\":\"invalid UTF-8 JSON: {{error,{3,\\n \\\"lexical error: invalid bytes in UTF8 string.\\\\n\\\"}},\\n <<\\\"{\\\\\\\"ÿþl\\\\\\\\u0000o\\\\\\\\u0000a\\\\\\\\u0000d\\\\\\\\u0000O\\\\\\\\u0000n\\\\\\\\u0000e\\\\\\\\u00001\\\\\\\\u00002\\\\\\\\u00003\\\\\\\\u00001\\\\\\\\u00002\\\\\\\\u00004\\\\\\\\u0000\\\\\\\":\\\\\\\"1-138efab654f3dba80000000000000000\\\\\\\"}\\\">>}\"}\n">>}}},
                     [{ejson,nif_decode,1,
                          [{file,
                               "/root/src/altoros/moxi/repo3/couchdb/src/ejson/ejson.erl"},
                           {line,50}]},
                      {couch_api_wrap_httpc,process_response,4,
                          [{file,"src/couch_api_wrap_httpc.erl"},{line,92}]},
                      {xdc_vbucket_rep_worker,find_missing_helper,3,
                          [{file,"src/xdc_vbucket_rep_worker.erl"},
                           {line,279}]},
                          [{file,"src/xdc_vbucket_rep_worker.erl"},
                           {line,229}]},
                      {xdc_vbucket_rep_worker,queue_fetch_loop,8,
                          [{file,"src/xdc_vbucket_rep_worker.erl"},
                           {line,57}]}]}
      in function gen_server:terminate/6 (gen_server.erl, line 744)
    ancestors: [<0.460.0>,<0.426.0>,xdc_replication_sup,xdcr_sup,
                  ns_server_sup,ns_server_cluster_sup,<0.58.0>]
    messages: []
    links: [<0.460.0>]
    dictionary: [{changes_reader,<0.2541.0>}]
    trap_exit: true
    status: running
    heap_size: 10958
    stack_size: 27
    reductions: 15281
  neighbours:
Comment by Aleksey Kondratenko [ 29/Aug/14 ]
Created MB-12099 for xdcr_trace bug.
Comment by Aruna Piravi [ 29/Aug/14 ]
ok, will look into the UTF-16 generation. The way we support UTF-16 is by converting it to UTF-8? Do we also support UTF-32?
Comment by Aleksey Kondratenko [ 29/Aug/14 ]
Aruna, I'm not exactly sure what happens but I was able to reproduce this locally and I think I understand what happens in this text.

But in any case please speak to me first before doing anything at all in this area. Because it's _very_ misunderstand-ful area.
Comment by Aleksey Kondratenko [ 29/Aug/14 ]
And for those who watch. No we don't support non-utf8 in keys. It's not exactly "no support at all" but it's more no than yes.




[MB-10718] DCP: Change Capture API and 3rd party consumable Created: 01/Apr/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: feature-backlog
Fix Version/s: feature-backlog
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Cihan Biyikoglu Assignee: Cihan Biyikoglu
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Chiyoung Seo [ 29/Aug/14 ]
This should be addressed as part of https://www.couchbase.com/issues/browse/MB-11627




[MB-12041] Disabling access.log on multiple buckets results in node failing to become available Created: 21/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Brent Woodruff Assignee: Abhinav Dangeti
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2014-08-26 at 8.53.15 AM.png     PNG File Screen Shot 2014-08-26 at 8.53.29 AM.png     PNG File Screen Shot 2014-08-26 at 8.58.37 AM.png     PNG File Screen Shot 2014-08-26 at 8.58.43 AM.png    
Issue Links:
Dependency
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
On review of a customer ticket today during support discussions, this particular issue was brought up. It is unclear from subsequent discussions in that ticket whether this issue was addressed and fixed.

Steps to reproduce:

* Initialize a Couchbase node with more than one bucket

* Disable the access.log on *both* buckets using the following command for each bucket:

wget -O- --user=Administrator --password=password --post-data='ns_bucket:update_bucket_props("bucket1", [{extra_config_string, "alog_path="}]).' http://localhost:8091/diag/eval

wget -O- --user=Administrator --password=password --post-data='ns_bucket:update_bucket_props("bucket2", [{extra_config_string, "alog_path="}]).' http://localhost:8091/diag/eval

where 'bucket1' and 'bucket2' are the bucket names.

* Restart the node and observe the following errors in the logs:

memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
memcached<0.89.0>: Unsupported key: <^A>

* Note that the node remains pending and never becomes available

 Comments   
Comment by Abhinav Dangeti [ 21/Aug/14 ]
I don't see the node failing to become available.

Started couchbase server with 2 buckets:
  1 Fri Aug 22 10:36:25.628702 PDT 3: (default) Trying to connect to mccouch: "127.0.0.1:13000"
  2 Fri Aug 22 10:36:25.628978 PDT 3: (default) Connected to mccouch: "127.0.0.1:13000"
  3 Fri Aug 22 10:36:25.644502 PDT 3: (No Engine) Bucket default registered with low priority
  4 Fri Aug 22 10:36:25.644528 PDT 3: (No Engine) Spawning 4 readers, 4 writers, 1 auxIO, 1 nonIO threads
  5 Fri Aug 22 10:36:25.646178 PDT 3: (default) metadata loaded in 982 usec
  6 Fri Aug 22 10:36:25.646205 PDT 3: (default) Enough number of items loaded to enable traffic
  7 Fri Aug 22 10:36:25.646559 PDT 3: (default) warmup completed in 1052 usec
  8 Fri Aug 22 10:36:33.495128 PDT 3: (default) Shutting down tap connections!
  9 Fri Aug 22 10:36:33.495174 PDT 3: (default) Shutting down dcp connections!
 10 Fri Aug 22 10:36:33.496244 PDT 3: (No Engine) Unregistering last bucket default
 11 Fri Aug 22 10:36:41.791797 PDT 3: (bucket1) Trying to connect to mccouch: "127.0.0.1:13000"
 12 Fri Aug 22 10:36:41.791932 PDT 3: (bucket1) Connected to mccouch: "127.0.0.1:13000"
 13 Fri Aug 22 10:36:41.800241 PDT 3: (No Engine) Bucket bucket1 registered with low priority
 14 Fri Aug 22 10:36:41.800273 PDT 3: (No Engine) Spawning 4 readers, 4 writers, 1 auxIO, 1 nonIO threads
 15 Fri Aug 22 10:36:41.801437 PDT 3: (bucket1) metadata loaded in 719 usec
 16 Fri Aug 22 10:36:41.801450 PDT 3: (bucket1) Enough number of items loaded to enable traffic
 17 Fri Aug 22 10:36:41.801593 PDT 3: (bucket1) warmup completed in 761 usec
 18 Fri Aug 22 10:36:46.922063 PDT 3: (bucket2) Trying to connect to mccouch: "127.0.0.1:13000"
 19 Fri Aug 22 10:36:46.922191 PDT 3: (bucket2) Connected to mccouch: "127.0.0.1:13000"
 20 Fri Aug 22 10:36:46.931024 PDT 3: (No Engine) Bucket bucket2 registered with low priority
 21 Fri Aug 22 10:36:46.932154 PDT 3: (bucket2) metadata loaded in 715 usec
 22 Fri Aug 22 10:36:46.932170 PDT 3: (bucket2) Enough number of items loaded to enable traffic
 23 Fri Aug 22 10:36:46.932314 PDT 3: (bucket2) warmup completed in 776 usec

Loaded 1000 items in each, and restarted node, after setting the alog_path to NULL, in the same way mentioned.
  1 Fri Aug 22 10:38:08.372050 PDT 3: (bucket2) Trying to connect to mccouch: "127.0.0.1:13000"
  2 Fri Aug 22 10:38:08.372307 PDT 3: (bucket2) Connected to mccouch: "127.0.0.1:13000"
  3 Fri Aug 22 10:38:08.382418 PDT 3: (No Engine) Bucket bucket2 registered with low priority
  4 Fri Aug 22 10:38:08.382445 PDT 3: (No Engine) Spawning 4 readers, 4 writers, 1 auxIO, 1 nonIO threads
  5 Fri Aug 22 10:38:08.434024 PDT 3: (bucket1) Trying to connect to mccouch: "127.0.0.1:13000"
  6 Fri Aug 22 10:38:08.434205 PDT 3: (bucket1) Connected to mccouch: "127.0.0.1:13000"
  7 Fri Aug 22 10:38:08.445064 PDT 3: (No Engine) Bucket bucket1 registered with low priority
  8 Fri Aug 22 10:38:08.481732 PDT 3: (bucket2) metadata loaded in 98 ms
  9 Fri Aug 22 10:38:08.507847 PDT 3: (bucket2) warmup completed in 124 ms
 10 Fri Aug 22 10:38:08.540342 PDT 3: (bucket1) metadata loaded in 92 ms
 11 Fri Aug 22 10:38:08.553951 PDT 3: (bucket1) warmup completed in 106 ms

[10:37:46] abhinav: ~/Documents/couchbase30/ep-engine $ ./management/cbstats localhost:12000 all -b bucket1 | grep alog
 ep_alog_block_size: 4096
 ep_alog_path:
 ep_alog_sleep_time: 1440
 ep_alog_task_time: 10
[10:38:50] abhinav: ~/Documents/couchbase30/ep-engine $ ./management/cbstats localhost:12000 all -b bucket2 | grep alog
 ep_alog_block_size: 4096
 ep_alog_path:
 ep_alog_sleep_time: 1440
 ep_alog_task_time: 10

I do see the duplicate entry warning, but that I'm guessing is because we set alog_path again after initializing it to the default value, in which case it would overwrite.
Comment by Abhinav Dangeti [ 22/Aug/14 ]
I tried your scenario with the latest 3.0 and then with 2.5.1, and noted similar behavior.
Can you point me to the build, with which you saw this issue, or perhaps the logs from where you hit the issue?
Comment by Abhinav Dangeti [ 25/Aug/14 ]
I also merged a change that will let the user enable/disable access log generation during run time:
http://review.couchbase.org/#/c/40884/
Comment by Brent Woodruff [ 26/Aug/14 ]
Hi Abhinav,

Apologies for the delay in getting back to you regarding a build that exhibits this issue. I believe it was the 2.5.1 release, since that was what the customer had just upgraded to in the originating issue. I was running that release on my Mac testing the commands provided by engineering to disable the access.log. I was able to reproduce this issue again today.

Attached screenshots of before and after these commands:

$./cbstats localhost:11210 config -b default |grep alog_path
 ep_alog_path: /Users/brent/Library/Application Support/Couchbase/var/lib/couchdb/default/access.log

$./cbstats localhost:11210 config -b beer-sample |grep alog_path
 ep_alog_path: /Users/brent/Library/Application Support/Couchbase/var/lib/couchdb/beer-sample/access.log

$ wget -O- --user=Administrator --password=couchbase --post-data='ns_bucket:update_bucket_props("default", [{extra_config_string, "alog_path="}]).' http://localhost:8091/diag/eval
# wget output removed

$ wget -O- --user=Administrator --password=couchbase --post-data='ns_bucket:update_bucket_props("beer-sample", [{extra_config_string, "alog_path="}]).' http://localhost:8091/diag/eval
# wget output removed

# Restarted Couchbase

$./cbstats localhost:11210 config -b default |grep alog_path
 ep_alog_path:

$./cbstats localhost:11210 config -b beer-sample |grep alog_path
# no output
Comment by Brent Woodruff [ 26/Aug/14 ]
Note: In the first screen shot showing the buckets, both buckets were availalbe, I just did not think to get the green indicator since my goal was just to show that there was more than one bucket configured.
Comment by Abhinav Dangeti [ 26/Aug/14 ]
Brent, I did not see this issue when I used a couchbase instance started by cluster_run. However I do see this issue with a mac build for some reason.
Note that this issue is not because of having multiple buckets in the cluster. I see this issue with a single bucket as well.

The duplicate entry warning showed just once while in cluster_run. With the build however, the babysitter logs are flooded with those messages:

37899 memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
37900 memcached<0.89.0>: Unsupported key: <Vû^F^A>
37901 memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
37902 memcached<0.89.0>: Unsupported key: <<86>^C^G^A>
37903 memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
37904 memcached<0.89.0>: Unsupported key: <öê^F^A>
37905 memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
37906 memcached<0.89.0>: Unsupported key: <&ó^F^A>
37907 memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
37908 memcached<0.89.0>: Unsupported key: <Vû^F^A>
...
Comment by Abhinav Dangeti [ 26/Aug/14 ]
However, this issue already seems to be resolved in 3.0. Verified with 3.0.0-1175-rel.
Comment by Abhinav Dangeti [ 27/Aug/14 ]
Please re-open if you see this with 3.0
Comment by Brent Woodruff [ 29/Aug/14 ]
Confirmed that this does not appear to be a problem with 3.0 beta 2 (3.0.0-974) on Mac. Thanks for the investigation.




[MB-11804] [Windows] Memcached error #132 'Internal error': Internal error for vbucket... when set key to bucket Created: 23/Jul/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Thuan Nguyen Assignee: Thuan Nguyen
Resolution: Fixed Votes: 0
Labels: windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: windows 2008 R2 64-bit

Attachments: Zip Archive 172.23.107.124-7232014-1631-diag.zip     Zip Archive 172.23.107.125-7232014-1633-diag.zip     Zip Archive 172.23.107.126-7232014-1634-diag.zip     Zip Archive 172.23.107.127-7232014-1635-diag.zip    
Triage: Untriaged
Operating System: Windows 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: Link to manifest file of this build from centos build. http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_3.0.0-999-rel.rpm.manifest.xml
Is this a Regression?: Yes

 Description   
Test warmup test in build 3.0.0-999 on 4 nodes windows 2008 R2 64-bit
python testrunner.py -i ../../ini/4-w-sanity-new.ini -t warmupcluster.WarmUpClusterTest.test_warmUpCluster,num_of_docs=100

The test failed when it loaded keys to bucket default. This test passed in both centos 6.4 and ubuntu 12.04 64-bit


 Comments   
Comment by Sriram Ganesan [ 06/Aug/14 ]
The error here seems similar to one of the issues that was fixed a while ago MB-9990.
Comment by Sriram Ganesan [ 06/Aug/14 ]
I am getting the following error in a 2 node windows cluster

Memcached error #132 'Internal error': Internal error for vbucket :589 to mc 10.2.1.65:11211

and AFAIK, 11211 is the moxi port.
Comment by Chiyoung Seo [ 29/Aug/14 ]
http://review.couchbase.org/41065

Merged.
Comment by Thuan Nguyen [ 29/Aug/14 ]
Tested on build 3.0.1-1240 on windows. I could not reproduce this bug.




[MB-10262] corrupted key in data file rolls backwards to an earlier version or disappears without detection Created: 19/Feb/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.2.0
Fix Version/s: 3.0.1, 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Ruth Harris
Resolution: Fixed Votes: 0
Labels: corrupt
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Observed on Mac OS X, but presumed to affect all versions

Triage: Triaged
Flagged:
Release Note
Is this a Regression?: Yes

 Description   
By shutting down Couchbase Server, intentionally corrupting one recently stored key, then starting up the server and trying to read said key, an older version of that key is seen. The corruption wasn't logged (that I could find).

Note, the actual component here is couchstore.

Steps to reproduce:
1) Add a new document to a given bucket. Call the key something known, like "corruptme"
2) Edit the document once (so you'll have two versions of it)
3) Shut down the server
4) grep for that string in the vbucket data files
5) Edit the vbucket file for the given key. Change "corruptme" to "corruptm3"
6) Start the server
7) Perform a get for the given key (with cbc or the like)

Expected behavior: either the right key is returned (assumes replicated metadata) or an error is returned.

Observed behavior: the old version of the key is returned.


The probability of encountering this goes up dramatically in environments where there are many nodes, disks.

Related reading:
http://static.googleusercontent.com/media/research.google.com/en/us/archive/disk_failures.pdf

 Comments   
Comment by Anil Kumar [ 17/Jul/14 ]
Triage - Chiyoung, Anil, Venu, Wayne .. July 17th
Comment by Chiyoung Seo [ 30/Jul/14 ]
This is not a regression in 3.0, but has been there since 2.0 release. Given the 3.0 release schedule and the list of 3.0 blockers that we have as of this time, we will fix this issue in 3.0.1.
Comment by Sundar Sridharan [ 01/Aug/14 ]
Matt, I do not see this behavior in 3.0 cluster. If I shutdown the server and corrupt a key the subsequent warmup of the bucket returns fewer items. Just wondering how is it that you still see a corrupted key after warmup? thanks
Comment by Matt Ingenthron [ 01/Aug/14 ]
I don't "see a corrupted key" but rather corruption occurs, the item goes back to an older version when retrieved, and there's not even a warning in a log file. Another case I didn't check for was if the modified item was in the long past and there are good items after it.

Have you carried out the reproduction steps? Is there something that isn't clear?
Comment by Sundar Sridharan [ 01/Aug/14 ]
Matt, here are my steps and results, please help correct me if I am missing something..
1) Disable couchstore compression so the keys and values are visible also disable compaction.
2) Set key "corruptme" with value "test"
3) Set key "corruptme" with value "test" after a few seconds so there is a second commit writing the key again.
4) Shut down the server
5) grep for that string in the vbucket data files
6) Edit the vbucket file for the given key. Change "corruptme" to "corruptm3"
7) Start the server (warmup refuses to load key corruptm3)
8) Perform a get for the given key (with cbc or the like) (fails as there is no such key)
thanks
Comment by Matt Ingenthron [ 01/Aug/14 ]
Since the record should be checksummed, then either at the time of warmup or at the time of retrieving the corrupted entry, the checksum should not match. I would expect either an error, or a log message and a retrieval of an old value, or both. As it exists currently, if a bit changes for whatever reason, the corruption passes by silently. This could lead to very confusing and incorrect behavior in applications.
Comment by Sundar Sridharan [ 01/Aug/14 ]
We do log an error.. for example on my machine at warmup I see the following...
Fri Aug 1 14:14:39.886305 PDT 3: (default) couchstore_changes_since failed, error=checksum fail [none]
Fri Aug 1 14:14:39.886408 PDT 3: (default) couchstore_changes_since failed, error=checksum fail [none]
Do you believe this might not be sufficient?
thanks
Comment by Matt Ingenthron [ 04/Aug/14 ]
That definitely seems like different behavior from when I tried this. I think it's probably up to you and your team and PM on whether or not this is sufficient. From an application perspective, an item could have been stored, and then when later going to read it back an older version of it comes up with an error log message.

Comment by Chiyoung Seo [ 27/Aug/14 ]
As Sundar mentioned above, we log an error message when a given key is corrupted in a database file, and return its older version to the application if exists. I think that's enough at this time, but should be noted in 3.0 release note.

Ruth,

Please feel free to grab me if you need more information.




[MB-11955] [windows] could not re-create default bucket after delete default bucket Created: 13/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Thuan Nguyen Assignee: Abhinav Dangeti
Resolution: Fixed Votes: 0
Labels: windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: windows server 2008 R2 64-bit

Attachments: Zip Archive 172.23.107.124-8132014-1238-diag.zip     Zip Archive 172.23.107.125-8132014-1239-diag.zip     Zip Archive 172.23.107.126-8132014-1241-diag.zip    
Triage: Untriaged
Operating System: Windows 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: Link to manifest file of this build (I don't see manifest file for windows)

http://latestbuilds.hq.couchbase.com/couchbase-server-enterprise_x86_64_3.0.0-1144-rel.rpm.manifest.xml
Is this a Regression?: Yes

 Description   
Run sanity test on windows 2008 R2 64-bit on build 3.0.0-1144, there were a lot of failed tests.
Check error log, see test failed due to unable to create default bucket.

[2014-08-13 11:54:31,069] - [rest_client:1524] INFO - http://172.23.107.124:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=200&proxyPort=12211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=
[2014-08-13 11:54:31,078] - [rest_client:747] ERROR - http://172.23.107.124:8091/pools/default/buckets

Sanity tests passed in build 3.0.0-1139

 Comments   
Comment by Abhinav Dangeti [ 15/Aug/14 ]
Hey Tony, can I get one windows vm for testing here?
Comment by Thuan Nguyen [ 15/Aug/14 ]
I gave to Siram some physical windows servers.
Can you check with him if he still needs server with IP 10.17.1.166?
Comment by Abhinav Dangeti [ 15/Aug/14 ]
Bucket deletion doesn't complete, reason why you cannot recreate the bucket.
Comment by Thuan Nguyen [ 15/Aug/14 ]
This bug does not happen in build 3.0.0-1139 and previous build as shown in
http://qa.sc.couchbase.com/job/CouchbaseServer-SanityTest-4Node-Windows_2008_x64/
Comment by Abhinav Dangeti [ 15/Aug/14 ]
Changes between 1140 and 1142 to be precise.
Comment by Abhinav Dangeti [ 18/Aug/14 ]
Seeing multiple mcCouch connection failures in the logs:

Mon Aug 18 11:28:22.741780 Pacific Daylight Time 3: (default) Trying to connect to mccouch: "127.0.0.1:11213"
Mon Aug 18 11:28:22.741780 Pacific Daylight Time 3: (default) Connected to mccouch: "127.0.0.1:11213"
Mon Aug 18 11:28:22.748780 Pacific Daylight Time 3: (default) Failed to read from mccouch for select_bucket: "The operation completed successfully.

"
Mon Aug 18 11:28:22.748780 Pacific Daylight Time 3: (default) Resetting connection to mccouch, lastReceivedCommand = select_bucket lastSentCommand = select_bucket currentCommand =unknown
Mon Aug 18 11:28:22.748780 Pacific Daylight Time 3: (default) Trying to connect to mccouch: "127.0.0.1:11213"
Mon Aug 18 11:28:22.748780 Pacific Daylight Time 3: (default) Connected to mccouch: "127.0.0.1:11213"
Mon Aug 18 11:28:22.758780 Pacific Daylight Time 3: (No Engine) Bucket default registered with low priority
Mon Aug 18 11:28:22.758780 Pacific Daylight Time 3: (No Engine) Spawning zu readers, zu writers, zu auxIO, zu nonIO threads
Mon Aug 18 11:28:22.761781 Pacific Daylight Time 3: (default) metadata loaded in 1000 usec
Mon Aug 18 11:28:22.761781 Pacific Daylight Time 3: (default) Enough number of items loaded to enable traffic
Mon Aug 18 11:28:22.761781 Pacific Daylight Time 3: (default) warmup completed in 1000 usec
Mon Aug 18 11:28:23.839842 Pacific Daylight Time 3: (default) Failed to read from mccouch for notify_vbucket_update: "The operation completed successfully.

"
Mon Aug 18 11:28:23.839842 Pacific Daylight Time 3: (default) Resetting connection to mccouch, lastReceivedCommand = notify_vbucket_update lastSentCommand = notify_vbucket_update currentCommand =unknown
Mon Aug 18 11:28:23.839842 Pacific Daylight Time 3: (default) Trying to connect to mccouch: "127.0.0.1:11213"
Mon Aug 18 11:28:23.839842 Pacific Daylight Time 3: (default) Connected to mccouch: "127.0.0.1:11213"
Mon Aug 18 11:28:23.888845 Pacific Daylight Time 3: (default) Failed to read from mccouch for notify_vbucket_update: "The operation completed successfully.

"

...
Comment by Abhinav Dangeti [ 22/Aug/14 ]
These failures are very alike the ones seen in MB-11948.
Fix: http://review.couchbase.org/#/c/40865/
Comment by Sriram Ganesan [ 25/Aug/14 ]
Merged in the fix to handle the issue. Please verify and update the ticket if there are any more issues.
Comment by Thuan Nguyen [ 29/Aug/14 ]
Tested on build 3.0.1-1232, I could not reproduce this bug.




[MB-11935] Failed connections to 11213 causing 'Unable to listen' errors and server restarts Created: 12/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Mark Woosey Assignee: Chiyoung Seo
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
Triage: Untriaged
Operating System: Centos 64-bit
Is this a Regression?: Unknown

 Description   
Customer has been reporting "IP address seems to have changed. Unable to listen on 'ns_1@xxx.xxx.xxx.xxx'." errors. Investigation has revealed these to appear to be a result of failed 11213 connections.

 Comments   
Comment by Chiyoung Seo [ 12/Aug/14 ]
We plan to remove the mccouch communication dependency between ep-engine and ns-server in 3.0.1 release. That will fix this issue because we don't need to maintain the socket between these two components.
Comment by Chiyoung Seo [ 28/Aug/14 ]
We removed the mccouch communication between ep-engine and ns-server for 3.0.1 release:

http://review.couchbase.org/#/c/40195/

Consequently, we won't have any connection issues on the port 11213.




[MB-11722] Remove the couch notifier Created: 14/Jul/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Mike Wiederhold Assignee: Abhinav Dangeti
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump: Related to https://www.couchbase.com/issues/browse/MB-11935
Is this a Regression?: Unknown

 Comments   
Comment by Abhinav Dangeti [ 01/Aug/14 ]
ns_server: http://review.couchbase.org/#/c/38834/
ep-engine: http://review.couchbase.org/#/c/40195/

Unit tests, and make simple-test pass for now.
Setting up a toy build.
Comment by Abhinav Dangeti [ 01/Aug/14 ]
ToyBuild: http://latestbuilds.hq.couchbase.com/couchbase-server-community_cent58-3.0.0-toy-couchstore-x86_64_3.0.0-noMcCouch-toy.rpm
Comment by Chiyoung Seo [ 26/Aug/14 ]
The above changes were merged into the master branch.




[MB-11528] ep_bg_load and ep_bg_load_avg stats are missing in 2.5.1 Created: 24/Jun/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1
Fix Version/s: bug-backlog
Security Level: Public

Type: Task Priority: Major
Reporter: Patrick Varley Assignee: Chiyoung Seo
Resolution: Fixed Votes: 0
Labels: stats, supportability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
In 2.5.1 ns_server.stats.log or stats.log does not have ep_bg_load or ep_bg_load_avg.

Looking at the code it is still there.
http://src.couchbase.org/source/xref/2.5.1/ep-engine/src/ep_engine.cc#2665

 Comments   
Comment by Chiyoung Seo [ 28/Aug/14 ]
Those stats are available in 3.0 release.




[MB-12065] [Offline Upgrade 2.1.1-766 -> 3.0.0-1174] replica items mismatch after upgrade Created: 26/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sangharsh Agarwal Assignee: Abhinav Dangeti
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Ubuntu 64 bit (12.04)

Issue Links:
Gantt: start-finish
is triggered by MB-12075 Couchbase Server constantly shutting ... Resolved
Triage: Untriaged
Operating System: Ubuntu 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: [Source]
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12065/40f0a9b2/10.3.3.239-8222014-2216-diag.zip
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12065/7584aa98/10.3.3.239-8222014-2222-couch.tar.gz
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12065/a15d63ee/10.3.3.239-diag.txt
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12065/802987e6/10.3.3.240-8222014-2222-couch.tar.gz
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12065/c555bf6e/10.3.3.240-diag.txt
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12065/c7c31c83/10.3.3.240-8222014-2214-diag.zip


[Destination]
10.3.3.199 : https://s3.amazonaws.com/bugdb/jira/MB-12065/85abaa8a/10.3.3.199-8222014-2220-diag.zip
10.3.3.199 : https://s3.amazonaws.com/bugdb/jira/MB-12065/99f3fb96/10.3.3.199-diag.txt.gz
10.3.3.199 : https://s3.amazonaws.com/bugdb/jira/MB-12065/c11ea857/10.3.3.199-8222014-2222-couch.tar.gz
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12065/27bf3a49/10.3.3.218-8222014-2219-diag.zip
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12065/402915cc/10.3.3.218-8222014-2222-couch.tar.gz
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12065/851c2f72/10.3.3.218-diag.txt.gz
Is this a Regression?: Unknown

 Description   
http://qa.hq.northscale.net/job/ubuntu_x64--36_01--XDCR_upgrade-P1/36/consoleFull

[Test]
./testrunner -i ubuntu_x64--36_01--XDCR_upgrade-P1.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,upgrade_version=3.0.0-1174-rel,initial_vbuckets=1024 -t xdcr.upgradeXDCR.UpgradeTests.offline_cluster_upgrade,initial_version=2.1.1-766-rel,sdata=False,bucket_topology=default:1>2;bucket0:1><2,upgrade_nodes=dest;src,use_encryption_after_upgrade=src


[Test Steps]
1. Create 2-2 Nodes Source and Destination Cluster.

Source Nodes 10.3.3.240 (Master), 10.3.3.239
Source Nodes 10.3.3.218 (Master), 10.3.3.199


2. Setup CAPI Mode XDCR

bucket: bucket0 (BiXDCR)
bucket: default (UniXDCR)

3. Load 1000 items on bucket0 bucket on the both the cluster.
4. Load 1000 items on default bucket on Source cluster.
5. Offline upgrade both the cluster with 3.0.0-1174-rel.
6. Modify XDCR Settings to use SSL on Source Cluster Only*****.
7. Load 1000 Items on bucket0 and default bucket on Source.
8. Verify items.

Replica items mismatch on Source Cluster itself.

[2014-08-22 22:11:14,388] - [task:459] WARNING - Not Ready: vb_replica_curr_items 1500 == 2000 expected on '10.3.3.240:8091''10.3.3.239:8091', default bucket
[2014-08-22 22:11:16,410] - [task:459] WARNING - Not Ready: vb_replica_curr_items 2500 == 3000 expected on '10.3.3.240:8091''10.3.3.239:8091', bucket0 bucket
[2014-08-22 22:11:19,432] - [task:459] WARNING - Not Ready: vb_replica_curr_items 1500 == 2000 expected on '10.3.3.240:8091''10.3.3.239:8091', default bucket





 Comments   
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Some updates from the test Logs:

1. After upgrade, we see the errors that test were not able to connect with 10.3.3.239:

[2014-08-22 22:08:44,291] - [data_helper:295] INFO - creating direct client 10.3.3.239:11210 bucket0
[2014-08-22 22:08:44,440] - [data_helper:295] INFO - creating direct client 10.3.3.240:11210 bucket0
[2014-08-22 22:08:45,351] - [task:772] INFO - Batch create documents done #: 0 with exp:0
[2014-08-22 22:08:45,691] - [task:772] INFO - Batch create documents done #: 1000 with exp:0
[2014-08-22 22:08:46,955] - [data_helper:295] INFO - creating direct client 10.3.3.239:11210 default
[2014-08-22 22:08:46,970] - [rest_client:750] ERROR - socket error while connecting to http://10.3.3.239:8091/pools/default/buckets/default?basic_stats=true error [Errno 111] Connection refused
[2014-08-22 22:08:47,974] - [rest_client:750] ERROR - socket error while connecting to http://10.3.3.239:8091/pools/default/buckets/default?basic_stats=true error [Errno 111] Connection refused
[2014-08-22 22:08:49,172] - [data_helper:295] INFO - creating direct client 10.3.3.240:11210 default
[2014-08-22 22:08:49,356] - [task:772] INFO - Batch create documents done #: 0 with exp:0
[2014-08-22 22:08:49,753] - [task:772] INFO - Batch create documents done #: 1000 with exp:0


2. Test is passed on CentOS too.
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
There were lot of logs, where buckets are going to shutdown here:

2014-08-22 22:08:30.085 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:30.085 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "default" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:32.731 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:32.862 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "default" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:32.915 mb_master:0:warning:message(ns_1@10.3.3.239) - Somebody thinks we're master. Not forcing mastership takover over ourselves
2014-08-22 22:08:32.962 menelaus_sup:1:info:web start ok(ns_1@10.3.3.239) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:08:32.997 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "default" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:33.035 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:35.795 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:35.925 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "default" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:35.992 mb_master:0:warning:message(ns_1@10.3.3.239) - Somebody thinks we're master. Not forcing mastership takover over ourselves
2014-08-22 22:08:36.033 menelaus_sup:1:info:web start ok(ns_1@10.3.3.239) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:08:36.069 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "default" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:36.073 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:39.216 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:39.292 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "default" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:39.368 mb_master:0:warning:message(ns_1@10.3.3.239) - Somebody thinks we're master. Not forcing mastership takover over ourselves
2014-08-22 22:08:39.404 menelaus_sup:1:info:web start ok(ns_1@10.3.3.239) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".



[ns_server.info.log]

[user:info,2014-08-22T22:08:00.713,ns_1@10.3.3.239:ns_memcached-default<0.12208.0>:ns_memcached:terminate:783]Shutting down bucket "default" on 'ns_1@10.3.3.239' for server shutdown
[ns_server:info,2014-08-22T22:08:00.769,ns_1@10.3.3.239:ns_memcached-default<0.12208.0>:ns_memcached:terminate:795]This bucket shutdown is not due to bucket deletion or reconfiguration. Doing nothing
[ns_server:info,2014-08-22T22:08:01.801,ns_1@10.3.3.239:ns_server_sup<0.13612.0>:dir_size:start_link:49]Starting quick version of dir_size with program name: i386-linux-godu
[ns_server:info,2014-08-22T22:08:01.816,ns_1@10.3.3.239:ns_config_rep<0.13631.0>:ns_config_rep:do_pull:343]Pulling config from: 'ns_1@10.3.3.240'

[user:warn,2014-08-22T22:08:01.838,ns_1@10.3.3.239:ns_server_sup<0.13612.0>:mb_master:check_master_takeover_needed:153]Somebody thinks we're master. Not forcing mastership takover over ourselves
[ns_server:info,2014-08-22T22:08:01.864,ns_1@10.3.3.239:ns_doctor<0.13655.0>:ns_doctor:update_status:235]The following buckets became ready on node 'ns_1@10.3.3.240': ["bucket0",
                                                               "default"]
[user:info,2014-08-22T22:08:01.879,ns_1@10.3.3.239:ns_server_sup<0.13612.0>:menelaus_sup:start_link:44]Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".
[ns_server:info,2014-08-22T22:08:01.880,ns_1@10.3.3.239:<0.13739.0>:mc_tcp_listener:init:24]mccouch is listening on port 11213
[ns_server:info,2014-08-22T22:08:01.882,ns_1@10.3.3.239:<0.13743.0>:ns_memcached_log_rotator:init:28]Starting log rotator on "/opt/couchbase/var/lib/couchbase/logs"/"memcached.log"* with an initial period of 39003ms
[ns_server:info,2014-08-22T22:08:01.900,ns_1@10.3.3.239:<0.13807.0>:compaction_new_daemon:spawn_scheduled_kv_compactor:468]Start compaction of vbuckets for bucket default with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2014-08-22T22:08:01.903,ns_1@10.3.3.239:janitor_agent-default<0.13805.0>:janitor_agent:read_flush_counter:1048]Loading flushseq failed: {error,enoent}. Assuming it's equal to global config.
[ns_server:info,2014-08-22T22:08:01.904,ns_1@10.3.3.239:ns_memcached-default<0.13784.0>:ns_memcached:handle_cast:675]Main ns_memcached connection established: {ok,#Port<0.9250>}
[ns_server:info,2014-08-22T22:08:01.904,ns_1@10.3.3.239:janitor_agent-bucket0<0.13811.0>:janitor_agent:read_flush_counter:1048]Loading flushseq failed: {error,enoent}. Assuming it's equal to global config.

Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Trying to reproduce this issue to give you live cluster.
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
There is one more issued logged MB-12066. Not sure of this issue is similar to MB-12066 as in this bug only replica items are mismatch.
Comment by Abhinav Dangeti [ 26/Aug/14 ]
I see from the diags of 10.3.3.240, that the node is continuously restarting after upgrade:

2014-08-22 22:14:52.811 menelaus_sup:1:info:web start ok(ns_1@10.3.3.239) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:14:52.860 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "default" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:14:52.885 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:14:54.853 ns_memcached:0:info:message(ns_1@10.3.3.240) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.240' for server shutdown
2014-08-22 22:14:55.256 ns_memcached:0:info:message(ns_1@10.3.3.240) - Shutting down bucket "default" on 'ns_1@10.3.3.240' for server shutdown
2014-08-22 22:14:56.166 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:14:56.452 menelaus_sup:1:info:web start ok(ns_1@10.3.3.240) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.240'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:14:56.509 ns_memcached:0:info:message(ns_1@10.3.3.240) - Bucket "default" loaded on node 'ns_1@10.3.3.240' in 0 seconds.
2014-08-22 22:14:56.510 ns_memcached:0:info:message(ns_1@10.3.3.240) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.240' in 0 seconds.
2014-08-22 22:14:56.538 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "default" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:14:57.705 menelaus_sup:1:info:web start ok(ns_1@10.3.3.239) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:14:57.782 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:14:57.819 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "default" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:15:00.409 ns_memcached:0:info:message(ns_1@10.3.3.240) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.240' for server shutdown
2014-08-22 22:15:00.693 ns_memcached:0:info:message(ns_1@10.3.3.240) - Shutting down bucket "default" on 'ns_1@10.3.3.240' for server shutdown
2014-08-22 22:15:00.917 menelaus_sup:1:info:web start ok(ns_1@10.3.3.240) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.240'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:15:00.972 ns_memcached:0:info:message(ns_1@10.3.3.240) - Bucket "default" loaded on node 'ns_1@10.3.3.240' in 0 seconds.

This can cause replicators to break, so I'm going to mark this as a duplicate of MB-12075.




[MB-12057] apparent deadlock in ep-engine/bucket-engine (was: node_in is in pending state/ unable to restart cb service there/Rebalance exited with reason {not_all_nodes_are_ready_yet ) Created: 24/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Andrei Baranouski
Resolution: Fixed Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1174

Triage: Triaged
Is this a Regression?: Unknown

 Description   
steps:
1)run data load ~12hours on source cluster http://172.23.105.156/
2) then start replication for all 4 buckets on destination nodes(172.23.105.159, 172.23.105.160, 172.23.105.206)
3) almost immediately after step#2 add 172.23.105.207 to destination cluster and rebalance


Rebalance exited with reason {not_all_nodes_are_ready_yet,
['ns_1@172.23.105.207']}
ns_orchestrator002 ns_1@172.23.105.159 10:49:49 - Sun Aug 24, 2014
Started rebalancing bucket UserInfo ns_rebalancer000 ns_1@172.23.105.159 10:48:49 - Sun Aug 24, 2014
Starting rebalance, KeepNodes = ['ns_1@172.23.105.159','ns_1@172.23.105.160',
'ns_1@172.23.105.206','ns_1@172.23.105.207'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
ns_orchestrator004 ns_1@172.23.105.159 10:48:49 - Sun Aug 24, 2014
Control connection to memcached on 'ns_1@172.23.105.207' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
cmd_vocal_recv,
5,
[{file,
"src/mc_client_binary.erl"},
{line,
151}]},
{mc_client_binary,
select_bucket,
2,
[{file,
"src/mc_client_binary.erl"},
{line,
346}]},
{ns_memcached,
ensure_bucket,
2,
[{file,
"src/ns_memcached.erl"},
{line,
1269}]},
{ns_memcached,
handle_info,
2,
[{file,
"src/ns_memcached.erl"},
{line,
744}]},
{gen_server,
handle_msg,
5,
[{file,
"gen_server.erl"},
{line,
604}]},
{ns_memcached,
init,
1,
[{file,
"src/ns_memcached.erl"},
{line,
171}]},
{gen_server,
init_it,
6,
[{file,
"gen_server.erl"},
{line,
304}]},
{proc_lib,
init_p_do_apply,
3,
[{file,
"proc_lib.erl"},
{line,
239}]}]} (repeated 1 times) ns_memcached000 ns_1@172.23.105.207 10:44:42 - Sun Aug 24, 2014
Control connection to memcached on 'ns_1@172.23.105.207' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
cmd_vocal_recv,
5,
[{file,
"src/mc_client_binary.erl"},
{line,
151}]},
{mc_client_binary,
select_bucket,
2,
[{file,
"src/mc_client_binary.erl"},
{line,
346}]},
{ns_memcached,
ensure_bucket,
2,
[{file,
"src/ns_memcached.erl"},
{line,
1269}]},
{ns_memcached,
handle_info,
2,
[{file,
"src/ns_memcached.erl"},
{line,
744}]},
{gen_server,
handle_msg,
5,
[{file,
"gen_server.erl"},
{line,
604}]},
{ns_memcached,
init,
1,
[{file,
"src/ns_memcached.erl"},
{line,
171}]},
{gen_server,
init_it,
6,
[{file,
"gen_server.erl"},
{line,
304}]},
{proc_lib,
init_p_do_apply,
3,
[{file,
"proc_lib.erl"},
{line,
239}]}]} ns_memcached000 ns_1@172.23.105.207 10:43:56 - Sun Aug 24, 2014
Rebalance exited with reason {not_all_nodes_are_ready_yet,
['ns_1@172.23.105.207']}


trying to restart 172.23.105.207 node(firewall is turn off there):

[root@centos-64-x64 logs]# /etc/init.d/couchbase-server status
couchbase-server is running
[root@centos-64-x64 logs]# /etc/init.d/couchbase-server restart
Stopping couchbase-server
^C
BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution
a
                                                           [ OK ]
Starting couchbase-server [ OK ]
[root@centos-64-x64 logs]# /etc/init.d/couchbase-server status
couchbase-server is running
[root@centos-64-x64 logs]# /etc/init.d/couchbase-server stop
Stopping couchbase-serverNOTE: shutdown failed
{badrpc,nodedown}
                                                           [FAILED]
[root@centos-64-x64 logs]# /etc/init.d/couchbase-server start
couchbase-server is already started [WARNING]


cluster will be available a few hours
 

 Comments   
Comment by Andrei Baranouski [ 24/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12057/de39e575/172.23.105.159-8242014-1113-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12057/de39e575/172.23.105.160-8242014-1116-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12057/de39e575/172.23.105.206-8242014-1119-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12057/de39e575/172.23.105.207-8242014-1122-diag.zip
Comment by Aleksey Kondratenko [ 24/Aug/14 ]
Grabbing backtraces from memcached process on bad node might be very handy.
Comment by Andrei Baranouski [ 24/Aug/14 ]
[root@centos-64-x64 ~]# ps -ef| grep couch
root 3795 3469 0 12:17 pts/2 00:00:00 grep couch
498 27065 1 0 Aug20 ? 00:00:14 /opt/couchbase/lib/erlang/erts-5.10.4/bin/epmd -daemon
498 27102 1 0 Aug20 ? 00:01:05 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /opt/couchbase -- -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@127.0.0.1 -setcookie nocookie -noshell -noinput -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" -ns_server pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie-ns-server" -ns_server enable_mlockall false
498 27136 27102 3 Aug20 ? 02:52:05 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /opt/couchbase -- -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -nouser -run child_erlang child_start ns_bootstrap -- -smp enable -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
498 27171 27136 0 Aug20 ? 00:00:15 /opt/couchbase/lib/erlang/lib/os_mon-2.2.14/priv/bin/memsup
498 27172 27136 0 Aug20 ? 00:00:00 /opt/couchbase/lib/erlang/lib/os_mon-2.2.14/priv/bin/cpu_sup
498 27228 27102 0 Aug20 ? 00:17:42 /opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
498 31489 27136 0 10:36 ? 00:00:15 /opt/couchbase/lib/ns_server/erlang/lib/ns_server/priv/i386-linux-godu
[root@centos-64-x64 ~]# gdb -p 27102
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-64.el6_5.2)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 27102
Reading symbols from /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp...done.
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 27130]
[New LWP 27129]
[New LWP 27128]
[New LWP 27127]
[New LWP 27126]
[New LWP 27125]
[New LWP 27124]
[New LWP 27123]
[New LWP 27122]
[New LWP 27121]
[New LWP 27120]
[New LWP 27119]
[New LWP 27118]
[New LWP 27117]
[New LWP 27116]
[New LWP 27115]
[New LWP 27114]
[New LWP 27113]
[New LWP 27112]
[New LWP 27111]
[New LWP 27110]
[New LWP 27109]
[New LWP 27108]
[New LWP 27107]
[New LWP 27106]
[New LWP 27105]
[New LWP 27104]
[New LWP 27103]
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
0x00007f4f4c9614f3 in select () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install couchbase-server-3.0.0-1174.x86_64
(gdb) thread app all bt

Thread 29 (Thread 0x7f4f4af7f700 (LWP 27103)):
#0 0x00007f4f4ce2954d in read () from /lib64/libpthread.so.0
#1 0x0000000000550641 in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2916
#2 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178974f0) at pthread/ethread.c:106
#3 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f4f4a0ff700 (LWP 27104)):
#0 0x00007f4f4ce2643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005a7699 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1368
#2 0x0000000000463e3f in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:1788
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:938
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3286
#5 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975d0) at pthread/ethread.c:106
#6 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f4f4a57e700 (LWP 27105)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80138) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80138) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4bec9c40) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4bec9c40) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4bec9c40) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f4f496fe700 (LWP 27106)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80178) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80178) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4bec9d80) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4bec9d80) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4bec9d80) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f4f496dc700 (LWP 27107)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af801b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af801b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4bec9ec0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4bec9ec0) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4bec9ec0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f4f496ba700 (LWP 27108)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af801f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af801f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca000) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca000) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca000) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f4f49698700 (LWP 27109)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80238) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80238) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca140) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca140) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca140) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f4f49676700 (LWP 27110)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80278) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80278) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca280) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca280) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca280) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f4f49654700 (LWP 27111)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af802b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af802b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca3c0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca3c0) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca3c0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f4f49632700 (LWP 27112)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af802f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af802f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca500) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca500) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca500) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 19 (Thread 0x7f4f49610700 (LWP 27113)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80338) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80338) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca640) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca640) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca640) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f4f495ee700 (LWP 27114)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80378) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80378) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca780) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca780) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca780) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f4f495cc700 (LWP 27115)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af803b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af803b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca8c0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca8c0) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca8c0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f4f495aa700 (LWP 27116)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af803f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af803f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4becaa00) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4becaa00) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4becaa00) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f4f49588700 (LWP 27117)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80438) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80438) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4becab40) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4becab40) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4becab40) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f4f49566700 (LWP 27118)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80478) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80478) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4becac80) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4becac80) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4becac80) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f4f49544700 (LWP 27119)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af804b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af804b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4becadc0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4becadc0) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4becadc0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f4f49522700 (LWP 27120)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af804f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af804f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4becaf00) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4becaf00) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4becaf00) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f4f4d913700 (LWP 27121)):
#0 0x00007f4f4ce2a09d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000054f158 in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2840
#2 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897520) at pthread/ethread.c:106
#3 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f4f48ebf700 (LWP 27122)):
#0 0x00007f4f4c95f253 in poll () from /lib64/libc.so.6
#1 0x000000000055a971 in check_fd_events (ps=0x7f4f4bdd4910, pr=0x7f4f48ebe320, len=0x7f4f48ebeb3c, utvp=<value optimized out>) at sys/common/erl_poll.c:2071
#2 erts_poll_wait_nkp (ps=0x7f4f4bdd4910, pr=0x7f4f48ebe320, len=0x7f4f48ebeb3c, utvp=<value optimized out>) at sys/common/erl_poll.c:2184
#3 0x000000000055d5c3 in erts_check_io_nkp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1183

#4 0x00000000004989d6 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a442340, rq=0x7f4f4a440b40) at beam/erl_process.c:2533
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a442340) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f4f484be700 (LWP 27123)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af805b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af805b8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a44c600, rq=0x7f4f4a440cc0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a44c600, rq=0x7f4f4a440cc0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a44c600) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f4f47abd700 (LWP 27124)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af805f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af805f8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a4568c0, rq=0x7f4f4a440e40) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a4568c0, rq=0x7f4f4a440e40) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a4568c0) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f4f470bc700 (LWP 27125)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80638) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80638) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a460b80, rq=0x7f4f4a440fc0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a460b80, rq=0x7f4f4a440fc0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a460b80) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f4f466bb700 (LWP 27126)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80678) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80678) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a46ae40, rq=0x7f4f4a441140) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a46ae40, rq=0x7f4f4a441140) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a46ae40) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 5 (Thread 0x7f4f45cba700 (LWP 27127)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af806b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af806b8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a475100, rq=0x7f4f4a4412c0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a475100, rq=0x7f4f4a4412c0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a475100) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f4f452b9700 (LWP 27128)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af806f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af806f8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a47f3c0, rq=0x7f4f4a441440) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a47f3c0, rq=0x7f4f4a441440) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a47f3c0) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f4f448b8700 (LWP 27129)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80738) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80738) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a489680, rq=0x7f4f4a4415c0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a489680, rq=0x7f4f4a4415c0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a489680) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f4f43eb7700 (LWP 27130)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80778) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80778) at pthread/ethr_event.c:218
#3 0x0000000000498c8e in erts_tse_wait (unused=<value optimized out>) at beam/erl_threads.h:2710
#4 aux_thread (unused=<value optimized out>) at beam/erl_process.c:2272
#5 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#6 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f4f4dafa700 (LWP 27102)):
#0 0x00007f4f4c9614f3 in select () from /lib64/libc.so.6
#1 0x0000000000550a70 in erts_sys_main_thread () at sys/unix/sys.c:3059
---Type <return> to continue, or q <return> to quit---
#2 0x00000000004490b9 in erl_start (argc=56, argv=<value optimized out>) at beam/erl_init.c:1775
#3 0x00000000004273a9 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29
(gdb)
(gdb)
(gdb)
(gdb)
(gdb)
(gdb) q
A debugging session is active.

Inferior 1 [process 27102] will be detached.

Quit anyway? (y or n) Y
Detaching from program: /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp, process 27102
[root@centos-64-x64 ~]# gdb -p 27136
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-64.el6_5.2)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 27136
Reading symbols from /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp...done.

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected address (wrong library or version mismatch?)
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 27174]
[New LWP 27167]
[New LWP 27166]
[New LWP 27165]
[New LWP 27164]
[New LWP 27163]
[New LWP 27162]
[New LWP 27161]
[New LWP 27160]
[New LWP 27159]
[New LWP 27158]
[New LWP 27157]
[New LWP 27156]
[New LWP 27155]
[New LWP 27154]
[New LWP 27153]
[New LWP 27152]
[New LWP 27151]
[New LWP 27150]
[New LWP 27149]
[New LWP 27148]
[New LWP 27147]
[New LWP 27146]
[New LWP 27145]
[New LWP 27144]
[New LWP 27143]
[New LWP 27142]
[New LWP 27141]
[New LWP 27140]
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-3.2/priv/lib/crypto.so...done.
Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-3.2/priv/lib/crypto.so
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-3.2/priv/lib/crypto_callback.so...done.
Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-3.2/priv/lib/crypto_callback.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-961ad59-git/priv/lib/couch_icu_driver.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-961ad59-git/priv/lib/couch_icu_driver.so
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
0x00007f7bfcfdc4f3 in select () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install couchbase-server-3.0.0-1174.x86_64
(gdb) thread app all bt

Thread 30 (Thread 0x7f7bfb5ff700 (LWP 27140)):
#0 0x00007f7bfd4a454d in read () from /lib64/libpthread.so.0
#1 0x0000000000550641 in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2916
#2 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e070) at pthread/ethread.c:106
#3 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f7bfa37f700 (LWP 27141)):
#0 0x00007f7bfd4a143c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005a7699 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1368
#2 0x0000000000463e3f in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:1788
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:938
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3286
#5 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e150) at pthread/ethread.c:106
#6 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f7bfabfe700 (LWP 27142)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600138) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600138) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2380) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2380) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2380) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f7bf997e700 (LWP 27143)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600178) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600178) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb24c0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb24c0) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb24c0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f7bf973f700 (LWP 27144)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6001b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6001b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2600) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2600) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2600) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f7bf971d700 (LWP 27145)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6001f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6001f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2740) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2740) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2740) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f7bf96fb700 (LWP 27146)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600238) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600238) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2880) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2880) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2880) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f7bf96d9700 (LWP 27147)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600278) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600278) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb29c0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb29c0) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb29c0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f7bf96b7700 (LWP 27148)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6002b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6002b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2b00) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2b00) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2b00) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f7bf9695700 (LWP 27149)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6002f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6002f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2c40) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2c40) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2c40) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 20 (Thread 0x7f7bf9673700 (LWP 27150)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600338) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600338) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2d80) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2d80) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2d80) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f7bf9651700 (LWP 27151)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600378) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600378) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2ec0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2ec0) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2ec0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f7bf962f700 (LWP 27152)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6003b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6003b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb3000) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb3000) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb3000) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f7bf960d700 (LWP 27153)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6003f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6003f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb3140) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb3140) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb3140) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f7bf95eb700 (LWP 27154)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600438) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600438) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb3280) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb3280) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb3280) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f7bf95c9700 (LWP 27155)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600478) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600478) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb33c0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb33c0) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb33c0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f7bf95a7700 (LWP 27156)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6004b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6004b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb3500) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb3500) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb3500) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f7bf9585700 (LWP 27157)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6004f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6004f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb3640) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb3640) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb3640) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f7bfdf8e700 (LWP 27158)):
#0 0x00007f7bfd4a509d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000054f158 in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2840
#2 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e0a0) at pthread/ethread.c:106
#3 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f7bf90ff700 (LWP 27159)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600578) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600578) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4e3680, rq=0x7f7bfc4e1e40) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4e3680, rq=0x7f7bfc4e1e40) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc4e3680) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f7bf86fe700 (LWP 27160)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6005b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6005b8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4ed940, rq=0x7f7bfc4e1fc0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4ed940, rq=0x7f7bfc4e1fc0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc4ed940) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f7bf7cfd700 (LWP 27161)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6005f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6005f8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4f7c00, rq=0x7f7bfc4e2140) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4f7c00, rq=0x7f7bfc4e2140) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc4f7c00) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f7bf72fc700 (LWP 27162)):
#0 0x00007f7bfcfe3f03 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000555b6c in check_fd_events (ps=0x7f7bfc454910, pr=0x7f7bf72fb320, len=0x7f7bf72fbb3c, utvp=0x7f7bf72fbb20) at sys/common/erl_poll.c:2023
#2 erts_poll_wait_kp (ps=0x7f7bfc454910, pr=0x7f7bf72fb320, len=0x7f7bf72fbb3c, utvp=0x7f7bf72fbb20) at sys/common/erl_poll.c:2184
#3 0x00000000005589b3 in erts_check_io_kp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1183
#4 0x00000000004989d6 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc501ec0, rq=0x7f7bfc4e22c0) at beam/erl_process.c:2533
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc501ec0) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f7bf68fb700 (LWP 27163)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600678) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600678) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc50c180, rq=0x7f7bfc4e2440) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc50c180, rq=0x7f7bfc4e2440) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc50c180) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 6 (Thread 0x7f7bf5efa700 (LWP 27164)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6006b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6006b8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc516440, rq=0x7f7bfc4e25c0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc516440, rq=0x7f7bfc4e25c0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc516440) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f7bf54f9700 (LWP 27165)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6006f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6006f8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc520700, rq=0x7f7bfc4e2740) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc520700, rq=0x7f7bfc4e2740) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc520700) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f7bf4af8700 (LWP 27166)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600738) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600738) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc52a9c0, rq=0x7f7bfc4e28c0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc52a9c0, rq=0x7f7bfc4e28c0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc52a9c0) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f7bf40f7700 (LWP 27167)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600778) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600778) at pthread/ethr_event.c:218
#3 0x0000000000498c8e in erts_tse_wait (unused=<value optimized out>) at beam/erl_threads.h:2710
#4 aux_thread (unused=<value optimized out>) at beam/erl_process.c:2272
#5 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#6 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f7ba97aa700 (LWP 27174)):
#0 0x00007f7bfd4a4d2d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f7baa112c21 in terminatorLoop (args=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchdb/src/mapreduce/mapreduce_nif.cc:480
---Type <return> to continue, or q <return> to quit---
#2 0x00000000005a95d6 in thr_wrapper (vtwd=0x7f7bf90fead0) at pthread/ethread.c:106
#3 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f7bfe175700 (LWP 27136)):
#0 0x00007f7bfcfdc4f3 in select () from /lib64/libc.so.6
#1 0x0000000000550a70 in erts_sys_main_thread () at sys/unix/sys.c:3059
#2 0x00000000004490b9 in erl_start (argc=46, argv=<value optimized out>) at beam/erl_init.c:1775
#3 0x00000000004273a9 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29
Comment by Aleksey Kondratenko [ 24/Aug/14 ]
beam's backtraces are irrelevant
Comment by Andrei Baranouski [ 24/Aug/14 ]
gdb -p 27228
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-64.el6_5.2)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 27228
Reading symbols from /opt/couchbase/bin/memcached...done.

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected address (wrong library or version mismatch?)
Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0
Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so...done.
Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so
Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1
Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4
Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done.
Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.6
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 31578]
[New LWP 31577]
[New LWP 31576]
[New LWP 31575]
[New LWP 31574]
[New LWP 31573]
[New LWP 31572]
[New LWP 31571]
[New LWP 31570]
[New LWP 31569]
[New LWP 31568]
[New LWP 27271]
[New LWP 27270]
[New LWP 27269]
[New LWP 27268]
[New LWP 27267]
[New LWP 27266]
[New LWP 27265]
[New LWP 27242]
[New LWP 27241]
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so
Reading symbols from /opt/couchbase/lib/libdirutils.so.0.1.0...done.
Loaded symbols for /opt/couchbase/lib/libdirutils.so.0.1.0
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
0x00007f78968df0ad in pthread_join () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install couchbase-server-3.0.0-1174.x86_64
(gdb) thread app all bt

Thread 21 (Thread 0x7f78943df700 (LWP 27241)):
#0 0x00007f7895a7360d in read () from /lib64/libc.so.6
#1 0x00007f7895a09f68 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007f7895a0ba6e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007f7895a0014a in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007f78959fefa9 in fgets () from /lib64/libc.so.6
#5 0x00007f78943e08b1 in check_stdin_thread (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#6 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce0e0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#7 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f78937db700 (LWP 27242)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x7f78939de6e0, mutex=0x7f78939de6a0, ms=<value optimized out>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f78937ddd53 in logger_thead_main (arg=0x140e900) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:342
#3 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce080) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f7892bce700 (LWP 27265)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007f7892bd2db0 in lock_engines (name=0x13d23f0 "RevAB") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:356
#5 find_bucket (name=0x13d23f0 "RevAB") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:715
#6 0x00007f7892bd45d0 in handle_select_bucket (handle=0x7f7892dda840, cookie=0x5b51600, request=0x5b85000, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:3054
#7 bucket_unknown_command (handle=0x7f7892dda840, cookie=0x5b51600, request=0x5b85000, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:3206
#8 0x0000000000418701 in process_bin_unknown_packet (c=0x5b51600) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:2616
#9 process_bin_packet (c=0x5b51600) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5414
#10 complete_nread (c=0x5b51600) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5818
#11 conn_nread (c=0x5b51600) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7032
#12 0x000000000040b4bd in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x5b51600)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7305
#13 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4280, flags=<value optimized out>) at event.c:1308
#14 event_process_active (base=0x5ce4280, flags=<value optimized out>) at event.c:1375
#15 event_base_loop (base=0x5ce4280, flags=<value optimized out>) at event.c:1572
#16 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce140) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#17 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f78921cd700 (LWP 27266)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007f7892bd2db0 in lock_engines (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:356
#5 find_bucket (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:715
#6 0x00007f7892bd3e02 in handle_connect (cookie=0x5be5b00, type=<value optimized out>, event_data=<value optimized out>, cb_data=0x7f7892dda840)
---Type <return> to continue, or q <return> to quit---
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1291
#7 0x000000000040dcaf in perform_callbacks (sfd=124, parent_port=11210, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4500, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:248
#8 conn_new (sfd=124, parent_port=11210, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4500, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:915
#9 0x00000000004198ea in thread_libevent_process (fd=<value optimized out>, which=<value optimized out>, arg=0x5cb2af0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:312
#10 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4500, flags=<value optimized out>) at event.c:1308
#11 event_process_active (base=0x5ce4500, flags=<value optimized out>) at event.c:1375
#12 event_base_loop (base=0x5ce4500, flags=<value optimized out>) at event.c:1572
#13 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce150) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#14 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f78917cc700 (LWP 27267)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007f7892bd2db0 in lock_engines (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:356
#5 find_bucket (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:715
#6 0x00007f7892bd3e02 in handle_connect (cookie=0x5c76300, type=<value optimized out>, event_data=<value optimized out>, cb_data=0x7f7892dda840)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1291
#7 0x000000000040dcaf in perform_callbacks (sfd=126, parent_port=11210, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4780, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:248
#8 conn_new (sfd=126, parent_port=11210, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4780, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:915
#9 0x00000000004198ea in thread_libevent_process (fd=<value optimized out>, which=<value optimized out>, arg=0x5cb2be0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:312
#10 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4780, flags=<value optimized out>) at event.c:1308
#11 event_process_active (base=0x5ce4780, flags=<value optimized out>) at event.c:1375
#12 event_base_loop (base=0x5ce4780, flags=<value optimized out>) at event.c:1572
#13 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce160) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#14 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f7890dcb700 (LWP 27268)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007f7892bd2db0 in lock_engines (name=0x1052b980 "UserInfo") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:356
#5 find_bucket (name=0x1052b980 "UserInfo") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:715
#6 0x00007f7892bd45d0 in handle_select_bucket (handle=0x7f7892dda840, cookie=0x5c79000, request=0x5ca3000, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:3054
#7 bucket_unknown_command (handle=0x7f7892dda840, cookie=0x5c79000, request=0x5ca3000, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:3206
#8 0x0000000000418701 in process_bin_unknown_packet (c=0x5c79000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:2616
#9 process_bin_packet (c=0x5c79000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5414
#10 complete_nread (c=0x5c79000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5818
#11 conn_nread (c=0x5c79000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7032
#12 0x000000000040b4bd in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x5c79000)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7305
---Type <return> to continue, or q <return> to quit---
#13 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4a00, flags=<value optimized out>) at event.c:1308
#14 event_process_active (base=0x5ce4a00, flags=<value optimized out>) at event.c:1375
#15 event_base_loop (base=0x5ce4a00, flags=<value optimized out>) at event.c:1572
#16 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce170) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#17 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f78903ca700 (LWP 27269)):
#0 0x00007f78968e243c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f788d8b671d in wait (this=0xd525b00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:39
#2 EventuallyPersistentStore::initialize (this=0xd525b00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:346

#3 0x00007f788d8c342c in EventuallyPersistentEngine::initialize (this=0x21072a00, config=<value optimized out>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:2011
#4 0x00007f788d8c3786 in EvpInitialize (handle=0x21072a00,
    config_str=0x1c106003 "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_size=314572800;tap_keepalive=300;dbname=/data/MsgsCalls;allow_data_loss_during_shutdown=true;backend=couchdb;couch_bucket=MsgsCalls;couch_port=11213;ma"...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:135
#5 0x00007f7892bd325b in create_bucket_UNLOCKED (e=0x7f7892dda840, bucket_name=0x1052c840 "MsgsCalls", path=0x1c105fe0 "/opt/couchbase/lib/memcached/ep.so",
    config=0x1c106003 "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_size=314572800;tap_keepalive=300;dbname=/data/MsgsCalls;allow_data_loss_during_shutdown=true;backend=couchdb;couch_bucket=MsgsCalls;couch_port=11213;ma"..., e_out=0x0, msg=0x7f78903c9730 "", msglen=1024) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:857
#6 0x00007f7892bd3496 in handle_create_bucket (handle=0x7f7892dda840, cookie=0x5b4fe00, request=<value optimized out>, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:2841
#7 0x00007f7892bd4979 in bucket_unknown_command (handle=0x7f7892dda840, cookie=0x5b4fe00, request=0x5b6f000, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:3193
#8 0x0000000000418701 in process_bin_unknown_packet (c=0x5b4fe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:2616
#9 process_bin_packet (c=0x5b4fe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5414
#10 complete_nread (c=0x5b4fe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5818
#11 conn_nread (c=0x5b4fe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7032
#12 0x000000000040b4bd in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x5b4fe00)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7305
#13 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4c80, flags=<value optimized out>) at event.c:1308
#14 event_process_active (base=0x5ce4c80, flags=<value optimized out>) at event.c:1375
#15 event_base_loop (base=0x5ce4c80, flags=<value optimized out>) at event.c:1572
#16 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce180) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#17 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f788f9c9700 (LWP 27270)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007f7892bd2db0 in lock_engines (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:356
#5 find_bucket (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:715
#6 0x00007f7892bd3e02 in handle_connect (cookie=0x5a30400, type=<value optimized out>, event_data=<value optimized out>, cb_data=0x7f7892dda840)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1291
#7 0x000000000040dcaf in perform_callbacks (sfd=91, parent_port=11209, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4f00, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:248
#8 conn_new (sfd=91, parent_port=11209, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4f00, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:915
#9 0x00000000004198ea in thread_libevent_process (fd=<value optimized out>, which=<value optimized out>, arg=0x5cb2eb0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:312
#10 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4f00, flags=<value optimized out>) at event.c:1308
#11 event_process_active (base=0x5ce4f00, flags=<value optimized out>) at event.c:1375
---Type <return> to continue, or q <return> to quit---
#12 event_base_loop (base=0x5ce4f00, flags=<value optimized out>) at event.c:1572
#13 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce190) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#14 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f788efc8700 (LWP 27271)):
#0 0x00007f7895a80f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f78970c0376 in epoll_dispatch (base=0x5ce5180, tv=<value optimized out>) at epoll.c:404
#2 0x00007f78970abc44 in event_base_loop (base=0x5ce5180, flags=<value optimized out>) at event.c:1558
#3 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce1a0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f788e5c7700 (LWP 31568)):
#0 0x00007f7895a44b8d in nanosleep () from /lib64/libc.so.6
#1 0x00007f7895a79d64 in usleep () from /lib64/libc.so.6
#2 0x00007f788d8f40b5 in updateStatsThread (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/memory_tracker.cc:36
#3 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce240) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f788b31f700 (LWP 31569)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x000000000041a236 in notify_io_complete (cookie=0x5c27800, status=ENGINE_SUCCESS) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:459
#5 0x00007f788d8ba536 in EventuallyPersistentEngine::notifyIOComplete(const void *, ._101) (this=0x621e000, cookie=0x5c27800, status=ENGINE_SUCCESS)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.h:478
#6 0x00007f788d8a43e3 in EventuallyPersistentStore::completeBGFetchMulti (this=0x5d4f8c0, vbId=<value optimized out>, fetchedItems=std::vector of length 4, capacity 4 = {...},
    startTime=1593905183966731) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:1620
#7 0x00007f788d88f0a2 in BgFetcher::doFetch (this=0x5d31ba0, vbId=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:90
#8 0x00007f788d88f817 in BgFetcher::run (this=0x5d31ba0, task=0x81161c0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:156
#9 0x00007f788d91b603 in BgFetcherTask::run (this=0xfffffffffffffe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:89

#10 0x00007f788d8f5b3a in ExecutorThread::run (this=0x5eba000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:110
#11 0x00007f788d8f6296 in launch_executor_thread (arg=0x5cb2e68) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#12 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cec50) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#13 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f788a91e700 (LWP 31570)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x000000000041a236 in notify_io_complete (cookie=0x5b03200, status=ENGINE_SUCCESS) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:459
#5 0x00007f788d8ba536 in EventuallyPersistentEngine::notifyIOComplete(const void *, ._101) (this=0x621e000, cookie=0x5b03200, status=ENGINE_SUCCESS)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.h:478
#6 0x00007f788d8a43e3 in EventuallyPersistentStore::completeBGFetchMulti (this=0x5d4f8c0, vbId=<value optimized out>, fetchedItems=std::vector of length 4, capacity 4 = {...},
    startTime=1593905182650801) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:1620
#7 0x00007f788d88f0a2 in BgFetcher::doFetch (this=0x5d32560, vbId=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:90
#8 0x00007f788d88f817 in BgFetcher::run (this=0x5d32560, task=0x7ccbc50) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:156
#9 0x00007f788d91b603 in BgFetcherTask::run (this=0xfffffffffffffe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:89
---Type <return> to continue, or q <return> to quit---
#10 0x00007f788d8f5b3a in ExecutorThread::run (this=0x5eba0e0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:110
#11 0x00007f788d8f6296 in launch_executor_thread (arg=0x5cb2e68) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#12 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cec70) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#13 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f7889f1d700 (LWP 31571)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x000000000041a236 in notify_io_complete (cookie=0x5b03500, status=ENGINE_SUCCESS) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:459
#5 0x00007f788d8ba536 in EventuallyPersistentEngine::notifyIOComplete(const void *, ._101) (this=0x621e000, cookie=0x5b03500, status=ENGINE_SUCCESS)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.h:478
#6 0x00007f788d8a43e3 in EventuallyPersistentStore::completeBGFetchMulti (this=0x5d4f8c0, vbId=<value optimized out>, fetchedItems=std::vector of length 1, capacity 1 = {...},
    startTime=1593905182702662) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:1620
#7 0x00007f788d88f0a2 in BgFetcher::doFetch (this=0x5d305b0, vbId=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:90
#8 0x00007f788d88f817 in BgFetcher::run (this=0x5d305b0, task=0x8115d60) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:156
#9 0x00007f788d91b603 in BgFetcherTask::run (this=0xfffffffffffffe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:89
#10 0x00007f788d8f5b3a in ExecutorThread::run (this=0x5eba1c0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:110
#11 0x00007f788d8f6296 in launch_executor_thread (arg=0x5cb2e68) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#12 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cec60) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#13 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f788951c700 (LWP 31572)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x000000000041a236 in notify_io_complete (cookie=0x5b05300, status=ENGINE_SUCCESS) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:459
#5 0x00007f788d8ba536 in EventuallyPersistentEngine::notifyIOComplete(const void *, ._101) (this=0x621e000, cookie=0x5b05300, status=ENGINE_SUCCESS)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.h:478
#6 0x00007f788d8a43e3 in EventuallyPersistentStore::completeBGFetchMulti (this=0x5d4f8c0, vbId=<value optimized out>, fetchedItems=std::vector of length 4, capacity 4 = {...},
    startTime=1593905193178279) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:1620
#7 0x00007f788d88f0a2 in BgFetcher::doFetch (this=0x5d32630, vbId=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:90
#8 0x00007f788d88f817 in BgFetcher::run (this=0x5d32630, task=0x7cc8730) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:156
#9 0x00007f788d91b603 in BgFetcherTask::run (this=0xfffffffffffffe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:89
#10 0x00007f788d8f5b3a in ExecutorThread::run (this=0x5eba2a0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:110
#11 0x00007f788d8f6296 in launch_executor_thread (arg=0x5cb2e68) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#12 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cec80) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#13 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f7888b1b700 (LWP 31573)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x14370c0, mutex=0x1437088, ms=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f788d91d7b5 in TaskQueue::_doSleep (this=0x1437080, t=...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f788d92056f in TaskQueue::_fetchNextTask (this=0x1437080, t=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:98
#4 0x00007f788d92090e in TaskQueue::fetchNextTask (this=0x1437080, thread=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:142

#5 0x00007f788d8e397c in ExecutorPool::_nextTask (this=0x5eb1840, t=..., tick=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:214
#6 0x00007f788d8e3a4e in ExecutorPool::nextTask (this=0x5eb1840, t=..., tick=236 '\354') at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:229
#7 0x00007f788d8f597b in ExecutorThread::run (this=0x5eba380) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:78
---Type <return> to continue, or q <return> to quit---
#8 0x00007f788d8f6296 in launch_executor_thread (arg=0x14370c4) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#9 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cec90) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#10 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f788811a700 (LWP 31574)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x14370c0, mutex=0x1437088, ms=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f788d91d7b5 in TaskQueue::_doSleep (this=0x1437080, t=...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f788d92056f in TaskQueue::_fetchNextTask (this=0x1437080, t=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:98
#4 0x00007f788d92090e in TaskQueue::fetchNextTask (this=0x1437080, thread=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:142
#5 0x00007f788d8e397c in ExecutorPool::_nextTask (this=0x5eb1840, t=..., tick=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:214
#6 0x00007f788d8e3a4e in ExecutorPool::nextTask (this=0x5eb1840, t=..., tick=114 'r') at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:229
#7 0x00007f788d8f597b in ExecutorThread::run (this=0x5eba460) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:78
#8 0x00007f788d8f6296 in launch_executor_thread (arg=0x14370c4) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#9 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ceca0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#10 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f7887719700 (LWP 31575)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x14370c0, mutex=0x1437088, ms=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f788d91d7b5 in TaskQueue::_doSleep (this=0x1437080, t=...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f788d92056f in TaskQueue::_fetchNextTask (this=0x1437080, t=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:98
#4 0x00007f788d92090e in TaskQueue::fetchNextTask (this=0x1437080, thread=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:142
#5 0x00007f788d8e397c in ExecutorPool::_nextTask (this=0x5eb1840, t=..., tick=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:214
#6 0x00007f788d8e3a4e in ExecutorPool::nextTask (this=0x5eb1840, t=..., tick=10 '\n') at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:229
#7 0x00007f788d8f597b in ExecutorThread::run (this=0x5eba540) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:78
#8 0x00007f788d8f6296 in launch_executor_thread (arg=0x14370c4) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#9 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cecb0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#10 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f7886d18700 (LWP 31576)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x14370c0, mutex=0x1437088, ms=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f788d91d7b5 in TaskQueue::_doSleep (this=0x1437080, t=...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f788d92056f in TaskQueue::_fetchNextTask (this=0x1437080, t=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:98
#4 0x00007f788d92090e in TaskQueue::fetchNextTask (this=0x1437080, thread=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:142
#5 0x00007f788d8e397c in ExecutorPool::_nextTask (this=0x5eb1840, t=..., tick=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:214
#6 0x00007f788d8e3a4e in ExecutorPool::nextTask (this=0x5eb1840, t=..., tick=187 '\273') at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:229
#7 0x00007f788d8f597b in ExecutorThread::run (this=0x5eba620) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:78
#8 0x00007f788d8f6296 in launch_executor_thread (arg=0x14370c4) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#9 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cecc0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#10 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f7886317700 (LWP 31577)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x1437380, mutex=0x1437348, ms=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f788d91d7b5 in TaskQueue::_doSleep (this=0x1437340, t=...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f788d92056f in TaskQueue::_fetchNextTask (this=0x1437340, t=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:98
#4 0x00007f788d92090e in TaskQueue::fetchNextTask (this=0x1437340, thread=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:142
---Type <return> to continue, or q <return> to quit---
#5 0x00007f788d8e397c in ExecutorPool::_nextTask (this=0x5eb1840, t=..., tick=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:214
#6 0x00007f788d8e3a4e in ExecutorPool::nextTask (this=0x5eb1840, t=..., tick=191 '\277') at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:229
#7 0x00007f788d8f597b in ExecutorThread::run (this=0x5eba700) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:78
#8 0x00007f788d8f6296 in launch_executor_thread (arg=0x1437384) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#9 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cecd0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#10 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f7885916700 (LWP 31578)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x000000000041a236 in notify_io_complete (cookie=0x5c77500, status=ENGINE_SUCCESS) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:459
#5 0x00007f788d8ba536 in EventuallyPersistentEngine::notifyIOComplete(const void *, ._101) (this=0x5d2c000, cookie=0x5c77500, status=ENGINE_SUCCESS)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.h:478
#6 0x00007f788d912c26 in UprConnMap::manageConnections (this=0x5d40000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tapconnmap.cc:1107
#7 0x00007f788d91887f in ConnManager::run (this=0x1433310) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tapconnmap.cc:151
#8 0x00007f788d8f5b3a in ExecutorThread::run (this=0x5eba7e0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:110
#9 0x00007f788d8f6296 in launch_executor_thread (arg=0x5cb2aa8) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#10 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cece0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#11 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f78983427e0 (LWP 27228)):
#0 0x00007f78968df0ad in pthread_join () from /lib64/libpthread.so.0
#1 0x0000000000419f74 in threads_shutdown () at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:659
#2 0x000000000040f5f5 in main (argc=<value optimized out>, argv=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:8762
Comment by Aleksey Kondratenko [ 24/Aug/14 ]
memcached backtraces look dead-lock-ful
Comment by Mike Wiederhold [ 24/Aug/14 ]
It looks like all of the executor threads are running background fetches which are blocked from completing due to bucket creation. The bucket creation is waiting on warmup to complete, but it is not able to since there is no available executor thread to run it.
Comment by Sundar Sridharan [ 25/Aug/14 ]
Thanks Mike, you are right about the lack of reader threads. There is a quick fix for this issue, but instead of merging that, I wish to understand why one bucket's bgfetch completion notification should get blocked on a new bucket creation in memcached. Will try to update once I understand this better. thanks
Comment by Andrei Baranouski [ 25/Aug/14 ]
possible root of the problem lies in the http://www.couchbase.com/issues/browse/MB-12056
on this cluster I've played with the issue
Comment by Sundar Sridharan [ 25/Aug/14 ]
Andrei, I looked at MB-12056, from the first looks at least it appears like they are separate issues.
Comment by Sundar Sridharan [ 25/Aug/14 ]
Discussed this with Trond, the root cause is that the front-end thread is getting blocked by a ep-engine background thread due to wait for warmup in initialize() phase of bucket creation.
The fix is to change this behavior to make bucket creation non-blocking to avoid the deadlock seen here.
thanks
Comment by Sundar Sridharan [ 26/Aug/14 ]
fix uploaded for review at http://review.couchbase.org/#/c/40892 thanks
Comment by Wayne Siu [ 26/Aug/14 ]
Reviewed with PM/Cihan, this ticket is approved for rc2.
Comment by Sundar Sridharan [ 26/Aug/14 ]
fix has been merged commit id 46df358fadbd1f2b57996ad5546702b0e66731ad thanks




[MB-12074] {Windows}:: Rebalance-in hangs Created: 26/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0.1, 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Parag Agarwal Assignee: Sriram Ganesan
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1191, windows 2008, 10.1.2.107, 10.1.2.108

Triage: Untriaged
Operating System: Windows 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/bugdb/jira/MB-12074/windows_1191_logs.tar.gz
Is this a Regression?: Yes

 Description   
1191

1. Create 1 node cluster
2. Create default bucket and add 60K items
3. Rebalance-in 1 node

Rebalance hangs


 Comments   
Comment by Parag Agarwal [ 26/Aug/14 ]
cluster is live:: http://10.1.2.107:8091
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
Let me first note that my job could be easier if you stop adding logs of unrelated nodes. At least 2 nodes in this collection are not part of cluster and cannot be relevant for this bug.
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
Rebalance is waiting for seqno persistence which is apparently stuck.
Comment by Sriram Ganesan [ 26/Aug/14 ]
From the logs, it seems to be the same persistence problem as observed MB-11948 and MB-11955. Windows fixes are going into the master branch of ep-engine. So, please test/verify windows in 3.0.1 builds when they are available.
Comment by Parag Agarwal [ 28/Aug/14 ]
Works now, thanks for fixing it




[MB-11948] [Windows]: Simple-test broken - Rebalance exited with reason {unexpected_exit..{dcp_wait_for_data_move_failed,"default",254,..wrong_rebalancer_pid}}} Created: 13/Aug/14  Updated: 29/Aug/14  Resolved: 29/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Meenakshi Goel Assignee: Meenakshi Goel
Resolution: Fixed Votes: 0
Labels: windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1143-rel

Attachments: Zip Archive ep_persistence_failures.zip    
Issue Links:
Duplicate
is duplicated by MB-11981 [3.0.0-1166-Windows] items are stucke... Resolved
Triage: Triaged
Operating System: Windows 64-bit
Is this a Regression?: Yes

 Description   
Jenkins Ref Link:
http://qa.hq.northscale.net/job/win_2008_x64--01_00--qe-sanity-P0/68/console
http://qa.hq.northscale.net/job/win_2008_x64--01_00--qe-sanity-P0/66/consoleFull

Test to Reproduce:
./testrunner -i <yourfile>.ini -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,doc_ops=create;update;delete

Logs:
[ns_server:error,2014-08-13T7:46:46.007,ns_1@10.3.3.213:janitor_agent-default<0.18311.0>:janitor_agent:handle_call:639]Rebalance call failed due to the wrong rebalancer pid <0.18169.0>. Should be undefined.
[ns_server:error,2014-08-13T7:46:46.007,ns_1@10.3.3.213:<0.18299.0>:ns_single_vbucket_mover:spawn_and_wait:129]Got unexpected exit signal {'EXIT',<0.18312.0>,
                               {dcp_wait_for_data_move_failed,"default",254,
                                   'ns_1@10.3.3.213',
                                   ['ns_1@10.1.2.66'],
                                   wrong_rebalancer_pid}}
[ns_server:error,2014-08-13T7:46:46.007,ns_1@10.3.3.213:<0.18299.0>:misc:sync_shutdown_many_i_am_trapping_exits:1430]Shutdown of the following failed: [{<0.18312.0>,
                                    {dcp_wait_for_data_move_failed,"default",
                                     254,'ns_1@10.3.3.213',
                                     ['ns_1@10.1.2.66'],
                                     wrong_rebalancer_pid}}]
[ns_server:error,2014-08-13T7:46:46.007,ns_1@10.3.3.213:<0.18245.0>:ns_single_vbucket_mover:spawn_and_wait:129]Got unexpected exit signal {'EXIT',<0.18289.0>,
                            {bulk_set_vbucket_state_failed,
                             [{'ns_1@10.3.3.213',
                               {'EXIT',
                                {{{{case_clause,
                                    {error,
                                     {{{badmatch,
                                        {error,
                                         {{badmatch,{error,enobufs}},
                                          [{mc_replication,connect,1,
                                            [{file,"src/mc_replication.erl"},
                                             {line,30}]},
                                           {mc_replication,connect,1,
                                            [{file,"src/mc_replication.erl"},
                                             {line,49}]},
                                           {dcp_proxy,connect,4,
                                            [{file,"src/dcp_proxy.erl"},
                                             {line,179}]},
                                           {dcp_proxy,maybe_connect,1,
                                            [{file,"src/dcp_proxy.erl"},
                                             {line,166}]},
                                           {dcp_consumer_conn,init,2,
                                            [{file,
                                              "src/dcp_consumer_conn.erl"},
                                             {line,55}]},
                                           {dcp_proxy,init,1,
                                            [{file,"src/dcp_proxy.erl"},
                                             {line,48}]},
                                           {gen_server,init_it,6,
                                            [{file,"gen_server.erl"},
                                             {line,304}]},
                                           {proc_lib,init_p_do_apply,3,
                                            [{file,"proc_lib.erl"},
                                             {line,239}]}]}}},
                                       [{dcp_replicator,init,1,
                                         [{file,"src/dcp_replicator.erl"},
                                          {line,47}]},
                                        {gen_server,init_it,6,
                                         [{file,"gen_server.erl"},{line,304}]},
                                        {proc_lib,init_p_do_apply,3,
                                         [{file,"proc_lib.erl"},{line,239}]}]},
                                      {child,undefined,'ns_1@10.1.2.66',
                                       {dcp_replicator,start_link,
                                        ['ns_1@10.1.2.66',"default"]},
                                       temporary,60000,worker,
                                       [dcp_replicator]}}}},
                                   [{dcp_sup,start_replicator,2,
                                     [{file,"src/dcp_sup.erl"},{line,78}]},
                                    {dcp_sup,
                                     '-set_desired_replications/2-lc$^2/1-2-',
                                     2,
                                     [{file,"src/dcp_sup.erl"},{line,55}]},
                                    {dcp_sup,set_desired_replications,2,
                                     [{file,"src/dcp_sup.erl"},{line,55}]},
                                    {replication_manager,handle_call,3,
                                     [{file,"src/replication_manager.erl"},
                                      {line,130}]},
                                    {gen_server,handle_msg,5,
                                     [{file,"gen_server.erl"},{line,585}]},
                                    {proc_lib,init_p_do_apply,3,
                                     [{file,"proc_lib.erl"},{line,239}]}]},
                                  {gen_server,call,
                                   ['replication_manager-default',
                                    {change_vbucket_replication,255,
                                     'ns_1@10.1.2.66'},
                                    infinity]}},
                                 {gen_server,call,
                                  [{'janitor_agent-default','ns_1@10.3.3.213'},
                                   {if_rebalance,<0.18169.0>,
                                    {update_vbucket_state,255,replica,
                                     undefined,'ns_1@10.1.2.66'}},
                                   infinity]}}}}]}}

Uploading Logs.

Live Cluster
1:10.3.3.213
2:10.3.2.21
3:10.3.2.23
4:10.1.2.66

 Comments   
Comment by Meenakshi Goel [ 13/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-11948/f806d72b/10.3.3.213-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-11948/11dd43ca/10.3.2.21-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-11948/75af6ef6/10.1.2.66-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-11948/9dc45b16/10.3.2.23-diag.zip
Comment by Aleksey Kondratenko [ 13/Aug/14 ]
You digged out right error message.

ENOBUFS is mapped from WSAENOBUFS and googling for it found me http://support.microsoft.com/kb/196271 which reminds me about that installer thing that we did this registry setting. It looks like you still need to do it.
Comment by Sriram Melkote [ 13/Aug/14 ]
I think we support only Windows 2008 and they fixed that issue,

it has 16k ports by default.
http://support.microsoft.com/kb/929851

Alk, any idea why we initiate so many outgoing connections?
Comment by Aleksey Kondratenko [ 13/Aug/14 ]
We shouldn't. And I don't know what "many" is.
Comment by Aleksey Kondratenko [ 14/Aug/14 ]
Rebalance is stuck waiting for seqno persistence. It's something in ep-engine.
Comment by Aleksey Kondratenko [ 14/Aug/14 ]
And memcached logs are full of this:

Thu Aug 14 10:51:05.630859 Pacific Daylight Time 3: (default) Warning: failed to open database file for vBucket = 1013 rev = 1
Thu Aug 14 10:51:05.633789 Pacific Daylight Time 3: (default) Warning: couchstore_open_db failed, name=c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default/246.couch.1 option=2 rev=1 error=no such file [errno = 0: 'The operation completed successfully.

So indeed persistence is not working.
Comment by Chiyoung Seo [ 14/Aug/14 ]
Sriram,

I think this issue was caused by the file path name issue on windows that we discussed the other day.
Comment by Sriram Ganesan [ 15/Aug/14 ]
From bug MB-11934, Meenakshi had run the simple test on build 3.0.0-1130-rel (http://qa.hq.northscale.net/job/win_2008_x64--01_00--qe-sanity-P0/65/consoleFull) and it looks like all the tests passed except for warmup. So, it looks like persistence was okay at that point. Did this manifest in any build earlier than 3.0.0-1143-rel? It might help pinpoint the check-ins that caused the regression.
Comment by Meenakshi Goel [ 18/Aug/14 ]
I had the last successful run with 3.0.0-1137-rel and after that picked 3.0.0-1143-rel in which observed this issue.
Also sanity tests seems to worked fine till 3.0.0-1139-rel
http://qa.sc.couchbase.com/job/CouchbaseServer-SanityTest-4Node-Windows_2012_x64/213/consoleFull
Comment by Sriram Ganesan [ 18/Aug/14 ]
Also observing a lot of couch notifier errors

memcached<0.81.0>: Wed Aug 13 07:45:39.978617 Pacific Daylight Time 3: (default) Resetting connection to mccouch, lastReceivedCommand = select_bucket lastSentCommand = select_bucket currentCommand =unknown
memcached<0.81.0>: Wed Aug 13 07:45:39.979593 Pacific Daylight Time 3: (default) Trying to connect to mccouch: "127.0.0.1:11213"
memcached<0.81.0>: Wed Aug 13 07:45:39.979593 Pacific Daylight Time 3: (default) Connected to mccouch: "127.0.0.1:11213"
memcached<0.81.0>: Wed Aug 13 07:45:39.980570 Pacific Daylight Time 3: (default) Failed to read from mccouch for select_bucket: "The operation completed successfully.

Before all those, there is an error from moxi

[ns_server:info,2014-08-13T7:45:38.812,babysitter_of_ns_1@127.0.0.1:<0.79.0>:ns_port_server:log:169]moxi<0.79.0>: 2014-08-13 07:45:40: (C:\Jenkins\workspace\cs_300_win6408\couchbase\moxi\src\agent_config.c.721) ERROR: bad JSON configuration from http://127.0.0.1:8091/pools/default/saslBucketsStreaming: No vBuckets available; service maybe still initializing

Comment by Meenakshi Goel [ 21/Aug/14 ]
Please let me know if cluster is no longer required. Thanks.
Comment by Sriram Ganesan [ 21/Aug/14 ]
I was able to simulate the same problem in a much simpler warmup test (using a toy windows builder with some custom logs). We see the read failures in couch notifier when invoking recv() from the mccouch connection. The connection keeps getting reset and eventually we can't even connect to mccouch.

Thu Aug 21 16:26:06.005468 Pacific Daylight Time 3: (default) Failed to connect to: "127.0.0.1:11213"
Thu Aug 21 16:26:06.005468 Pacific Daylight Time 3: (default) Failed to connect to: "127.0.0.1:11213"
Thu Aug 21 16:26:06.012468 Pacific Daylight Time 3: (default) Failed to connect to: "127.0.0.1:11213"
Thu Aug 21 16:26:06.012468 Pacific Daylight Time 3: (default) Failed to connect to: "127.0.0.1:11213"
Thu Aug 21 16:26:06.019468 Pacific Daylight Time 3: (default) Failed to connect to: "127.0.0.1:11213"

We have only started seeing this problem from build 3.0.0-1142 onwards. On the couch notifier side, recv() returns -1 but WSAGetLastError() returns 0 which is not one of the errors mentioned at http://msdn.microsoft.com/en-us/library/windows/desktop/ms740121(v=vs.85).aspx. In order to triage this better, it might be good to know what requests mccouch is receiving and what responses it is providing in order to find the explanation for the bizarre behavior on the couch notifier side. Test can be run using the following command on a couple of windows boxes

./testrunner -i ./dev.ini -t "memcapable.WarmUpMemcachedTest.do_warmup_10k"
Comment by Sriram Ganesan [ 21/Aug/14 ]
Uploading logs from one of the nodes in the warmup test that was run. Let me know if you need more details.
Comment by Sriram Ganesan [ 21/Aug/14 ]
Assigning to ns_server to find out more details on the activity on the mccouch side.
Comment by Aleksey Kondratenko [ 21/Aug/14 ]
From our side we see this:

[ns_server:debug,2014-08-21T16:26:06.006,ns_1@127.0.0.1:<0.418.0>:mc_tcp_listener:accept_loop:31]Got new connection
[ns_server:debug,2014-08-21T16:26:06.006,ns_1@127.0.0.1:<0.22291.1>:mc_connection:handle_select_bucket:131]Got select bucket default
[ns_server:debug,2014-08-21T16:26:06.006,ns_1@127.0.0.1:<0.418.0>:mc_tcp_listener:accept_loop:33]Passed connection to mc_conn_sup: <0.22291.1>
[ns_server:debug,2014-08-21T16:26:06.006,ns_1@127.0.0.1:<0.22291.1>:mc_connection:handle_select_bucket:133]Sent reply on select bucket
[ns_server:info,2014-08-21T16:26:06.006,ns_1@127.0.0.1:<0.22291.1>:mc_connection:run_loop:162]mccouch connection was normally closed

I.e. our side sees normal connection closure.

Eventual inability of ep-engine side to connect is likely caused by exhausting ports space by TIME_WAIT sockets.
Comment by Sriram Ganesan [ 25/Aug/14 ]
http://review.couchbase.org/#/c/40865/

Merged in a fix for handling the persistence failures in ep-engine. Please verify the fix and update the ticket if there are any more issues.
Comment by Ketaki Gangal [ 26/Aug/14 ]
Hi Sriram,

Can you provide the exact build number for this - we can test on that build.
Comment by Sriram Ganesan [ 26/Aug/14 ]
Hello Ketaki

The 3.0.0 builds seem to be pointing to the 3.0 branch of ep-engine. I believe you will need to 3.0.1 builds to verify the above change as the 3.0.1 release manifest https://github.com/membase/manifest/blob/master/rel-3.0.1.xml points to the ep-engine master branch where this change was checked in but I don't see any 3.0.1 windows builds at http://latestbuilds.hq.couchbase.com. You might want to ping the build folks about generating those builds.

Thanks
Sriram
Comment by Meenakshi Goel [ 27/Aug/14 ]
Verified with 3.0.1-1206-rel
http://qa.hq.northscale.net/job/win_2008_x64--01_00--qe-sanity-P0/74/console




[MB-10867] Prominently display current SDK version Created: 16/Apr/14  Updated: 28/Aug/14  Resolved: 28/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 2.5.1
Fix Version/s: feature-backlog
Security Level: Public

Type: Improvement Priority: Major
Reporter: Brian Shumate Assignee: Matt Ingenthron
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
It would be extremely handy for users if the current client SDK version could be prominently displayed somewhere on the http://www.couchbase.com/communities/* pages.

 Comments   
Comment by Amy Kurtzman [ 16/Apr/14 ]
The current version of each SDK is displayed on the left navigation bar on each page. Clicking those links downloads the SDK, which is not necessarily what people expect.
Comment by Amy Kurtzman [ 16/Apr/14 ]
Reassigning to Matt ... Maybe this is an issue for the website redesign?
Comment by Matt Ingenthron [ 28/Aug/14 ]
The website is being reworked and I'll keep this request in mind.




[MB-12026] [BUG BASH] Removing production views during a reindex renders console unusable Created: 19/Aug/14  Updated: 28/Aug/14  Resolved: 28/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0-Beta
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: tnguyen Assignee: Aleksey Kondratenko
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: GZip Archive cblogs2.tar.gz     GZip Archive cblogs3.tar.gz     GZip Archive cblogs.tar.gz     PNG File Screen Shot 2014-08-19 at 4.00.58 PM.png    
Triage: Untriaged
Operating System: Centos 64-bit
Is this a Regression?: Yes

 Description   
We've been able to do this in the past with v2.2. During a reindex of Production views, if we delete a view, it causes both the Development Views and Production Views windows to remain at this "Loading" screen. Refreshing, logging out and back in, trying a different node in the cluster, etc. does nothing and you're not able to see the list of designdocs/views.

The attached image shows "Lost connection..." because the only way to get it back to a working state is to restart the server. This is after waiting several minutes for the page to return.

 Comments   
Comment by Aleksey Kondratenko [ 19/Aug/14 ]
Please upload logs of all nodes to help diagnose this.
Comment by Aleksey Kondratenko [ 19/Aug/14 ]
Best way to gather diagnostics is to use new cluster-wide cbcollectinfo gathering facility. See Logs section and tab "Collect Information".
Comment by Aleksey Kondratenko [ 19/Aug/14 ]
I only see http errors in logs as part of 2 server restarts. I see no obvious signs of issues on node 1 caused by removal of indexes.

And I was unable to reproduce this too.

I'll need you to either:

* post more verbose description of how you managed to hit this

* or reproduce this again and post cbcollectinfos.
Comment by tnguyen [ 20/Aug/14 ]
The cluster we were using got into a corrupted state (unrelated to this issue) so we wiped it out and started over. The views we were using were huge so I'm not sure if this could possibly be load related in some way. In any case, we do not have the same setup as we did when we hit this issue. Once we get there, hopefully in the next day or two, I will test again and update
Comment by Don Pinto [ 28/Aug/14 ]
Closing this one as no logs reported.

Thanks,




[MB-12062] Documentation link for 3.0 build in admin UI goes to 2.5 docs Created: 25/Aug/14  Updated: 28/Aug/14  Resolved: 28/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0-Beta
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Don Pinto Assignee: Aleksey Kondratenko
Resolution: Fixed Votes: 0
Labels: Rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Please see description below.

 Comments   
Comment by Don Pinto [ 28/Aug/14 ]
Don needs to send email to marketing to carve out a link that we can control. Maybe a docs.couchbase.com subpath?
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
we need to clearly isolate the redirect to a URL we can control late in the release.
Lets point to http://docs.couchbase.com/redirect/documentation-3.0.0 in the console and ask for the web link to be a ridect to the top doc page for now. we'll add a checklist item to release to ensure we correct the redirect point before GA to the correct page.
thanks
-cihan
Comment by Matt Ingenthron [ 28/Aug/14 ]
nitpick, but using /redirect/ seems icky to me. it's definitely a nitpick though.

I might recommend http://docs.couchbase.com/release/couchbase-server/3.0.0

we'd reserve the /release/ namespace for these "persistent URLs" and then have the product name under it. It's nice and SEO happy and is a "hackable URL" meaning if a search dumps you on 3.0.0 but you know you want 3.0.1, it's easy to work out how to get there.
Comment by Don Pinto [ 28/Aug/14 ]
Today's URLs for docs have pre-built in them - see
http://docs.couchbase.com/prebuilt/couchbase-manual-3.0/beta-intro.html

I had asked for this to be changed before and it never happened. Having /release redirect to pre-built will be weird. Hence would favor redirect in this format -
http://docs.couchbase.com/redirect/couchbase-server/3.0.0
Comment by Don Pinto [ 28/Aug/14 ]
Hi Alk,

We would need your help to include this in the RC2 build for tomorrow. This is the documentation link in the UI which currently points to 2.5 docs.

Request to change that to point to - http://docs.couchbase.com/redirect/couchbase-server/3.0.0 (This is a redirect URL that we will then re-direct to the final documentation page).

Thank you for your help.

-Don
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
let make sure Amy and Ruth can add this URL on their as well otherwise Alk may need to make more changes. Don could you coordinate with Amy on this one.
thanks
-cihan
Comment by Don Pinto [ 28/Aug/14 ]
Cihan,

The redirects are happening at the domain level through our provider. Ray is controlling that but I'll check with Amy to make sure there are no URL collisions. Attached sub-task CBIT-1576 and Ray is on that.

Thanks,
Comment by Amy Kurtzman [ 28/Aug/14 ]
Hi Alk, Please get with me before you make the change. The URL Don requested above is incorrect. Thanks, Amy
Comment by Don Pinto [ 28/Aug/14 ]
Amy,

As discussed in the 3.0 meeting today (with Ruth), thats the re-directing URL we decided to go with. We do not want to go directly to docs.couchbase.com but to 3.0 specific content and I believe the final 3.0 introduction page URL in docs might change for GA. Thus we decided to make the product go to the redirect URL which could go to the final 3.0 introduction GA doc page when it is finalized.

Please sync up with Cihan if there is any confusion.

Thanks,

Comment by Amy Kurtzman [ 28/Aug/14 ]
I never said to send it to docs.couchbase.com and I understand that it is desired that it does to the 3.0 manual.
Comment by Don Pinto [ 28/Aug/14 ]
OK. Can you please provide the target 3.0 introduction page URL that the /redirect URL should point to?

Thanks,
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
http://review.couchbase.org/41059
Comment by Don Pinto [ 28/Aug/14 ]
Discussed with respective folks here - Alk, Ruth, Amy, and Ray.

We decided to go with a domain level link - http://www.couchbase.com/redirect/couchbase-server-docs/3.0.0 which will be in the UI.

Ray will setup a redirect through Contegix from http://www.couchbase.com/redirect/couchbase-server-docs/3.0.0 -> http://docs.couchbase.com/prebuilt/couchbase-manual-3.0/beta-intro.html

Later on, when we are going to GA 3.0, we will refresh the redirect target to point to the latest link docs team provides us with.

Thank you,
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
http://review.couchbase.org/41066




[MB-11389] Upgrade OpenSSL to 1.0.1h [windows] Created: 11/Jun/14  Updated: 28/Aug/14  Resolved: 28/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Trond Norbye Assignee: Chris Hillery
Resolution: Fixed Votes: 0
Labels: windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Comments   
Comment by Trond Norbye [ 11/Jun/14 ]
Depot is updated
Comment by Chris Hillery [ 28/Aug/14 ]
Builders were updated some weeks ago.




[MB-12075] Couchbase Server constantly shutting down and restarting after upgrade from 2.1.1-766 Created: 26/Aug/14  Updated: 28/Aug/14  Resolved: 26/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Aruna Piravi Assignee: Sangharsh Agarwal
Resolution: Fixed Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Ubuntu 64 bit, 3.0.0-1174-rel

Issue Links:
Gantt: start-finish
is triggering MB-12065 [Offline Upgrade 2.1.1-766 -> 3.0.0-1... Resolved
is triggering MB-12066 [Offline Upgrade 2.1.1.766 -> 3.0.0-1... Resolved
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Pls look at http://10.3.3.240:8091/index.html#sec=servers, node 10.3.3.239 in particular.

The node is unstable, Couchbase server is restarting every 4 secs as you can see from the GUI log.

--> This happened after the cluster was upgraded from 2.1.1.766 to 3.0.0-1174.
--> After offline upgrade, source cluster was forced to use encrypted xdcr.
--> No replication seems to have happened after this change and node .239 entered a bad state.
--> Seen only on Ubuntu
--> Is probably the reason for MB-12066(yes, there is data loss - 1000 items less on each bucket on C2).

Cbcollect
-------------

[C1]
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12066/d30b70e7/10.3.3.239-8252014-2346-couch.tar.gz
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12066/4b079374/10.3.3.240-8252014-2340-diag.zip

[C2]
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12066/e4f38323/10.3.3.218-8252014-2343-diag.zip
10.3.3.225 : https://s3.amazonaws.com/bugdb/jira/MB-12066/6f010183/10.3.3.225-8252014-2345-diag.zip


 Comments   
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
Indeed happens because of ns_server bug. And also indeed causes xdcr to malfunction.
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
I've fixed the thing for live cluster already. Will post fix in few hours.
Comment by Aruna Piravi [ 26/Aug/14 ]
Thanks, I can now see all keys replicated. Will close 12066 as duplicate of this bug.
Comment by Cihan Biyikoglu [ 26/Aug/14 ]
approved for RC2 if we can get the fix by wed EOD
thanks
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
manifest bumped here: http://review.couchbase.org/40954

actual fixes are:

* http://review.couchbase.org/40953
* http://review.couchbase.org/40949

Comment by Aruna Piravi [ 27/Aug/14 ]
Alk, I see 140 lines of code changed/added. Do you foresee any risks of regression at this point?
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
No. The second patch is large because I had to refactor remote_cluster_info facility a bit. I tested it thoroughly, plus it's quite deterministic piece of code. If we have any regressions they will be caught as part of usual xdcr testing.




[MB-12081] Remove counting mutations introduced for MB-11589 Created: 27/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Sriram Melkote Assignee: Volker Mische
Resolution: Fixed Votes: 0
Labels: RC2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to MB-11589 Sliding endseqno during initial index... Open
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
We are currently counting number of mutations requested vs received. This was diagnostic code used to get closer to resolving MB-11589. However, that bug has been deferred to 3.0.1 - the diagnostic code must be removed, to avoid performance and logging overhead in release build.

http://review.couchbase.org/40790

 Comments   
Comment by Wayne Siu [ 27/Aug/14 ]
Reviewed with PM/Cihan. Approved for RC2.
Comment by Cihan Biyikoglu [ 27/Aug/14 ]
approved for RC2.




[MB-12080] unable to build cbq-engine Created: 27/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Task Priority: Test Blocker
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
output is:
[root@grape-001 query]# ./build.sh
cd parser/n1ql
nex...
goyacc...
go build...
cd server/main
go build -o cbq-engine...
# github.com/couchbaselabs/query/accounting/logger_retriever
../../accounting/logger_retriever/logger_retriever.go:73: undefined: logger.LogLevel
cd shell
go build -o cbq...
cd tutorial
go build

shell is built fine, but cbq-engine is not built




[MB-12048] View engine 2.5 to 3.0 index file upgrade Created: 22/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sarath Lakshman Assignee: Ketaki Gangal
Resolution: Fixed Votes: 0
Labels: RC2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Gantt: start-finish
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
View engine 2.5 index files are not compatible with 3.0 index files. Hence, it requires index rebuild for 3.0.
We need a method that renames index files to new compatible filenames (with signature) and append new header.

 Comments   
Comment by Volker Mische [ 22/Aug/14 ]
I created a hacky script that still involves some manual steps, though i think it works in general.

Though I'm hitting a bigger issue with DCP. It expectes that you send the correct partition/vBucket version with your request whenever you don't start indexing from scratch. The problem is that this information is persisted on disk, hence in the index header. When we offline upgrade from 2.x to 3.0 we don't know which partition that server might have, hence we can't save the correct one.

Currently the only sane way I see is changing the DCP semantics and making it possible to resume from a certain seq number with sending {0, 0} as partition version (currently you need to send the correct partition version in case you want to resume).
Comment by Sriram Melkote [ 22/Aug/14 ]
Folks - we need to do this as an automatic feature (i.e., fully automated without needing any manual steps) or not at all. Let's talk with EP engine folks to spec this.
Comment by Sarath Lakshman [ 22/Aug/14 ]
I am guessing it can be done as part of installation postscript which checks the current version and performs upgrade of existing files.
Eg. RPM and Deb has a way to specify post scripts.
Comment by Volker Mische [ 22/Aug/14 ]
Sarath, yes, that would be a way.
Comment by Volker Mische [ 22/Aug/14 ]
Siri, I misunderstood you comment as you probably did mine. The manual steps are only needed atm to verify my idea works. The final result will be a script that can be run without any manual steps.

My misunderstanding was that I thought you talk about "online upgrade", but you didn't really say that.
Comment by Sriram Melkote [ 22/Aug/14 ]
Ketaki, can we add a test to detect this situation? The test must fail until we fix this issue.
Comment by Ketaki Gangal [ 22/Aug/14 ]
Hi Siri,

We run automation tests which do offline upgrades from 2.X to 3.X, what these tests dont check is whether index is rebuilt /not.
https://github.com/couchbase/testrunner/blob/master/conf/py-newupgrade.conf#L39

I ll update the tests to add a check for index-rebuild verification.

Sarath: Can you provide details on how to check if indexes are rebuilt/not.


Comment by Sarath Lakshman [ 23/Aug/14 ]
I can think of a very easy way where you can have a timeout for stale=false after Couchbase is up soon after warmup. Run a stale=false and it should run immediately. Since index rebuild is going on, it would take more time.
Comment by Sriram Melkote [ 25/Aug/14 ]
Product Managers, please note:

This is UPGRADE step. We'll need to discuss how we'll handle upgrade logic in the product before we decide on closure of this issue.

+Anil, Ilam, Cihan
Comment by Cihan Biyikoglu [ 25/Aug/14 ]
lets pick this up on the daily synup tomorrow.
thanks
-cihan
Comment by Volker Mische [ 26/Aug/14 ]
I think I found a nice solution. Now it's an online upgrade:

http://review.couchbase.org/40914
http://review.couchbase.org/40915
http://review.couchbase.org/40916
Comment by Sriram Melkote [ 27/Aug/14 ]
This has been discussed earlier this week and is a 3.0 approved exception.




[MB-11867] A method to find the position of a key in a view. Created: 01/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.5.1, 3.0.1
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Patrick Varley Assignee: Nimish Gupta
Resolution: Done Votes: 0
Labels: community
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
This came up in the #couchbase IRC channel today:

If you have a game where players are in different zones (US, EU and Asia), there is no simple way to find the player's position base on score in a given zone.

 Comments   
Comment by Patrick Varley [ 04/Aug/14 ]
Using a reduce and count this is possible:

http://blog.couchbase.com/using-map-and-reduce-view-ranking




[MB-11808] GeoSpatial in 3.0 Created: 24/Jul/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: ns_server, UI, view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sriram Melkote Assignee: Volker Mische
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
We must hide GeoSpatial related UI elements in 3.0 release, as we have not completed the task of moving GeoSpatial features over to UPR.

We should use the simplest way to hide elements (like "display:none" attribute) because we fully expect to resurface this in 3.0.1


 Comments   
Comment by Sriram Melkote [ 24/Jul/14 ]
In the 3.0 release meeting, it was fairly clear that we won't be able to add Geo support for 3.0 due to the release being in Beta phase now and heading to code freeze soon. So, we should plan for it in 3.0.1 - updating description to reflect this.
Comment by Volker Mische [ 27/Aug/14 ]
No longer needed, spatial views are in 3.0 now :)




[MB-9720] kickoff indexing on view creation Created: 11/Dec/13  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.2.0
Fix Version/s: bug-backlog
Security Level: Public

Type: Bug Priority: Major
Reporter: Jon Adams Assignee: Unassigned
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We have data and then we programmatically create views via CouchDB directly, and then our API hits the views with stale=true which produces zero results (which makes sense because we didn't index data yet)

It seems interesting that in this state (data exists, views just created, request made to view that has not yet indexed and returns zero results) that CB does not detect the view has never indexed and should at least kickoff index once. Note: I'm not saying the client should trigger the index (e.g. update_after) as it has specified stale is ok. But, it seems like the server should detect it has never indexed data at all for a view a client has attempted to utilize and assume it should probably index something. Thoughts?

 Comments   
Comment by Perry Krug [ 12/Dec/13 ]
I'm not sure I would agree in the general case Jon. Certainly there is merit to doing this, but the downside would be forcing an index to take place beyond the control of the user. Especially on very large datasets, it may be desirable to actually delay the indexing itself to prevent undo load on the system. It doesn't sound too onerous for the external user/client/application to upload a view and then trigger an index update (with stale=update_after likely) as part of the workflow which gives everyone the control to decide when or when not to do it.

Would be interested in hearing more about your application and what you've been doing with CouchDB...feel free to email me on the side at perry@couchbase.com
Comment by Volker Mische [ 27/Aug/14 ]
When you publish your view as a production view and you have more than 5000 documents (that's the default value) then index will be created by the automatic updater.




[MB-9784] too large emit value returns no response Created: 20/Dec/13  Updated: 27/Aug/14  Due: 20/Jun/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.5.0
Fix Version/s: bug-backlog
Security Level: Public

Type: Bug Priority: Minor
Reporter: Iryna Mironava Assignee: Unassigned
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 2.5.0-1011-rel
<manifest><remote name="couchbase" fetch="git://github.com/couchbase/"/><remote name="membase" fetch="git://github.com/membase/"/><remote name="apache" fetch="git://github.com/apache/"/><remote name="erlang" fetch="git://github.com/erlang/"/><default remote="couchbase" revision="master"/><project name="tlm" path="tlm" revision="db49bf5d4e601c5994f8bd7f61ca6cff6840af5d"><copyfile src="Makefile.top" dest="Makefile"/></project><project name="bucket_engine" path="bucket_engine" revision="0a3a9df0a55d759b5b78a3a7d001a97a4d35af1c"/><project name="cbsasl" path="cbsasl" revision="578523010d4efaa9fed1a32880c67bfb03c20728"/><project name="couchbase-cli" path="couchbase-cli" revision="b8c21c7462e3b45cc0c69259547613a4c45b6be3" remote="couchbase"/><project name="couchbase-examples" path="couchbase-examples" revision="cd9c8600589a1996c1ba6dbea9ac171b937d3379"/><project name="couchbase-python-client" path="couchbase-python-client" revision="f14c0f53b633b5313eca1ef64b0f241330cf02c4"/><project name="couchdb" path="couchdb" revision="01dda76eab9edb6b64490c524ccdaf8e5a8b655b"/><project name="couchdbx-app" path="couchdbx-app" revision="cc4fe0884faeebbb36a45fcdf6a072d736b0ca5d"/><project name="couchstore" path="couchstore" revision="30f8f0872ef28f95765a7cad4b2e45e32b95dff8"/><project name="ep-engine" path="ep-engine" revision="7c2254cd57a987d087c092897fa35bc2e4833039"/><project name="geocouch" path="geocouch" revision="000096996e57b2193ea8dde87e078e653a7d7b80"/><project name="healthchecker" path="healthchecker" revision="829e18598bfef537263c0cf014420d499a467a7d"/><project name="libconflate" path="libconflate" revision="c0d3e26a51f25a2b020713559cb344d43ce0b06c"/><project name="libmemcached" path="libmemcached" revision="ea579a523ca3af872c292b1e33d800e3649a8892" remote="membase"/><project name="libvbucket" path="libvbucket" revision="408057ec55da3862ab8d75b1ed25d2848afd640f"/><project name="memcached" path="memcached" revision="639cd3ee86d7a72f1a00d01c51fc49b5966f7f2d" remote="membase"/><project name="moxi" path="moxi" revision="2b5a228f58fcfd1a836d6ad9a8f279b4f0ebfe80"/><project name="ns_server" path="ns_server" revision="7f17d27b0b971710c93b3fe1ef553fec83ae1e17"/><project name="portsigar" path="portsigar" revision="2204847c85a3ccaecb2bb300306baf64824b2597"/><project name="sigar" path="sigar" revision="a402af5b6a30ea8e5e7220818208e2601cb6caba"/><project name="testrunner" path="testrunner" revision="67b9b374b552312b6addf3fd4a3c17891450eb7b"/><project name="otp" path="otp" revision="b6dc1a844eab061d0a7153d46e7e68296f15a504" remote="erlang"/><project name="icu4c" path="icu4c" revision="26359393672c378f41f2103a8699c4357c894be7" remote="couchbase"/><project name="snappy" path="snappy" revision="5681dde156e9d07adbeeab79666c9a9d7a10ec95" remote="couchbase"/><project name="v8" path="v8" revision="447decb75060a106131ab4de934bcc374648e7f2" remote="couchbase"/><project name="gperftools" path="gperftools" revision="674fcd94a8a0a3595f64e13762ba3a6529e09926" remote="couchbase"/><project name="pysqlite" path="pysqlite" revision="0ff6e32ea05037fddef1eb41a648f2a2141009ea" remote="couchbase"/><!--
  <project name="voltron" path="voltron" revision="20c2d314ad110bd4d301c44f12c22c7ae6365596" />
  --></manifest>

Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/bugdb/jira/MB-9784/447a45ae/10.3.121.110-12202013-53-diag.zip

 Description   
Development view with too large emitted value returns emty resonse for development subset queries, and empty for full set
If perform query from UI, it redirects to page with login, and need to login to application again

iryna@ubuntu:~/couchbase/testrunner$ curl -v 'http://10.3.121.110:8092/default/_design/dev_large&#39;
curl: /usr/local/lib/libcurl.so.4: no version information available (required by curl)
* About to connect() to 10.3.121.110 port 8092 (#0)
* Trying 10.3.121.110... connected
* Connected to 10.3.121.110 (10.3.121.110) port 8092 (#0)
> GET /default/_design/dev_large HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.21.4 OpenSSL/1.0.1 zlib/1.2.3.4
> Host: 10.3.121.110:8092
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Couchbase-Meta: {"id":"_design/dev_large","rev":"2-a9936869","type":"json"}
< Server: MochiWeb/1.0 (Any of you quaids got a smint?)
< Date: Fri, 20 Dec 2013 12:52:19 GMT
< Content-Type: application/json
< Content-Length: 196
< Cache-Control: must-revalidate
<
* Connection #0 to host 10.3.121.110 left intact
* Closing connection #0
{"views":{"large":{"map":"function (doc, meta) {\n var val_test = 'test';\n while (val_test.length < 700000000) {\n val_test = val_test.concat(val_test);\n }\n emit(meta.id, val_test);}"}}}


iryna@ubuntu:~/couchbase/testrunner$ curl 'http://10.3.121.110:8092/default/_design/dev_large/_view/large?stale=false&connection_timeout=60000&limit=10&skip=0&#39;
curl: /usr/local/lib/libcurl.so.4: no version information available (required by curl)
curl: (52) Empty reply from server
iryna@ubuntu:~/couchbase/testrunner$ curl -v 'http://10.3.121.110:8092/default/_design/dev_large/_view/large?full_set=true&connection_timeout=60000&limit=10&skip=0&#39;
curl: /usr/local/lib/libcurl.so.4: no version information available (required by curl)
* About to connect() to 10.3.121.110 port 8092 (#0)
* Trying 10.3.121.110... connected
* Connected to 10.3.121.110 (10.3.121.110) port 8092 (#0)
> GET /default/_design/dev_large/_view/large?full_set=true&connection_timeout=60000&limit=10&skip=0 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.21.4 OpenSSL/1.0.1 zlib/1.2.3.4
> Host: 10.3.121.110:8092
> Accept: */*
>
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Server: MochiWeb/1.0 (Any of you quaids got a smint?)
< Date: Fri, 20 Dec 2013 13:02:10 GMT
< Content-Type: text/plain;charset=utf-8
< Cache-Control: must-revalidate
<
{"total_rows":0,"rows":[
]
}
* Connection #0 to host 10.3.121.110 left intact
* Closing connection #0
iryna@ubuntu:~/couchbase/testrunner$


 Comments   
Comment by Filipe Manana [ 20/Dec/13 ]
Similar answer to the other ticket.

This message is only implemented for production views in 2.5 and below. Your query example is doing it for a dev view (and not full_set=true).
For 3.0 that's not the case anymore.

About the UI issue, I don't know, not my area.
Comment by Volker Mische [ 24/Jun/14 ]
The test uses such a big emit that it crashes v8 on my machine (the emitted value is > 1GB). With one zero less, I get the correct mapreduce error in the logs and the result is empty as expected. I think the issue can be closed.
Comment by Volker Mische [ 27/Aug/14 ]
As per previous comment. It works as expected unless you've insanely large emits that crash V8. Hence I close it as a won't fix.




[MB-9160] Add `include_ids` parameter to view query api with reduce support Created: 22/Sep/13  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.1.0
Fix Version/s: feature-backlog
Security Level: Public

Type: Improvement Priority: Major
Reporter: Jon Adams Assignee: Unassigned
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
It would be awesome if there was a simple `include_ids` parameter for use in the context of views w/ complex/custom map/reduce. ID is included when not reducing but it would be really powerful to be able to get a collection of document ids that a reduced result is made up of. Thoughts?

 Comments   
Comment by Filipe Manana [ 23/Sep/13 ]
Can you give a practical example where that would be useful?
thanks
Comment by Jon Adams [ 23/Sep/13 ]
Let's say you have documents for repositories and and a languages array with the language as the key and number of lines as the value.

{
"name": "couchycouch",
"age": 3
"languages": [
 "Erlang": 392,
 "JavaScript" :185,
]}

And you wrote a view with a compound index [age,language] and value is the number emitted. for simplicity we use one of the built in reduce functions.

Now, we could query the view and get the reduces results, which is great. I am able to group results by age and/or language. However, if one result is particularly interesting, and I'd like to also understand which docs were included that made up that result it would be nice to hit that same query with a new parameter to also include (an array?) the document ids for all docs that matched or made up that particular row. of the reduced result. For example, it's nice that I know I have over 9,000 lines of perl that 10 years old but which documents made up that result (without having to write a new view)?
Comment by Filipe Manana [ 23/Sep/13 ]
Sorry but that's not possible.

Keeping track of the document IDs that contributed to a reduction value would explode the data structures (B+trees) as it means the IDs would have to be stored together with every full or partial reduce value stored in a B+Tree node.

If want to learn more about the data structures, see http://guide.couchdb.org/draft/views.html, it applies to Couchbase too.
Comment by Jon Adams [ 23/Sep/13 ]
nooooooooooooo
Comment by Jon Adams [ 23/Sep/13 ]
;) Thanks for the link. I'll just write some views.
Comment by Volker Mische [ 27/Aug/14 ]
As Filipe mentions in a comment, it's not possible. The original but reported agreed and said he'll use normal views instead.




[MB-10138] Fix spatial views Created: 06/Feb/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Volker Mische Assignee: Volker Mische
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged

 Description   
Currently the spatial views are broken on 3.0. Fix them.

 Comments   
Comment by Sriram Melkote [ 01/Apr/14 ]
We can bring it back to 3.0 if we make good progress. With currently discussed timelines, it appears unlikely and so, we should plan this for the dot release immediately after 3.0
Comment by Volker Mische [ 27/Aug/14 ]
Spatial views work again on current 3.0.




[MB-8746] Set view based spatial views Created: 01/Aug/13  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Major
Reporter: Volker Mische Assignee: Volker Mische
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   
Make the spatial views work like the mapreduce set views (aka b-superstar).

 Comments   
Comment by Volker Mische [ 01/Aug/13 ]
I closed the old CBD based issue in favor of creating this new (clearer) one.
Comment by Sriram Melkote [ 15/Apr/14 ]
As 3.0 is feature complete, we will need to address this in the next minor release
Comment by Volker Mische [ 15/Apr/14 ]
We'll so how long 3.0 will be delayed. I got the OK from Ravi ages ago to consider it a bug in order to get it in. Though I'm not even sure if I can meet that deadline.
Comment by Volker Mische [ 27/Aug/14 ]
3.0 now uses set view based spatial views.




[MB-5733] Stale does not mean what it should Created: 28/Jun/12  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.0-developer-preview-4
Fix Version/s: bug-backlog
Security Level: Public

Type: Bug Priority: Major
Reporter: d3fault Assignee: Dipti Borkar
Resolution: Fixed Votes: 0
Labels: usability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: n/a

Triage: Untriaged

 Description   
Views are updated asynchronously. There is no atomic guarantee that after I insert an item, it will show up in affecting Views if I query them immediately after. Currently, it will only show up when the value hits the hard disk. Stale=false currently means "among all the values that have already hit the disk, do not give me stale data". I have been reading your forums and a lot of users seem to be getting confused by this.

I would suggest one of the two things:
1) make stale=false force get everything in the working set. i don't see why you have to wait until the value hits the disk before you are able to include it in a view... but it probably has to do with something internal. perhaps a redesign? also: what happens if i use a memcached bucket? can you even use views? there is no persist stage with memcached buckets so when would you be able to query the view? (i have no current interest in using memcached buckets but am just wondering)
2) stop using the word stale altogether. change it to something along the lines of "not outdated but already persisted data" (that is what your current implementation of stale means)

 Comments   
Comment by d3fault [ 28/Jun/12 ]
Also, I have heard about 'observe' already. While I think it will help the problem somewhat and definitely is handy functionality, a clarification of the word 'stale' will go a lot further in terms of improving usability.
Comment by Dipti Borkar [ 28/Jun/12 ]
thank you for the feedback d3fault
Comment by d3fault [ 01/Feb/13 ]
I would like to remove #1 from the two suggestions. After more thought, I *think* it is impossible to perform because of the CAP theorem requirements. Go with #2 :-P
Comment by Volker Mische [ 27/Aug/14 ]
In 3.0 stale=false also includes the in-memory items.




[MB-9667] 2.2 EE node does not stay running on Windows 7 Created: 03/Dec/13  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: installer
Affects Version/s: 2.2.0
Fix Version/s: bug-backlog
Security Level: Public

Type: Task Priority: Major
Reporter: Mel Boulos Assignee: Mel Boulos
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Windows 7, 2.2 EE


 Description   
I'm working with Ubisoft who is trying to install 2.2 EE on windows 7. This is a single node install on a desktop. The install completes, they are saying the node does not stay running and they are not able to maintain a connection to the admin console. I had them run cbcollect, this is what I see if logs. I've attached the logs, just in case you wanted to see the entire log.

Diag.log
** this error prints throughout the entire log. According to my research this is expected when a new bucket is created, once babysitting restarts the moxi should have the new bucket map. It doesn't look the bucket map is updated so it keeps -retrying.

mb_master:0:info:message(ns_1@127.0.0.1) - I'm the only node, so I'm the master.
2013-11-27 10:18:11.250 menelaus_sup:1:info:web start ok(ns_1@127.0.0.1) - Couchbase Server has started on web port 8091 on node 'ns_1@127.0.0.1'.
2013-11-27 10:18:11.266 ns_log:0:info:message(ns_1@127.0.0.1) - Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: WARNING: curl error: transfer closed with outstanding read data remaining from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming
EOL on stdin. Exiting
2013-11-27 10:18:11.578

ns_server.debug.log
** I initially thought their disk was full. They verified they have 90 GB free.

=========================CRASH REPORT=========================
  crasher:
    initial call: disksup:init/1
    pid: <0.22663.0>
    registered_name: disksup
    exception exit: {badarith,[{disksup,check_disks_win32,2},
                               {disksup,check_disks_win32,2},
                               {disksup,handle_info,2},
                               {gen_server,handle_msg,5},
                               {proc_lib,init_p_do_apply,3}]}
      in function gen_server:terminate/6
    ancestors: [os_mon_sup,<0.22561.0>]
    messages: [{'$gen_call',{<0.22549.0>,#Ref<0.0.3.192839>},
                               get_disk_data}]
    links: [<0.22562.0>]
    dictionary: [{{disk_almost_full,"D:\\"},set}]
    trap_exit: true
    status: running
    heap_size: 987
    stack_size: 24
    reductions: 730

ns_server.babysitting.log
[ns_server:info,2013-11-28T10:03:24.713,babysitter_of_ns_1@127.0.0.1:<0.84.0>:ns_port_server:log:168]memcached<0.84.0>: Thu Nov 28 10:03:24.510621 Est 3: (default) Failed to read from mccouch: "Unknown error"
memcached<0.84.0>: Thu Nov 28 10:03:24.510621 Est 3: (default) Resetting connection to mccouch, lastReceivedCommand = notify_vbucket_update lastSentCommand = notify_vbucket_update currentCommand =unknown
memcached<0.84.0>: Thu Nov 28 10:03:24.510621 Est 3: (default) Trying to connect to mccouch: "127.0.0.1:11213"

[ns_server:info,2013-11-28T10:03:24.728,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:168]ns_server<0.75.0>: win32sysinfo:Erlang has closed.fatal error: runtime: cannot reserve arena virtual address space
ns_server<0.75.0>: in message_loop
ns_server<0.75.0>: win32sysinfo:Erlang has closed.in message_loop
ns_server<0.75.0>: win32sysinfo:Erlang has closed.fatal error: runtime: cannot reserve arena virtual address space
ns_server<0.75.0>: in message_loop
ns_server<0.75.0>: win32sysinfo:Erlang has closed.fatal error: runtime: cannot reserve arena virtual address space
ns_server<0.75.0>: in message_loop
ns_server<0.75.0>: win32sysinfo:Erlang has closed.fatal error: runtime: cannot reserve arena virtual address space


 Comments   
Comment by Bin Cui [ 03/Dec/13 ]
" win32sysinfo:Erlang has closed.fatal error: runtime: cannot reserve arena virtual address space"

By default, installer will put install everything under c:\progrom files. I suspect that they don't have much space on c drive. Maybe they can try on d drive or find another machine and see what happens.
Comment by Michael Catanzariti [ 15/Dec/13 ]
Hi Bin Cui,

As mentioned to Mel, my C drive is far from beeing full (252 Gb free out of 399 Gb).




[MB-12078] JSON detection may not be correct at views Created: 27/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.5.1, 3.0.1, 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Matt Ingenthron Assignee: Sriram Melkote
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Brett Lawson and I were reviewing something for encoding for client libraries and in the process we checked what JSON specs consider legal. There's this interesting bit:

http://tools.ietf.org/html/rfc7159#section-2

   JSON text is a sequence of tokens. The set of tokens includes six
   structural characters, strings, numbers, and three literal names.

   A JSON text is a serialized value. Note that certain previous
   specifications of JSON constrained a JSON text to be an object or an
   array. Implementations that generate only objects or arrays where a
   JSON text is called for will be interoperable in the sense that all
   implementations will accept these as conforming JSON texts.

This actually means that the numeric values stored by memcached protocol are valid JSON, though the view engine doesn't treat them that way. I believe they're detected as non-JSON at view engine. I'm not sure if this is still the case with 3.0, but I thought I should file this since the revelation that a sequence of digits is valid JSON may trigger some thoughts (or unit tests).

 Comments   
Comment by Brett Lawson [ 27/Aug/14 ]
View-engine actually handles this properly, although the document editor does not. A ticket has been opened on that instead.




[MB-12037] ns_server may lose replicas on stopped rebalance/graceful failover (was: {DCP} : Delta Recovery Impossible after re-try of graceful failover since in first attempt failed) Created: 21/Aug/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Parag Agarwal Assignee: Aleksey Kondratenko
Resolution: Fixed Votes: 0
Labels: RC2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 10.6.2.144-10.6.2.150
centos 6x
1174

Triage: Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8202014-2226-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8202014-2227-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8202014-2227-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8202014-2227-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8202014-2227-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8202014-2228-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8202014-2228-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/1174.log.tar.gz
Is this a Regression?: Unknown

 Description   
Scenario
1. Create a 7 Node cluster
2. Create default bucket with 100 K items
3. Graceful failover a node
4. Kill memcached of another node during graceful failover
5. Graceful failover the same node in step 3
6. Add-back the node with delta recovery
7. Hit Rebalance

In Step 7, Rebalance fails for delta recovery. Says delta recovery is not possible. Although we see nodes in the cluster are in healthy state.

We see the following warning:: "Fail Over Warning: Rebalance required, some data is not currently replicated!”

Seems like the delta recovery will not work in this condition, unless we rebalance the cluster. Also, I was able to cancel the delta recovery and do full recovery.

Opening the bug to follow-up on the issue. Attaching logs and data files



 Comments   
Comment by Aleksey Kondratenko [ 21/Aug/14 ]
I was able to reproduce it easily. There's indeed something wrong with restarting graceful failover which impacts delta recovery.
Comment by Aleksey Kondratenko [ 21/Aug/14 ]
And predictably happens with any stop/restart of graceful failover.
Comment by Parag Agarwal [ 21/Aug/14 ]
When the warning is showing (" Rebalance required, some data is not currently replicated!"), we don't expect delta recovery to succeed and this should be correct behavior? Asking since we will have to document it as well

Comment by Aleksey Kondratenko [ 21/Aug/14 ]
Warning has nothing to do with that. And warning is valid. Midway into graceful failover your're not balanced indeed.
Comment by Aleksey Kondratenko [ 21/Aug/14 ]
manifest updated here: http://review.couchbase.org/40811

fix merged here: http://review.couchbase.org/40803
Comment by Parag Agarwal [ 22/Aug/14 ]
Tested
Comment by Parag Agarwal [ 22/Aug/14 ]
Test Run:: http://qa.hq.northscale.net/job/centos_x64--02_01--Rebalance-In/6/console
Comment by Parag Agarwal [ 22/Aug/14 ]
Saw the issue again for the following scenario with build 1186 1

 Scenario
1. Create a 7 Node cluster
2. Create default bucket with 200 K items
3. Graceful failover a node
4. Kill memcached of another node during graceful failover
5. Graceful failover the same node in step 3
6. Add-back the node with delta recovery
7. Hit Rebalance

We see the following warning:: "Fail Over Warning: Rebalance required, some data is not currently replicated!”

In Step 7, Rebalance fails for delta recovery. Says delta recovery is not possible. Although we see nodes in the cluster are in healthy state. This is true when we have 200 K items Vs 100K items where it passes.

I am attaching the logs for you to analyze. Since the above warning comes in both cases. Not sure about the internal state of the system which stops the add-back delta recovery.

Test fails for 2k items
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1436-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-1437-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-1439-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-1440-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1441-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1442-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1444-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1446-couch.tar.gz

Test passes for 1 K Items

https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1458-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-150-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-151-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-153-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-154-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-156-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-157-diag.zip
Comment by Parag Agarwal [ 22/Aug/14 ]
I think the logs were not uploaded, will add them again
Comment by Parag Agarwal [ 22/Aug/14 ]
fixed the logs
Comment by Aleksey Kondratenko [ 22/Aug/14 ]
Please open new ticket for new instance of the issue
Comment by Parag Agarwal [ 22/Aug/14 ]
http://www.couchbase.com/issues/browse/MB-12055
Comment by Wayne Siu [ 26/Aug/14 ]
Reviewed with PM/Cihan. Approved for RC2.




[MB-12055] ns_janitor may lose replicas of nearly completed vbucket moves (was: {DCP} : Delta Recovery Impossible after re-try of graceful failover since in first attempt failed) Created: 22/Aug/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Parag Agarwal Assignee: Parag Agarwal
Resolution: Fixed Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 1186, centos 6x, 10.6.2.144-10.6.2.160

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
build 1186

 Scenario
1. Create a 7 Node cluster
2. Create default bucket with 200 K items
3. Graceful failover a node
4. Kill memcached of another node during graceful failover
5. Graceful failover the same node in step 3
6. Add-back the node with delta recovery
7. Hit Rebalance

We see the following warning:: "Fail Over Warning: Rebalance required, some data is not currently replicated!”

In Step 7, Rebalance fails for delta recovery. Says delta recovery is not possible. Although we see nodes in the cluster are in healthy state. This is true when we have 200 K items Vs 100K items where it passes.

I am attaching the logs for you to analyze. Since the above warning comes in both cases. Not sure about the internal state of the system which stops the add-back delta recovery.

Test fails for 2k items
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1436-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-1437-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-1439-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-1440-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1441-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1442-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1444-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1446-couch.tar.gz

Test passes for 1 K Items

https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1458-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-150-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-151-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-153-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-154-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-156-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-157-diag.zip

 Comments   
Comment by Aleksey Kondratenko [ 25/Aug/14 ]
http://review.couchbase.org/40886
Comment by Aleksey Kondratenko [ 25/Aug/14 ]
manifest updated here: http://review.couchbase.org/40888
Comment by Parag Agarwal [ 25/Aug/14 ]
The issues =is fixed for the scenario mentioned. Also tested it by killing 3 nodes, stopping 3 nodes, stop graceful failover.
Comment by Wayne Siu [ 26/Aug/14 ]
Reopening it for proper tagging (RC2).




[MB-12069] [windows] Dependency management Created: 26/Aug/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sriram Melkote Assignee: Chris Hillery
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
The windows 3rd party binary dependencies should be moved from Google Drive to Amazon S3 or some other suitable location that allows access without a special client.

It will be good if the depot is versioned. If not, we should at least have a directory structure that mirrors the manifest file names of our various builds (i.e, rel-3.0.0)




[MB-12066] [Offline Upgrade 2.1.1.766 -> 3.0.0-1174] items mismatch after upgrade Created: 26/Aug/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: cross-datacenter-replication, ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sangharsh Agarwal Assignee: Sangharsh Agarwal
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Ubuntu 12.04

Attachments: PNG File Screen Shot 2014-08-26 at 12.50.39 PM.png     PNG File Screen Shot 2014-08-26 at 12.54.36 PM.png    
Issue Links:
Gantt: start-finish
is triggered by MB-12075 Couchbase Server constantly shutting ... Resolved
Triage: Untriaged
Operating System: Ubuntu 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: [Source]
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12066/d30b70e7/10.3.3.239-8252014-2346-couch.tar.gz
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12066/e20435ef/10.3.3.239-8252014-2341-diag.zip
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12066/4b079374/10.3.3.240-8252014-2340-diag.zip
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12066/792d226b/10.3.3.240-8252014-2346-couch.tar.gz

[Destination]
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12066/d9c64bf9/10.3.3.218-8252014-2346-couch.tar.gz
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12066/e4f38323/10.3.3.218-8252014-2343-diag.zip
10.3.3.225 : https://s3.amazonaws.com/bugdb/jira/MB-12066/6f010183/10.3.3.225-8252014-2345-diag.zip
10.3.3.225 : https://s3.amazonaws.com/bugdb/jira/MB-12066/e5adac05/10.3.3.225-8252014-2346-couch.tar.gz
Is this a Regression?: Yes

 Description   
[Test Logs]
https://friendpaste.com/VPVWHuLxOZpWUx20pP37o

[Test Steps]
1. Create 2-2 Nodes Source and Destination Cluster.

Source Nodes 10.3.3.240 (Master), 10.3.3.239
Source Nodes 10.3.3.218 (Master), 10.3.3.225


2. Setup CAPI Mode XDCR

bucket: bucket0 (BiXDCR)
bucket: default (UniXDCR)

3. Load 1000 items on bucket0 bucket on the both the cluster.
4. Load 1000 items on default bucket on Source cluster.
5. Offline upgrade both the cluster with 3.0.0-1174-rel.
6. Modify XDCR Settings to use SSL on Source Cluster Only*****.
7. Load 1000 Items on bucket0 and default bucket on Source.
8. Verify items. Items mismatch on Destination Cluster. Items loaded after upgrade ( in Step-7) is not replicated to destination cluster.


[Test Error]
2014-08-25 23:40:08,126 - root - WARNING - Not Ready: curr_items 2000 == 3000 expected on '10.3.3.218:8091''10.3.3.225:8091', bucket0 bucket
2014-08-25 23:40:09,149 - root - WARNING - Not Ready: vb_active_curr_items 2000 == 3000 expected on '10.3.3.218:8091''10.3.3.225:8091', bucket0 bucket
2014-08-25 23:40:09,169 - root - WARNING - Not Ready: vb_replica_curr_items 2000 == 3000 expected on '10.3.3.218:8091''10.3.3.225:8091', bucket0 bucket

2014-08-25 23:40:10,189 - root - WARNING - Not Ready: curr_items 1000 == 2000 expected on '10.3.3.218:8091''10.3.3.225:8091', default bucket
2014-08-25 23:40:11,209 - root - WARNING - Not Ready: vb_active_curr_items 1000 == 2000 expected on '10.3.3.218:8091''10.3.3.225:8091', default bucket
2014-08-25 23:40:11,228 - root - WARNING - Not Ready: vb_replica_curr_items 1000 == 2000 expected on '10.3.3.218:8091''10.3.3.225:8091', default bucket


Issue occurred on Ubuntu VM.



 Comments   
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Cluster is Live for Investigation.

Source: 10.3.3.239, 10.3.3.240
Destination: 10.3.3.218, 10.3.3.225:
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Checked xdcr and info logs on 10.3.3.240 and founds lots of crashes:

[error_logger:error,2014-08-25T23:30:04.135,ns_1@10.3.3.240:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
  crasher:
    initial call: remote_clusters_info:-spawn_request_remote_bucket/5-fun-0-/0
    pid: <0.11425.0>
    registered_name: []
    exception error: no match of right hand side value false
      in function remote_clusters_info:do_with_mcd_to_couch_uri_dict/5 (src/remote_clusters_info.erl, line 1358)
      in call from remote_clusters_info:'-spawn_request_remote_bucket/5-fun-0-'/7 (src/remote_clusters_info.erl, line 702)
    ancestors: [remote_clusters_info,ns_server_sup,ns_server_cluster_sup,
                  <0.60.0>]
    messages: []
    links: [<0.324.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 28690
    stack_size: 27
    reductions: 118182
  neighbours:

[error_logger:error,2014-08-25T23:30:04.149,ns_1@10.3.3.240:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.11415.0> terminating
** Last message in was init
** When Server state == [{data,
                          [{"State",
                            {init_state,
                             {rep,
                              <<"30c42877cc3130e0225e567a98aacadb/bucket0/bucket0">>,
                              <<"bucket0">>,
                              <<"/remoteClusters/30c42877cc3130e0225e567a98aacadb/buckets/bucket0">>,
                              "capi",
                              [{max_concurrent_reps,16},
                               {checkpoint_interval,1800},
                               {doc_batch_size_kb,2048},
                               {failure_restart_interval,30},
                               {worker_batch_size,500},
                               {connection_timeout,180},
                               {worker_processes,4},
                               {http_connections,20},
                               {retries_per_request,2},
                               {optimistic_replication_threshold,256},
                               {socket_options,
                                [{keepalive,true},{nodelay,false}]},
                               {pause_requested,false},
                               {supervisor_max_r,25},
                               {supervisor_max_t,5}]},
                             1004,"capi",<0.1056.0>,<0.1057.0>,<0.1054.0>}}]}]
** Reason for termination ==
** {{badmatch,false},
    {gen_server,call,
        [remote_clusters_info,
         {get_remote_bucket,
             [{uuid,<<"30c42877cc3130e0225e567a98aacadb">>},
              {cert,
                  <<"-----BEGIN CERTIFICATE-----\nMIIC/jCCAeigAwIBAgIIE43leIvyau0wCwYJKoZIhvcNAQEFMCQxIjAgBgNVBAMT\nGUNvdWNoYmFzZSBTZXJ2ZXIgNTk2MmNkMDMwHhcNMTMwMTAxMDAwMDAwWhcNNDkx\nMjMxMjM1OTU5WjAkMSIwIAYDVQQDExlDb3VjaGJhc2UgU2VydmVyIDU5NjJjZDAz\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAn2C66HXyJDgz37O8N/Fq\nFFmrkknnIlxkvMKv4X1goGYKrx7uxhtAkCgzenISUb57IlzHx+rIvJ2DcbjQsDJo\n/F+cfEf7tYnRc/5mGw1r8sKHPQpZWahTc5KD1JFM7RxR+3vb3/n0AHvwN+6m6/w0\nRWFnrJrGlUwZ8t0Szod2ECAsr35k5lVxuiv7LfCTFS3+kpYy/eVRMZPaU9l23Nra\nYyAJEJCZ12RqRwYifZpmjjVRw32a9aGnSXJ5Ygiy3135FV0O47K48+lt1RG8lLjZ\nu33LUQf7cqn2wuplMUBhk2XZUxvUl/lF0GBUvm5PvtmR+r0H+YVGstUD1L8Q1VH8\nBwIDAQABozgwNjAOBgNVHQ8BAf8EBAMCAKQwEwYDVR0lBAwwCgYIKwYBBQUHAwEw\nDwYDVR0TAQH/BAUwAwEB/zALBgkqhkiG9w0BAQUDggEBAJQt+68WEL5tskVHrYgL\nGFcCEIlf29yrtzXHDIprcMhlNE6GiPCaV6BSgA7WH0ni4A9cLJzRfdUWUuLhT4iJ\nV04/qAVyQlYRuzrPC24znG2cfgJERkgdrePILRiCtInAHThfeFOJ8Z9jxjTdBCVh\nPrmsUYYXraN4YDVnDLyQZp/KIYYa5i/yZ7JGUL2mGpWItNs5hi0XJDTAtPCWU0EP\nV3beJqU8YZYrC3cE51A0je+92Bi2uAcmDmuboTXJQtT4Q8Y6WWdhCkkF8Wl1Ljo4\nfNosl8V6ss+AM5h63ntVZgPWkm5JOihvCGGMB91Ab2bpc01gWrBxxBj9oA/lJf/+\na7Q=\n-----END CERTIFICATE-----\n">>},
              {name,"cluster1"},
              {hostname,"10.3.3.218:8091"},
              {username,"Administrator"},
              {password,"password"}],
             "bucket0",false,30000},
         infinity]}}

Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Additionally, not able to open Admin Console on http://10.3.3.239:8091/index.html. Screen is attached.
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Crashes on 10.3.3.218:

[error_logger:error,2014-08-25T23:32:01.284,ns_1@10.3.3.218:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
  crasher:
    initial call: mochiweb_acceptor:init/3
    pid: <0.11931.0>
    registered_name: []
    exception error: no match of right hand side value {error,closed}
      in function mochiweb_http:request/2 (/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/mochiweb/mochiweb_http.erl, line 54)
    ancestors: [menelaus_web_ssl,ns_ssl_services_sup,menelaus_sup,
                  ns_server_sup,ns_server_cluster_sup,<0.60.0>]
    messages: [{ssl_closed,
                      {sslsocket,
                          {gen_tcp,#Port<0.8895>,tls_connection},
                          <0.12060.0>}}]
    links: [<0.3569.0>]
    dictionary: [{random_seed,{1410,4416,29898}}]
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 16778
  neighbours:
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
Do not have sufficient evidence for xdcr mismatch bug.
Comment by Aruna Piravi [ 26/Aug/14 ]
I understand keys are missing but the evidence as you can see from the GUI is this

C1
bucket0 3000
default 2000

C2
bucket0 2000
default 1000

You can see that the 1000 keys inserted after upgrade have not been replicated. Most likely due to MB-12075 that has caused master node in C1 to shutdown and restart every few secs. Pls check.
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
I'm going to bounce _every_ xdcr ticket that doesn't include required information.
Comment by Aruna Piravi [ 26/Aug/14 ]
1000 keys LoadTwo0-LoadTwo999 have not been replicated. We had discussed yesterday to include all the required artifacts. It's possible that since the nodes were unstable, Sangharsh could not retrieve the keys.
Comment by Aruna Piravi [ 26/Aug/14 ]
Fixing 12075 caused replication to complete without data loss. Marking this as duplicate of MB-12075.




[MB-8075] Integrate Error-Handling Blog Content Created: 11/Apr/13  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 2.1.0
Fix Version/s: techdebt-backlog
Security Level: Public

Type: Bug Priority: Major
Reporter: Anonymous Assignee: Don Pinto
Resolution: Fixed Votes: 0
Labels: info-request
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
4/11 Don preparing blog on error handling. In progress. Will send and to integrate into docs.

 Comments   
Comment by kzeller [ 11/Apr/13 ]
Hi Don,

After this has been publically published, please send the final version so I can integrate into the main doc set.


I may also need to schedule a call/skype to understand the best place to roll this in.


Thanks,

Karen
Comment by Amy Kurtzman [ 24/Sep/13 ]
Hi Don,

Do you know what's up on this one? Did the blog on error handling get published? Did it ever get integrated into the CB Server documentation?

Thanks,
Amy
Comment by Don Pinto [ 22/Jan/14 ]
These are the 2 blogs that were create on error handling -

http://blog.couchbase.com/error-handling-java-client-library
http://blog.couchbase.com/handling-runtime-errors-ruby-python-and-c-clients

Comment by Amy Kurtzman [ 22/Jan/14 ]
Matt, Can you tell me if these blog posts are still relevant to the clients? They are from last year, and things might have changed. Also, if we still want to incorporate the material into the documentation, where do they fit? Developer Guide or individual SDK guides? The first one is Java only, but the second one refers to multiple clients.

Thanks, Amy
Comment by Amy Kurtzman [ 23/Jun/14 ]
Matt, please verify whether this is still needed or can be closed.
Comment by Matt Ingenthron [ 23/Jun/14 ]
Yes and no. I know we've improved, but I don't know if all points are covered. Finding out will take just a careful checklist review.

Don: when you have time, since you're the original blog owner and now driving checklist-y things, can you review?




[MB-11930] [windows] all data lost after offline upgrade from 2.0.0 to 3.0.0-1134 Created: 11/Aug/14  Updated: 25/Aug/14  Resolved: 25/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: installer
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Thuan Nguyen Assignee: Bin Cui
Resolution: Fixed Votes: 0
Labels: Windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: windows 2008 R2 64-bit

Attachments: Zip Archive 12.11.10.137-8112014-1532-diag.zip     Zip Archive 1.zip    

 Description   
Install couchbase server 2.0.0 in one node windows server 2008 R2 64-bit
Create default bucket and load 10K items to default bucket
Offline upgrade to 3.0.0-1134.
After upgrade done, couchbase server 3.0.0 is up but no items loaded to default bucket

 Comments   
Comment by Bin Cui [ 22/Aug/14 ]
All data and config.dat files have been copied back to original locations after upgrade. However, the config.dat file is overwritten after server starts because it treats it as a brand new setup. Need to talk to Alk to find out why.
Comment by Bin Cui [ 25/Aug/14 ]
Reproduce steps:

1. Install 2.5.1 build 1083 on c:\t1
2. upload and populate bucket gamesim-sample
3. upgrade to build 3.0 build 1182.
4. Observe that both config.dat and data files are restored to original locations. However, config.dat is overwritten and file size is reduced from 150k to 10k.

Upload cbcollect_info logs.
Comment by Bin Cui [ 25/Aug/14 ]
Assign to Aliak A for further analysis. Feel free to assign it back to me if any file is missing or misplaced during upgrade.
Comment by Aliaksey Artamonau [ 25/Aug/14 ]
We read 10.17.49.186 ip address from ip file. But in 2.5 incarnation the server was configured to use 127.0.0.1 as an address. This is what leads to the described behavior.
Comment by Bin Cui [ 25/Aug/14 ]
http://review.couchbase.org/#/c/40891/




[MB-11156] able to failover all nodes from the cluster: used couchbase-cli Created: 19/May/14  Updated: 25/Aug/14  Resolved: 25/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: ns_server, tools, UI
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Andrei Baranouski Assignee: Parag Agarwal
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-692

Attachments: PNG File failover_all_nodes.png     PNG File MB-1156.png     PNG File Screen Shot 2014-07-07 at 7.10.50 PM.png    
Triage: Untriaged
Operating System: Centos 64-bit
Is this a Regression?: Unknown

 Description   
so, will try to give more detailed steps to reproduce, but I got it during the implementation of the tests on failover, delta recovery, rebalance operation, etc using couchbase-cli

I also filled separate UI ticket for the issue: MB-11155

https://s3.amazonaws.com/bugdb/jira/MB-11155/47461e2d/10.3.4.144-5192014-529-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-11155/47461e2d/10.3.4.146-5192014-530-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-11155/47461e2d/10.3.4.147-5192014-533-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-11155/47461e2d/10.3.4.148-5192014-532-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-11155/47461e2d/10.3.4.149-5192014-534-diag.zip

 Comments   
Comment by Steve Yen [ 05/Jun/14 ]
> so, will try to give more detailed steps to reproduce...

Hi Andrei,
Any chance (best case!) you might be able to give couchbase-cli command line? (Perhaps it's a couchbase-cli bug?)
Comment by Andrei Baranouski [ 05/Jun/14 ]
not sure that it's completely couchbase-cli issue. believe that ns_server should not be allowed to do so also

steps:

2 nodes in the cluster: 10.3.4.144 & 10.3.4.145

[root@localhost bin]# ./couchbase-cli failover --cluster 10.3.4.144:8091 --server-failover=10.3.4.145:8091 -u Administrator -p password
INFO: graceful failover .
SUCCESS: failover ns_1@10.3.4.145
[root@localhost bin]# ./couchbase-cli failover --cluster 10.3.4.144:8091 --server-failover=10.3.4.144:8091 -u Administrator -p password
INFO: graceful failover .
SUCCESS: failover ns_1@10.3.4.144

please, see screenshot after these steps
Comment by Steve Yen [ 05/Jun/14 ]
Thanks Andrei.

Hi Bin,
Can you take a quick peek at this on the chance that CLI is mis-reporting some error result as a success?

Otherwise, this bug is probably ns-server related (missing some error case, perhaps?)

Thanks,
steve

Comment by Bin Cui [ 09/Jun/14 ]
Andrei,

Can you try out the latest build? I cannot reproduce the problem by following your test steps:
1. create two node cluster.
2. failover first one. Return successful as expected.
3. failover the second one. Error returns as expected.

-bash-3.2$ ./couchbase-cli failover --cluster 10.5.2.133 --server-failover=10.5
.2.133:8091 -u Administrator -p 123456
INFO: graceful failover . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . .
SUCCESS: failover ns_1@10.5.2.133
-bash-3.2$ ./couchbase-cli failover --cluster 10.6.2.91 --server-failover=10.6.
2.91:8091 -u Administrator -p 123456
ERROR: unable to failover ns_1@10.6.2.91 (400) Bad Request
ERROR: command: failover: 10.6.2.91:8091, No JSON object could be decoded
-bash-3.2$ ./couchbase-cli failover --cluster 10.6.2.91:8091 --server-failover=
10.6.2.91:8091 -u Administrator -p 123456
ERROR: unable to failover ns_1@10.6.2.91 (400) Bad Request
ERROR: command: failover: 10.6.2.91:8091, No JSON object could be decoded
Comment by Andrei Baranouski [ 10/Jun/14 ]

there are 2 cases:
a) no buckets in the cluster:
[root@localhost bin]# ./couchbase-cli failover --cluster 10.3.4.144:8091 --server-failover=10.3.4.145:8091 -u Administrator -p password
INFO: graceful failover
SUCCESS: failover ns_1@10.3.4.145

root@localhost bin]# ./couchbase-cli failover --cluster 10.3.4.144:8091 --server-failover=10.3.4.144:8091 -u Administrator -p password
INFO: graceful failover
SUCCESS: failover ns_1@10.3.4.144

b) there is a bucket in the cluster

[root@localhost bin]# ./couchbase-cli failover --cluster 10.3.4.148:8091 --server-failover=10.3.4.148:8091 -u Administrator -p password
INFO: graceful failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SUCCESS: failover ns_1@10.3.4.148
[root@localhost bin]# ./couchbase-cli failover --cluster 10.3.4.149:8091 --server-failover=10.3.4.149:8091 -u Administrator -p password
ERROR: unable to failover ns_1@10.3.4.149 (400) Bad Request
ERROR: command: failover: 10.3.4.149:8091, No JSON object could be decoded

it's fine but with --force:
[root@localhost bin]# ./couchbase-cli failover --cluster 10.3.4.149:8091 --server-failover=10.3.4.149:8091 -u Administrator -p password --force
SUCCESS: failover ns_1@10.3.4.149


so, for case with buckets we have to apply "--force" to get the same picture as in MB-11155



Comment by Bin Cui [ 10/Jun/14 ]
Again, assign to ns_server team to check if the test results are in line with expectation.
Comment by Aleksey Kondratenko [ 10/Jun/14 ]
Certainly, not a bug with CLI. Error checking and validation is supposed to be done in ns_server. And I think I've seen duplicate of this bug somewhere.
Comment by Anil Kumar [ 19/Jun/14 ]
Triage - June 19 2014 Alk, Parag, Anil
Comment by Aliaksey Artamonau [ 30/Jun/14 ]
http://review.couchbase.org/38906
Comment by Andrei Baranouski [ 07/Jul/14 ]
build 3.0.0-918

[root@centos-64-x64 bin]# ./couchbase-cli failover --cluster 172.23.105.156:8091 --server-failover=172.23.105.156:8091 -u Administrator -p password
INFO: graceful failover
SUCCESS: failover ns_1@127.0.0.1

http://www.couchbase.com/issues/secure/attachment/21172/Screen%20Shot%202014-07-07%20at%207.10.50%20PM.png
Comment by Aliaksey Artamonau [ 07/Jul/14 ]
http://review.couchbase.org/39182
Comment by Parag Agarwal [ 24/Aug/14 ]

For last node failover in a cluster, It returns success for Hard failover

[root@palm-10307 bin]# ./couchbase-cli failover --cluster 10.3.4.144:8091 --server-failover=10.3.4.144:8091 -u Administrator -p password --force
SUCCESS: failover ns_1@10.3.4.144

However does not work for graceful, which is as expected

[root@palm-10307 bin]# ./couchbase-cli failover --cluster 10.3.4.144:8091 --server-failover=10.3.4.144:8091 -u Administrator -p password
ERROR: command: failover: 10.3.4.144:8091, node 10.3.4.144:8091 is not active
[root@palm-10307 bin]#

Why is hard failover allowed, meanwhile in UI NO failover is allowed. Also, running this comamnd causes no damage to the cluster.
Comment by Aliaksey Artamonau [ 25/Aug/14 ]
I cannot reproduce your problem and not able to do anything else without logs. There's a good chance that you're doing something wrong.
Comment by Parag Agarwal [ 25/Aug/14 ]
I ran hard failover via cli with one node present in cluster. ok, will attach logs asap
Comment by Parag Agarwal [ 25/Aug/14 ]
I verified this again. Could not repro. Works both for hard and graceful failover (with/without buckets). Closing this bug

Output of command execution

[root@palm-10307 bin]# ./couchbase-cli failover --cluster 10.6.2.144:8091 --server-failover=10.6.2.144:8091 -u Administrator -p password --force
ERROR: unable to failover ns_1@10.6.2.144 (400) Bad Request
ERROR: command: failover: 10.6.2.144:8091, No JSON object could be decoded
[root@palm-10307 bin]# ./couchbase-cli failover --cluster 10.6.2.144:8091 --server-failover=10.6.2.144:8091 -u Administrator -p password
ERROR: unable to failover ns_1@10.6.2.144 (400) Bad Request
ERROR: command: failover: 10.6.2.144:8091, No JSON object could be decoded
[root@palm-10307 bin]# ./couchbase-cli failover --cluster 10.6.2.144:8091 --server-failover=10.6.2.144:8091 -u Administrator -p password --force
ERROR: unable to failover ns_1@10.6.2.144 (400) Bad Request
ERROR: command: failover: 10.6.2.144:8091, No JSON object could be decoded
[root@palm-10307 bin]#




[MB-11685] windows uninstall failed to remove files Created: 10/Jul/14  Updated: 25/Aug/14  Resolved: 25/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: installer
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Thuan Nguyen Assignee: Bin Cui
Resolution: Cannot Reproduce Votes: 0
Labels: Windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: windows 2008 R2 64bit

Attachments: PNG File ss_2014-07-10_at_5.48.41 PM.png    
Triage: Untriaged
Operating System: Windows 64-bit
Is this a Regression?: Unknown

 Description   
Install couchbase server 3.0.0-936 on windows server 2008 R2 64-bit
Uninstall couchbase server 3.0.0-936, the uninstall process did finish but it did not delete couchbase server files
under c:/Program Files/Couchbase/Server

Install couchbase server 3.0.0-949 on windows server 2008 R2 64-bit
Uninstall couchbase server 3.0.0-9349. Got the same issue. All files are not deleted.
IP of windows server: 10.1.2.92
Vm is available for debug


 Comments   
Comment by Chris Hillery [ 11/Jul/14 ]
Bin, please take a look - I think you're the only one with an InstallShield license.
Comment by Thuan Nguyen [ 14/Jul/14 ]
Bin, do you still need vm do debug?
Comment by Bin Cui [ 25/Aug/14 ]
After i uninstalled build 1182, files are removed as expected.




[MB-11329] uninstall couchbase server 3.0.0 on windows did not delete files completely Created: 05/Jun/14  Updated: 25/Aug/14  Resolved: 25/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Thuan Nguyen Assignee: Bin Cui
Resolution: Cannot Reproduce Votes: 0
Labels: Windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: windows 2008 R2 64-bit

Attachments: PNG File ss_2014-06-05_at_11.18.37 AM.png    
Triage: Triaged
Operating System: Windows 64-bit
Is this a Regression?: Unknown

 Description   
Install couchbase server 3.0.0-779 on windows server 3.0.0-779 from
this link http://factory.hq.couchbase.com:8080/job/cs_300_win6408/186/artifact/voltron/couchbase-server-enterprise-3.0.0-779.setup.exe

Then uninstall couchbase server.
When uninstall complete, there are many files left in c:/Program Files/Couchbase/Server/var/lib/couchbase

 Comments   
Comment by Bin Cui [ 17/Jun/14 ]
It essentially means that uninstallation doesn't proceed correctly. And I think it is related to erlang process still running after uninstallation. We need to revisit the window build.
Comment by Bin Cui [ 25/Aug/14 ]
After i uninstalled build 1182, files are deleted as expected.




[MB-11328] old erlang processes were still running after uninstall couchbase server 3.0.0 on windows Created: 05/Jun/14  Updated: 25/Aug/14  Resolved: 25/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Thuan Nguyen Assignee: Bin Cui
Resolution: Cannot Reproduce Votes: 0
Labels: Windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: windows 2008 R2 64-bit

Attachments: PNG File ss_2014-06-05_at_10.48.16 AM.png    
Triage: Triaged
Operating System: Windows 64-bit
Is this a Regression?: Unknown

 Description   
Install couchbase server 3.0.0-779 on windows server 3.0.0-779 from
this link http://factory.hq.couchbase.com:8080/job/cs_300_win6408/186/artifact/voltron/couchbase-server-enterprise-3.0.0-779.setup.exe

Then uninstall couchbase server. In windows task manager, erlang processes were still there.
These left over erlang processes would make UI fail to run in next install couchbase server on windows.

 Comments   
Comment by Bin Cui [ 09/Jun/14 ]
Most likely, the erlang process get hunged and it won't exit request from service control manager when installation happens. Do we have other erlang issues during the run?
Comment by Thuan Nguyen [ 09/Jun/14 ]
Yes, we have issue during the run since there are some extra erlang processes running.
Comment by Sriram Melkote [ 10/Jun/14 ]
Bin, can we have the installer run:

taskkill.exe /im beam.smp /f
taskkill.exe /im epmd.exe /f
taskkill.exe /im memcached.exe /f

After stopping service and before beginning uninstall? The epmd is the important one, others are just to be safe.
Comment by Bin Cui [ 10/Jun/14 ]
This is definitely a bandaged kind of fix and it will cover the more fatal issue, i.e. a corrupted image in erlang process. Installer can double check and kill these unresponsive process. But we still need to dig deeper to find the root cause.

Since we register erlang as service and all these processes are under control of erlang management. Only corrupted processes will not response to parent process.
Comment by Bin Cui [ 25/Aug/14 ]
After i unintall build 1182 for windows, all erlang processes are gone.




[MB-12058] ip:18091 should redirect on https://ip:18091 Created: 24/Aug/14  Updated: 25/Aug/14  Resolved: 24/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Andrei Baranouski Assignee: Aleksey Kondratenko
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1974

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
unable to open url like 10.3.121.134:18091/index.html( or http://10.3.121.134:18091/index.html)


it's better to redirect on https because now loading 'ip:18091' stuck

 Comments   
Comment by Aleksey Kondratenko [ 24/Aug/14 ]
Unfortunately I'm not aware of anybody able to do that. You really have to specify https.
Comment by Andrei Baranouski [ 25/Aug/14 ]
I see, but the problem is that the request( ip:18091) hangs (we do not get any timeout error or 'page unavailable') and it is wrong
Comment by Aleksey Kondratenko [ 25/Aug/14 ]
you'll get this with _any_ https thing accessed as http




Generated at Sat Aug 30 08:27:18 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.