[MB-12098] [Query-XDCR] One extra returned from query with stale=false Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Sangharsh Agarwal Assignee: Sangharsh Agarwal
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: CentOS64

Triage: Untriaged
Operating System: Centos 64-bit
Is this a Regression?: Unknown

 Description   
http://qa.hq.northscale.net/job/centos_x64--107_01--rebalanceXDCR-P1/57/consoleFull

[Test Error]

======================================================================
ERROR: swap_rebalance_replication_with_view_queries_and_ops (xdcr.rebalanceXDCR.Rebalance)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/xdcr/rebalanceXDCR.py", line 283, in swap_rebalance_replication_with_view_queries_and_ops
    [task.result(self._poll_timeout) for task in tasks]
  File "lib/tasks/future.py", line 160, in result
    return self.__get_result()
  File "lib/tasks/future.py", line 112, in __get_result
    raise self._exception
QueryViewException: Error occured querying view default0: expected number of rows: '70000' is greater than expected 70001

----------------------------------------------------------------------
Ran 1 test in 2215.179s


[Test Steps]
Source: 10.5.2.228, 10.5.2.229, 10.5.2.230
Destination: 10.5.2.231, 10.5.2.232, 10.5.2.233

1. Setup 3-3 node Source and Destination Cluster.
2. Bucket: default, mode: CAPI mode unidirectional xdcr.
3. Load 1M items on Source.
4. Create DesignDoc: ddoc1 on Source and Destination
5. Created views: default0, default1, default2, default3, default4 on Source and Destination.
6. RUN Parallel
     i. 30K and Delete 30K items on Source
    ii. SWAP Rebalance one node on Source (Swaped 10.5.2.230 with 10.6.2.192 on Source)
   iii. Run queries on all views on source and destination with stale=false e.g.http://10.5.2.231:8092/default/_design/ddoc1/_view/default1?full_set=true&stale=false)

7. Wait for re-balance to finished.
8. Again run queries on all views with stale=false to verify expected rows=70000.

Error Expected rows: 70000 and Actual: 70001


 Comments   
Comment by Sangharsh Agarwal [ 29/Aug/14 ]
Bug needs to be updated, I will assign again.




[MB-12097] [Utf-16 XDCR] Data loss - No items replicated to destination Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Sangharsh Agarwal Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: CentOs5 (64 bit)

Attachments: PNG File Screen Shot 2014-08-29 at 5.00.49 PM.png    
Triage: Triaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: [Source]
10.3.2.109 : https://s3.amazonaws.com/bugdb/jira/MB-12097/2e776f21/10.3.2.109-8292014-433-couch.tar.gz
10.3.2.109 : https://s3.amazonaws.com/bugdb/jira/MB-12097/6eff2cf5/10.3.2.109-8292014-429-diag.zip
10.3.2.109 : https://s3.amazonaws.com/bugdb/jira/MB-12097/a4c3ccf0/10.3.2.109-diag.txt.gz

10.3.4.175 : https://s3.amazonaws.com/bugdb/jira/MB-12097/27df791c/10.3.4.175-diag.txt.gz
10.3.4.175 : https://s3.amazonaws.com/bugdb/jira/MB-12097/3db3fbfd/10.3.4.175-8292014-433-couch.tar.gz
10.3.4.175 : https://s3.amazonaws.com/bugdb/jira/MB-12097/70f9482c/10.3.4.175-8292014-428-diag.zip

[Erlang Crashed]
erlang : https://s3.amazonaws.com/bugdb/jira/MB-12097/1838e7ea/erlang-10.3.4.175-0.log
erlang : https://s3.amazonaws.com/bugdb/jira/MB-12097/f90a602e/erlang-10.3.2.109-0.log

[Destination]
10.3.4.176 : https://s3.amazonaws.com/bugdb/jira/MB-12097/2c2138ce/10.3.4.176-8292014-432-diag.zip
10.3.4.176 : https://s3.amazonaws.com/bugdb/jira/MB-12097/7a3deb96/10.3.4.176-diag.txt.gz
10.3.4.176 : https://s3.amazonaws.com/bugdb/jira/MB-12097/fd6123e8/10.3.4.176-8292014-433-couch.tar.gz
10.3.2.161 : https://s3.amazonaws.com/bugdb/jira/MB-12097/0468eed0/10.3.2.161-8292014-433-couch.tar.gz
10.3.2.161 : https://s3.amazonaws.com/bugdb/jira/MB-12097/158395ba/10.3.2.161-diag.txt.gz
10.3.2.161 : https://s3.amazonaws.com/bugdb/jira/MB-12097/f1c6caf0/10.3.2.161-8292014-431-diag.zip

Is this a Regression?: Yes

 Description   
[Test]
./testrunner -i INI/xdcr.4.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=True,get-coredumps=True -t cornercases.negativetests.NegativeTests2.test_utf_16_keys_with_xdcr,rdirection=unidirection,ctopology=chain,items=100

[Test Logs]
https://friendpaste.com/12jJEznsAjjuxZR6ZAF8Fn

[Live Cluster]


[Source]
10.3.2.109
10.3.4.175

[Destination]
10.3.2.161
10.3.4.176

1. Setup Uni-XDCR CAPI mode XDCR from Source -> Destination. Bucket: default
2. Load 100 Items encoded with UTF-16.
3. No items transferred to Destination.


test is passed in 2.5.1-1083


Keys Not Replicated: ��loadOne0, ��loadOne1, ��loadOne2, ..... ��loadOne99

Screen Shot attached for the documents on Source side. Cluster is Live for debugging.

 Comments   
Comment by Sangharsh Agarwal [ 29/Aug/14 ]
Erlang crash also happened on Source cluster, crash is also attached.




[MB-12096] collect_server_info.py does not work on a dev tree on windows.. Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Server
Component/s: test-execution
Affects Version/s: techdebt-backlog
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Trond Norbye Assignee: Tommie McAfee
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
cbcollectinfo.py tries to use ssh to collect the files even if the target machine is the same machine as the test is running on, and that doesn't seem to work on my windows development box. Since all of the files should be local it could might as well use normal copy.




[MB-12095] Change default values of the spatial view UI Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Server
Component/s: UI, view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Volker Mische Assignee: Volker Mische
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Spatial views don't need a bounding box anymore. If none is defined the full data set is returned. Additionally spatial views support skip and limit to page through the results.




[MB-12094] Integrate ForestDB into EP-Engine Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: feature-backlog
Fix Version/s: feature-backlog
Security Level: Public

Type: Bug Priority: Critical
Reporter: Chiyoung Seo Assignee: Sundar Sridharan
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
We plan to replace Couchstore with ForestDB as the underlying storage engine for the EP-Engine. This integration task mainly requires the implementation of the abstract KV APIs defined in the EP-Engine.
 




[MB-12093] Spatial views return value is double encoded Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: None
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Volker Mische Assignee: Volker Mische
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Yes

 Description   
When you query a spatial view then the `value` in the response is double JSON encoded as string. This means that if you return a number it becomes a string. And a string becomes a string with escaped quotes. For example:

The emit -> current value in the response -> expected value in the response
emit(key, 5) -> {"value: "5"} -> {"value": 5}
emit(key, "some string"} -> {"value": "\"some string\""} -> {"value": "some string"}
emit(key, null) -> {"value: "null"} -> {"value": null}


 Comments   
Comment by Volker Mische [ 28/Aug/14 ]
http://review.couchbase.org/41027




[MB-12092] Spatial views UI doesn't display results Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: UI, view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Volker Mische Assignee: Volker Mische
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
When you have a spatial view and click on "Show Results" no results are shown.

 Comments   
Comment by Volker Mische [ 28/Aug/14 ]
http://review.couchbase.org/41024




[MB-12091] [Windows].compact files not cleaned up after compaction Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Venu Uppalapati Assignee: Sriram Ganesan
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive cbinfo.zip    
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Steps to reproduce:
1)Load 1M items on default so that compaction runs several times by end of the loading.
2)It will be seen that all the temporary .compact files are kept even after compaction is done.
3)for ex:
*.couch.1.compact.btree-tmp-1
*.couch.1.compact.btree-tmp-2
*.couch.1.compact.btree-tmp-3
*.couch.1.compact.btree-tmp-4
when the current rev number of the vbucket file is 5
4)This is windows specific. Pretty soon disk space is claimed by all these temp files.

 Comments   
Comment by Chiyoung Seo [ 28/Aug/14 ]
Sriram,

Sundar is busy with working on the RC2-related issue. Can you please take a look at this window compaction issue?
Comment by Sriram Ganesan [ 28/Aug/14 ]
If there are any logs available from the test, please do upload them.
Comment by Venu Uppalapati [ 28/Aug/14 ]
Sundar mentioned that he used unlink function which is deprecated on Windows version of the compiler, http://msdn.microsoft.com/en-us/library/ms235350.aspx will upload the logs shortly.
Comment by Venu Uppalapati [ 28/Aug/14 ]
cbcollectinfo attached




[MB-12090] add stale=false semantic changes to dev guide Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0, 3.0-Beta
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Matt Ingenthron Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: No

 Description   
Need to change the dev guide to explain the semantics change with the stale parameter.

 Comments   
Comment by Matt Ingenthron [ 28/Aug/14 ]
I could not find the 3.0 dev guide to write up something. I've generated a diff based on the 2.5 dev guide. Note that much of that dev guide refers to the 3.0 admin guide section on views. I could not find that in the "dita" directory so I could contribute a change to the XML. I think based on this and what I put in MB-12052 should help.


diff --git a/content/couchbase-devguide-2.5/finding-data-with-views.markdown b/content/couchbase-devguide-2.5/finding-data-with-views.markdown
index 77735b9..811dff0 100644
--- a/content/couchbase-devguide-2.5/finding-data-with-views.markdown
+++ b/content/couchbase-devguide-2.5/finding-data-with-views.markdown
@@ -1,6 +1,6 @@
 # Finding Data with Views
 
-In Couchbase 2.1.0 you can index and query JSON documents using *views*. Views
+In Couchbase you can index and query JSON documents using *views*. Views
 are functions written in JavaScript that can serve several purposes in your
 application. You can use them to:
 
@@ -323,16 +323,25 @@ Forinformation about the sort order of indexes, see the
 [Couchbase Server Manual](http://docs.couchbase.com/couchbase-manual-2.5/cb-admin/).
 
 The real-time nature of Couchbase Server means that an index can become outdated
-fairly quickly when new entries and updates occur. Couchbase Server generates
-the index when it is queried, but in the meantime more data can be added to the
-server and this information will not yet be part of the index. To resolve this,
-Couchbase SDKs and the REST API provide a `stale` parameter you use when you
-query a view. With this parameter you can indicate you will accept the most
-current index as it is, you want to trigger a refresh of the index and retrieve
-these results, or you want to retrieve the existing index as is but also trigger
-a refresh of the index. For instance, to query a view with the stale parameter
-using the Ruby SDK:
+fairly quickly when new entries and updates occur. Couchbase Server updates
+the index at the time the query is received if you supply the argument
+`false` to the `stale` parameter.
+
+<div class="notebox">
+<p>Note</p>
+<p>Starting with the 3.0 release, the "stale" view query argument
+"false" has been enhanced so it will consider all document changes
+which have been received at the time the query has been received. This
+means that use of the `durability requirements` or `observe` feature
+to block for persistence in application code before issuing the
+`false` stale query is no longer needed. It is recommended that you
+remove all such application level checks after completing the upgrade
+to the 3.0 release.
+</p>
+</div>
 
+For instance, to query a view with the stale parameter
+using the Ruby SDK:
 
 ```
 doc.recent_posts(:body => {:stale => :ok})
@@ -905,13 +914,14 @@ for(ViewRow row : result) {
 }
 ```
 
-Before we create a Couchbase client instance and connect to the server, we set a
-system property 'viewmode' to 'development' to put the view into production
-mode. Then we query our view and limit the number of documents returned to 20
-items. Finally when we query our view we set the `stale` parameter to FALSE to
-indicate we want to reindex and include any new or updated beers in Couchbase.
-For more information about the `stale` parameter and index updates, see Index
-Updates and the Stale Parameter in the
+Before we create a Couchbase client instance and connect to the
+server, we set a system property 'viewmode' to 'development' to put
+the view into production mode. Then we query our view and limit the
+number of documents returned to 20 items. Finally when we query our
+view we set the `stale` parameter to FALSE to indicate we want to
+consider any recent changes to documents. For more information about
+the `stale` parameter and index updates, see Index Updates and the
+Stale Parameter in the
 [Couchbase Server Manual](http://docs.couchbase.com/couchbase-manual-2.5/cb-admin/#couchbase-views-writing-stale).
 
 The last part of this code sample is a loop we use to iterate through each item




[MB-12089] Support in packages for installing as a different user from 'couchbase' Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: installer
Affects Version/s: 2.5.1, 3.0.1, 3.0, 3.0-Beta
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Major
Reporter: Brent Woodruff Assignee: Bin Cui
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We package Couchbase on Linux using RPM and DEB. During installation, a 'couchbase' user and group is created if it does not exist. It would be useful and practical to support the installation/upgrade of the package using a username and group name different than 'couchbase'.

I suggest we support the exporting of a name in the environment prior to installation that would affect the packaging scripts, and then maintain the username in a configuration file for running and upgrades.

i.e.

export COUCHBASE_USER=cbuser
export COUCHBASE_GROUP=cbgroup
rpm -ivh couchbase-server*.rpm

Results in using cbuser:cbgroup and records this in a configuration file somewhere that is sourced by the init.d script and the installer in the future for upgrades.

Based on Couchbase's ability to run in a non-root install and also on Mac as any Mac user, I don't believe there would be any problem with supporting this, as the user does not appear to be a hard, baked-in assumption anywhere.

 Comments   
Comment by Bin Cui [ 28/Aug/14 ]
Please assign it back to me if this request can be met in couchbase-server.sh script.
Comment by Bin Cui [ 28/Aug/14 ]
http://review.couchbase.org/#/c/41070/




[MB-12088] Memcached should return an uninitiated error code Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: memcached
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Major
Reporter: Patrick Varley Assignee: Trond Norbye
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
This is a blow out from MB-11875 and CBSE-1370.

This is no way that CCCP can tell if a node is uninitiated versus auth error.

Currently LCB does not retry on auth error.

This can be painful when a user is doing maintenance as it mean they have to keep updating the bootstrap list when they have removed a node from the cluster which they plan to add back in.

If memcached returned an uninitiated error code LCB would know to try the other nodes in the bootstrap list.

 Comments   
Comment by Patrick Varley [ 28/Aug/14 ]
What I mean by "uninitiated" is the node has been removed from the cluster and is at the setup wizard.




[MB-12087] License data Created: 28/Aug/14  Updated: 28/Aug/14  Resolved: 28/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Sriram Melkote Assignee: Chris Hillery
Resolution: Fixed Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
We need to have proper license information for all modules, and third party components.

 Comments   
Comment by Sriram Melkote [ 28/Aug/14 ]
This is a placeholder bug to track approval to make change for 3.0 and capture all reviews that get merged
Comment by Volker Mische [ 28/Aug/14 ]
Our sample datasets don't have a proper license: http://review.couchbase.org/40983
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
Great catch - approved for RC2
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
sending over to Wayne for ensuring this will make it through the build into the final set of bits. pls add this to both community and enterprise editions.




[MB-12086] View Query timing out when rebalance out running in parallel Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server, view-engine
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Meenakshi Goel Assignee: Sarath Lakshman
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1199-rel, Debian 7.0

Attachments: File ns_logs.tar     File query.pcap     Text File Run.log    
Triage: Triaged
Operating System: Centos 64-bit
Is this a Regression?: Yes

 Description   
Jenkins Ref Link:
http://qa.hq.northscale.net/job/centos_x64--44_04--observe-P0/137/console
http://qa.hq.northscale.net/job/debian_x64--90_04--observe_tests-P0/5/console

Test to Reproduce:
./testrunner -i test.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False -t observe.observetest.ObserveTests.test_observe_basic_data_load_delete,items=100,nodes_in=1,rebalance=out
./testrunner -i tests.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False -t observe.observetest.ObserveTests.test_observe_basic_data_load_delete,items=100,nodes_in=1,mutate_by=multi_set,rebalance=out,GROUP=P1

Steps to Reproduce:
1. Create bucket
2. Rebalance-in 2 nodes
3. Load data 100 items
4. Wait for item to get persist on disk
5. Run re-balance out and Observe in parallel
6. Create View and Run query on view with stale=false

Uploading Logs



 Comments   
Comment by Meenakshi Goel [ 28/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12086/b2519932/10.1.3.74-8282014-131-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12086/748f9bed/10.1.3.75-8282014-132-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12086/66df8790/10.1.3.80-8282014-133-diag.zip

These logs are with RC1 build, I am uploading logs with latest build 3.0.0-1199-rel too
Comment by Meenakshi Goel [ 28/Aug/14 ]
With 3.0.0-1199-rel
http://qa.hq.northscale.net/job/centos_x64--44_04--observe-P0/142/console
https://s3.amazonaws.com/bugdb/jira/MB-12086/8d54b309/10.3.5.163-8282014-316-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12086/95a40c18/10.5.2.94-8282014-317-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12086/2382e371/10.5.3.20-8282014-318-diag.zip
Comment by Sriram Melkote [ 28/Aug/14 ]
I've attached a pcap. At packet #9716, there's a stale=false query made.

But that returns only after 1m4s with node_vbuckets_dict_failed / not_present error. Apparently, testrunner doesn't expect it to take that long.

Given this worked in the past, I think the question open is why it's taking much longer to report this.
Comment by Sarath Lakshman [ 28/Aug/14 ]
When a query hits a rebalanced out node, we are getting this error "{"error":"node_vbuckets_dict_failed","reason":"not_present"}", which is fine. The client can retry and hit a node which is part of the cluster.

Is it normal for ns_server to take 1m for vbucket_map_mirror lookup and return node_vbuckets_dict_failed error ?
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
It is not expected for vbucket_map_mirror to take 1 minute. I cannot rule it out completely but it's very unlikely.

What likely happens is that we retry request multiple times internally.
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
let research - we'll consider for RC2 once we know the resolution & impact.




[MB-12085] query with key=value requires inclusive_end=true Created: 27/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Tommie McAfee Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Yes

 Description   
In 3.0 the view editor creates urls with inclusive_end=false. For some reason this causes key=value queries to return empty results. In 2.5 the param was left unset.

1. load beer sample

2. created view with map:

function (doc, meta) {
   emit(doc.state, doc.name);
}

3. In filter result drop down:
key = "Alaska"

Generates:
http://172.23.106.53:8092/beer-sample/_design/dev_ddoc1/_view/view1?stale=false&inclusive_end=false&key=%22Alaska%22&connection_timeout=60000&limit=10&skip=0

need to manually check inclusive_end=false.

2.5 Generates:
http://172.23.106.53:8092/beer-sample/_design/dev_ddoc1/_view/view1?stale=false&key=%22Alaska%22&connection_timeout=60000&limit=10&skip=0
 

If I didn't look at the url generated in 2.5 I'm not sure if I would've got this to work.

 Comments   
Comment by Volker Mische [ 28/Aug/14 ]
Assigning to "UI" as this is not a view engine issue but a UI one. Also assigning it to Alk.
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
I need somebody to decide if it's 3.0.0 or 3.0.1.
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
at this point if it isn't related to data loss, we won't take it.
could this be causing apps to break? potentially turn into a hotfix if we don't fix it now?
-cihan
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
3.0.1 then




[MB-12084] Create 3.0.0 chef-based rightscale template for EE and CE Created: 27/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: cloud
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Major
Reporter: Anil Kumar Assignee: Wei-Li Liu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Need this before 3.0 GA




[MB-12083] Create 3.0.0 legacy rightscale templates for Enterprise and Community Edition (non-chef) Created: 27/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: cloud
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Major
Reporter: Anil Kumar Assignee: Wei-Li Liu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We need this before 3.0 GA




[MB-12082] Marketplace AMI - Enterprise Edition and Community Edition - provide AMI id to PM Created: 27/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: cloud
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Major
Reporter: Anil Kumar Assignee: Wei-Li Liu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Need AMI's before 3.0.0 GA




[MB-12081] Remove counting mutations introduced for MB-11589 Created: 27/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Sriram Melkote Assignee: Volker Mische
Resolution: Fixed Votes: 0
Labels: RC2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to MB-11589 Sliding endseqno during initial index... Open
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
We are currently counting number of mutations requested vs received. This was diagnostic code used to get closer to resolving MB-11589. However, that bug has been deferred to 3.0.1 - the diagnostic code must be removed, to avoid performance and logging overhead in release build.

http://review.couchbase.org/40790

 Comments   
Comment by Wayne Siu [ 27/Aug/14 ]
Reviewed with PM/Cihan. Approved for RC2.
Comment by Cihan Biyikoglu [ 27/Aug/14 ]
approved for RC2.




[MB-12080] unable to build cbq-engine Created: 27/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Task Priority: Test Blocker
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
output is:
[root@grape-001 query]# ./build.sh
cd parser/n1ql
nex...
goyacc...
go build...
cd server/main
go build -o cbq-engine...
# github.com/couchbaselabs/query/accounting/logger_retriever
../../accounting/logger_retriever/logger_retriever.go:73: undefined: logger.LogLevel
cd shell
go build -o cbq...
cd tutorial
go build

shell is built fine, but cbq-engine is not built




[MB-12079] Cannot edit documents with textual or numeric data. Created: 27/Aug/14  Updated: 28/Aug/14

Status: Reopened
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0-Beta
Fix Version/s: None
Security Level: Public

Type: Task Priority: Minor
Reporter: Brett Lawson Assignee: Sriram Melkote
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When attempting to create or modify a document in the Web UI, and that value is a string or a number, errors occur preventing you from saving the document.

 Comments   
Comment by Matt Ingenthron [ 27/Aug/14 ]
Related is MB-12078. Looks like the validator we're using in the console is wrong.
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
It is not "validator is wrong". It's deliberate choice to refuse editing such values.
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
See MB-9208
Comment by Matt Ingenthron [ 27/Aug/14 ]
It looks like things have changed for the better and the console is a bit out of sync. Now views do work with numbers such as "1" which may be incr/decr'd too. So, what is said in MB-9208 isn't valid any more. Brett verified this.

At this stage you can insert "1234", incr it, but you can't insert it through the console because it says "JSON should represent an object": http://puu.sh/b9H8i.png

Do you see a downside to syncing up the console to what views/view-engine actually do these days?

Comment by Aleksey Kondratenko [ 27/Aug/14 ]
Have you tried arrays ?
Comment by Matt Ingenthron [ 28/Aug/14 ]
I have not, but you certainly could. The use case we were looking to was the number with incr/decr. Strings should also be handled. From code inspection, it looks like the current JSON parser will handle these things correctly.
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
Parsers ? Or parser?
Comment by Matt Ingenthron [ 28/Aug/14 ]
To my knowledge, as of 3.0 the memcached process does the "isJson()" identification and then in the view engine, that'll control whether meta.type is document or base64. So, I was referring to that parser in 3.0. Sorry for the lack of clarity. There could be other issues at the view-engine level or at the console level, but at least for numbers it seems to do what users want it to do at the view-engine, memcached, SDK level. The console doesn't allow insert/edit even though 1234 is technically JSON.
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
3.0's ep-engine to my surprise indeed does full json detection (unlike 2.5 which AFAIK only allowed objects).

But AFAIK 3.0.0's view engine _does not_ use datatype inferred by memcached. Same is true for some obscure CAPI XDCR paths which need json-ness. For xdcr _I know_ that detection happens via couchdb routines. I'm sure there's no other way for views to do json detection.

BTW this is one of areas where some higher level entity could finally decide something. And let me say that I'm not eager to let my team work on this until there's clear resolution on json detection (from tag or from ep-engine folks or from anybody else).

Comment by Matt Ingenthron [ 28/Aug/14 ]
Siri, can you clarify? From one of our previous conversations I was under the impression that over DCP, view-engine is trusting the JSON detection done in the memcached process. Is that accurate that as of 3.0,
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
Haven't we made decision to disable data type in 3.0?
Comment by Matt Ingenthron [ 28/Aug/14 ]
Alk: agreed this should be a higher level thing. We ended up having to disable datatype at memcached HELLO because if it had gone in and a mixed version cluster had failed over, logical corruption would have occurred. That's one of the areas that probably should have had better review.

That said, I'm just in the "what do we do given the current state" kind of mode. The other changes are in and good-- I don't see any reason we'd avoid making the Web UI changes in a future release.




[MB-12078] JSON detection may not be correct at views Created: 27/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.5.1, 3.0.1, 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Matt Ingenthron Assignee: Sriram Melkote
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Brett Lawson and I were reviewing something for encoding for client libraries and in the process we checked what JSON specs consider legal. There's this interesting bit:

http://tools.ietf.org/html/rfc7159#section-2

   JSON text is a sequence of tokens. The set of tokens includes six
   structural characters, strings, numbers, and three literal names.

   A JSON text is a serialized value. Note that certain previous
   specifications of JSON constrained a JSON text to be an object or an
   array. Implementations that generate only objects or arrays where a
   JSON text is called for will be interoperable in the sense that all
   implementations will accept these as conforming JSON texts.

This actually means that the numeric values stored by memcached protocol are valid JSON, though the view engine doesn't treat them that way. I believe they're detected as non-JSON at view engine. I'm not sure if this is still the case with 3.0, but I thought I should file this since the revelation that a sequence of digits is valid JSON may trigger some thoughts (or unit tests).

 Comments   
Comment by Brett Lawson [ 27/Aug/14 ]
View-engine actually handles this properly, although the document editor does not. A ticket has been opened on that instead.




[MB-12077] Add 'Update/Fetch' button to UI docs query window. Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Venu Uppalapati Assignee: Anil Kumar
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2014-08-26 at 3.56.27 PM.png    
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
refer to attached screenshot.
Steps to reproduce:
1)On Bucket->Documents screen click the Documents filter to open the range query window
2)There is a reset button to clear user input and a close button that closes the range query and fetches the docs.
3)This is un-intuitive to close the window to fetch the docs. If the user wants try a new range, the window has to be closed and opened every time.
4)There should be an Update/Fetch button that allows user to get results without closing window.




[MB-12076] Internal moxi misconfiguration Created: 22/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: moxi
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: kay Assignee: Sergey Avseyev
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos 6.5

Attachments: Text File normal.log     Text File problem.log    
Triage: Untriaged
Operating System: Centos 64-bit
Is this a Regression?: Unknown

 Description   
I have 4 servers cluster. Four buckets. One of them is default couchbase bucket with replica=1.

On one server moxi behavior is very strange. Third server's moxi lives its own live.
I've telneted to 11211 moxi's port and tried to set test keys. This key apperared only on that server, not on whole cluster. Also couchbase monitoring tool doesn't show any activity on cluster.

I've noticed that problem moxi process listens only three TCP port's:
{code}
netstat -nlpt | grep 30070
tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 30070/moxi
tcp 0 0 :::11211 :::* LISTEN 30070/moxi
tcp 0 0 :::6696 :::* LISTEN 30070/moxi
{code}

Other servers' moxies have four listen ports:
{code}
netstat -nltp | grep 2577
tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 2577/moxi
tcp 0 0 0.0.0.0:60593 0.0.0.0:* LISTEN 2577/moxi
tcp 0 0 :::11211 :::* LISTEN 2577/moxi
tcp 0 0 :::18347 :::* LISTEN 2577/moxi

netstat -nlpt | grep 23001
tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 23001/moxi
tcp 0 0 0.0.0.0:11339 0.0.0.0:* LISTEN 23001/moxi
tcp 0 0 :::11211 :::* LISTEN 23001/moxi
tcp 0 0 :::5191 :::* LISTEN 23001/moxi

netstat -nlpt | grep 31535
tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 31535/moxi
tcp 0 0 0.0.0.0:33578 0.0.0.0:* LISTEN 31535/moxi
tcp 0 0 :::11211 :::* LISTEN 31535/moxi
tcp 0 0 :::53475 :::* LISTEN 31535/moxi
{code}

So it seems that moxi on problem server was not able to listen one TCP port.

I've attached debug logs for two servers: problem and normal server.

The problem process is still running. Please let me know which logs do you need for further problem investigation.

 Comments   
Comment by kay [ 22/Aug/14 ]
I use couchbase-server-2.5.1-1083.x86_64
Comment by kay [ 22/Aug/14 ]
please change subproject to moxi for this issue




[MB-12075] Couchbase Server constantly shutting down and restarting after upgrade from 2.1.1-766 Created: 26/Aug/14  Updated: 28/Aug/14  Resolved: 26/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Aruna Piravi Assignee: Sangharsh Agarwal
Resolution: Fixed Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Ubuntu 64 bit, 3.0.0-1174-rel

Issue Links:
Gantt: start-finish
is triggering MB-12065 [Offline Upgrade 2.1.1-766 -> 3.0.0-1... Resolved
is triggering MB-12066 [Offline Upgrade 2.1.1.766 -> 3.0.0-1... Resolved
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Pls look at http://10.3.3.240:8091/index.html#sec=servers, node 10.3.3.239 in particular.

The node is unstable, Couchbase server is restarting every 4 secs as you can see from the GUI log.

--> This happened after the cluster was upgraded from 2.1.1.766 to 3.0.0-1174.
--> After offline upgrade, source cluster was forced to use encrypted xdcr.
--> No replication seems to have happened after this change and node .239 entered a bad state.
--> Seen only on Ubuntu
--> Is probably the reason for MB-12066(yes, there is data loss - 1000 items less on each bucket on C2).

Cbcollect
-------------

[C1]
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12066/d30b70e7/10.3.3.239-8252014-2346-couch.tar.gz
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12066/4b079374/10.3.3.240-8252014-2340-diag.zip

[C2]
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12066/e4f38323/10.3.3.218-8252014-2343-diag.zip
10.3.3.225 : https://s3.amazonaws.com/bugdb/jira/MB-12066/6f010183/10.3.3.225-8252014-2345-diag.zip


 Comments   
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
Indeed happens because of ns_server bug. And also indeed causes xdcr to malfunction.
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
I've fixed the thing for live cluster already. Will post fix in few hours.
Comment by Aruna Piravi [ 26/Aug/14 ]
Thanks, I can now see all keys replicated. Will close 12066 as duplicate of this bug.
Comment by Cihan Biyikoglu [ 26/Aug/14 ]
approved for RC2 if we can get the fix by wed EOD
thanks
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
manifest bumped here: http://review.couchbase.org/40954

actual fixes are:

* http://review.couchbase.org/40953
* http://review.couchbase.org/40949

Comment by Aruna Piravi [ 27/Aug/14 ]
Alk, I see 140 lines of code changed/added. Do you foresee any risks of regression at this point?
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
No. The second patch is large because I had to refactor remote_cluster_info facility a bit. I tested it thoroughly, plus it's quite deterministic piece of code. If we have any regressions they will be caught as part of usual xdcr testing.




[MB-12074] {Windows}:: Rebalance-in hangs Created: 26/Aug/14  Updated: 28/Aug/14  Resolved: 26/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0.1, 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Parag Agarwal Assignee: Sriram Ganesan
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1191, windows 2008, 10.1.2.107, 10.1.2.108

Triage: Untriaged
Operating System: Windows 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/bugdb/jira/MB-12074/windows_1191_logs.tar.gz
Is this a Regression?: Yes

 Description   
1191

1. Create 1 node cluster
2. Create default bucket and add 60K items
3. Rebalance-in 1 node

Rebalance hangs


 Comments   
Comment by Parag Agarwal [ 26/Aug/14 ]
cluster is live:: http://10.1.2.107:8091
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
Let me first note that my job could be easier if you stop adding logs of unrelated nodes. At least 2 nodes in this collection are not part of cluster and cannot be relevant for this bug.
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
Rebalance is waiting for seqno persistence which is apparently stuck.
Comment by Sriram Ganesan [ 26/Aug/14 ]
From the logs, it seems to be the same persistence problem as observed MB-11948 and MB-11955. Windows fixes are going into the master branch of ep-engine. So, please test/verify windows in 3.0.1 builds when they are available.
Comment by Parag Agarwal [ 28/Aug/14 ]
Works now, thanks for fixing it




[MB-12073] cbq-engine crashes when use negative values in ARRAY_REPEAT Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
query:
cbq> select ARRAY_REPEAT(2, -2);
{
    "results": [cbq>

10:22:51.785959 cbq-engine started...
10:22:51.786116 version: 0.7.0
10:22:51.786127 datastore: dir:/tmp/data
panic: runtime error: makeslice: len out of range

goroutine 277 [running]:
github.com/couchbaselabs/query/expression.(*ArrayRepeat).eval(0xc200627d60, 0xc20016b420, 0x4000000000000000, 0xc20016b420, 0xc000000000000000, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/func_array.go:1087 +0x169
github.com/couchbaselabs/query/expression.(*binaryBase).evaluate(0xc200627d60, 0xc20017fa10, 0xc200627d60, 0xc20016bd10, 0xc200695040, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/binary.go:42 +0x175
github.com/couchbaselabs/query/expression.(*ArrayRepeat).Evaluate(0xc200627d60, 0xc20016bd10, 0xc200695040, 0xc20062c7b0, 0xc200654380, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/func_array.go:1051 +0x78
github.com/couchbaselabs/query/execution.(*InitialProject).processTerms(0xc200654200, 0xc20016a700, 0xc200695040, 0xc200654380, 0xc2006a30d8, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:100 +0x397
github.com/couchbaselabs/query/execution.(*InitialProject).processItem(0xc200654200, 0xc20016a700, 0xc200695040, 0xc200654380, 0x801, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:84 +0x4b1
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:142 +0x25a
sync.(*Once).Do(0xc200654250, 0x2b7ba0a77f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654200, 0xc20062d680, 0xc200654200, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200654200, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 1 [IO wait]:
net.runtime_pollWait(0x2b7b9ff73f00, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc200180230, 0xb, 0xc200136f90)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).accept(0xc2001801b0, 0x9bc7c8, 0x0, 0xc200136f90, 0xb, ...)
/usr/local/go/src/pkg/net/fd_unix.go:385 +0x2c1
net.(*TCPListener).AcceptTCP(0xc2000002d0, 0x562186, 0x2b7b9f301d00, 0x562186)
/usr/local/go/src/pkg/net/tcpsock_posix.go:229 +0x45
net.(*TCPListener).Accept(0xc2000002d0, 0xc20016c300, 0xc200000040, 0xc2001802d0, 0x0, ...)
/usr/local/go/src/pkg/net/tcpsock_posix.go:239 +0x25
net/http.(*Server).Serve(0xc20016c2b0, 0xc20015be40, 0xc2000002d0, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/server.go:1542 +0x85
net/http.(*Server).ListenAndServe(0xc20016c2b0, 0x0, 0xc200136360)
/usr/local/go/src/pkg/net/http/server.go:1532 +0x9e
github.com/couchbaselabs/query/server/http.(*HttpEndpoint).ListenAndServe(0xc20016c2a0, 0x80b6c0, 0xcfebd8)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_endpoint.go:36 +0x2a
main.main()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/main/main.go:61 +0x547

goroutine 2 [syscall]:

goroutine 261 [chan receive]:
github.com/couchbaselabs/query/server/http.(*HttpEndpoint).ServeHTTP(0xc20016c2a0, 0xc20015b480, 0xc20017f700, 0xc20015c750)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_endpoint.go:46 +0xb3
net/http.serverHandler.ServeHTTP(0xc20016c2b0, 0xc20015b480, 0xc20017f700, 0xc20015c750)
/usr/local/go/src/pkg/net/http/server.go:1517 +0x16c
net/http.(*conn).serve(0xc2001802d0)
/usr/local/go/src/pkg/net/http/server.go:1096 +0x765
created by net/http.(*Server).Serve
/usr/local/go/src/pkg/net/http/server.go:1564 +0x266

goroutine 5 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc20013baf0, 0x2b7ba0975ea0)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc20013baa0, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
github.com/couchbaselabs/query/server.(*Server).serviceRequest(0xc20016c1e0, 0xc20013bbe0, 0xc2005b3b00)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:127 +0x412
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:80 +0x6c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 6 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 7 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 8 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 9 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 10 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 11 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 12 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 13 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 14 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 15 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 16 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 17 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 18 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 19 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 20 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 21 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 22 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 23 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 24 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 25 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 26 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 27 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 28 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 29 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 30 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 31 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 32 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 33 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 34 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 35 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 36 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 37 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 38 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 39 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 40 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 41 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 42 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 43 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 44 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 45 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 46 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 47 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 48 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 49 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 50 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 51 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 52 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 53 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 54 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 55 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 56 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 57 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 58 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 59 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 60 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 61 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 62 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 63 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 64 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 65 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 66 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 67 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 68 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 69 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 70 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 71 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 72 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 73 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 74 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 75 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 76 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 77 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 78 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 79 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 80 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 81 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 82 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 83 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 84 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 85 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 86 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 87 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 88 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 89 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 90 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 91 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 92 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 93 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 94 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 95 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 96 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 97 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 98 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 99 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 100 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 101 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 102 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 103 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 104 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 105 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 106 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 107 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 108 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 109 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 110 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 111 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 112 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 113 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 114 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 115 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 116 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 117 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 118 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 119 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 120 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 121 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 122 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 123 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 124 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 125 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 126 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 127 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 128 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 129 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 130 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 131 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 132 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 133 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 134 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 135 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 136 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 137 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 138 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 139 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 140 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 141 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 142 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 143 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 144 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 145 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 146 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 147 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 148 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 149 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 150 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 151 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 152 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 153 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 154 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 155 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 156 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 157 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 158 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 159 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 160 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 161 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 162 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 163 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 164 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 165 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 166 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 167 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 168 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 169 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 170 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 171 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 172 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 173 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 174 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 175 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 176 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 177 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 178 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 179 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 180 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 181 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 182 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 183 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 184 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 185 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 186 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 187 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 188 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 189 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 190 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 191 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 192 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 193 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 194 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 195 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 196 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 197 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 198 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 199 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 200 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 201 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 202 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 203 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 204 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 205 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 206 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 207 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 208 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 209 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 210 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 211 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 212 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 213 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 214 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 215 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 216 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 217 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 218 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 219 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 220 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 221 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 222 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 223 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 224 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 225 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 226 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 227 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 228 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 229 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 230 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 231 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 232 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 233 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 234 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 235 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 236 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 237 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 238 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 239 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 240 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 241 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 242 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 243 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 244 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 245 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 246 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 247 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 248 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 249 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 250 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 251 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 252 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 253 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 254 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 255 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 256 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 257 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 258 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 259 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 260 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 262 [IO wait]:
net.runtime_pollWait(0x2b7b9ff73e60, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2001802c0, 0xb, 0xc200136f90)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc200180240, 0xc200605000, 0x8000, 0x8000, 0x0, ...)
/usr/local/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc200000040, 0xc200605000, 0x8000, 0x8000, 0x8000, ...)
/usr/local/go/src/pkg/net/net.go:123 +0xc3
io.Copy(0xc2005b5390, 0xc200000318, 0xc2005b5090, 0xc200000040, 0x0, ...)
/usr/local/go/src/pkg/io/io.go:348 +0x1c6
net/http.func·004()
/usr/local/go/src/pkg/net/http/server.go:162 +0x66
created by net/http.(*conn).closeNotify
/usr/local/go/src/pkg/net/http/server.go:168 +0x1c6

goroutine 263 [chan receive]:
github.com/couchbaselabs/query/server/http.func·001()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:93 +0x39
created by github.com/couchbaselabs/query/server/http.newHttpRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:95 +0x587

goroutine 265 [select]:
github.com/couchbaselabs/query/server/http.(*httpRequest).writeResults(0xc2005b3b00, 0x1)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_response.go:68 +0x155
github.com/couchbaselabs/query/server/http.(*httpRequest).Execute(0xc2005b3b00, 0xc20016cde0, 0xc200180501)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_response.go:43 +0xa7
created by github.com/couchbaselabs/query/server.(*Server).serviceRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:123 +0x337

goroutine 266 [select]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc200654350, 0x2b7ba0a82f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654300, 0xc20062d5c0, 0xc200654300, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*Stream).RunOnce(0xc200654300, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/stream.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 267 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc20013bf50, 0x2b7ba0a81f70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc20013bf00, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 268 [select]:
github.com/couchbaselabs/query/execution.func·010()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:78 +0x3fc
sync.(*Once).Do(0xc2001805f0, 0x2b7ba0a80f70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Parallel).RunOnce(0xc2001805a0, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:87 +0x6c
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 269 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc20013beb0, 0x2b7ba0a7ff70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc20013be60, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 270 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc20013bb90, 0x2b7ba0a7ef70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc20013bb40, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 271 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc200694050, 0x2b7ba0a7df70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200694000, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 272 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc2006940f0, 0x2b7ba0a7cf70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc2006940a0, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 273 [runnable]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2006542d0, 0x2b7ba0a7bf30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654280, 0xc20062d640, 0xc200654280, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200654280, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 274 [runnable]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2006544d0, 0x2b7ba0a7af30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654480, 0xc20062d640, 0xc200654480, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200654480, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 275 [runnable]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2006545d0, 0x2b7ba0a79f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654580, 0xc20062d640, 0xc200654580, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200654580, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 276 [runnable]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2006546d0, 0x2b7ba0a78f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654680, 0xc20062d640, 0xc200654680, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200654680, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 278 [runnable]:
sync.runtime_Semacquire(0xc200654254)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc200654250)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc200654250, 0x2b7ba0a76f30)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654200, 0xc20062d680, 0xc200654200, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200654200, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 279 [semacquire]:
sync.runtime_Semacquire(0xc200654254)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc200654250)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc200654250, 0x2b7ba0a85f30)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654200, 0xc20062d680, 0xc200654200, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200654200, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 280 [semacquire]:
sync.runtime_Semacquire(0xc200654254)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc200654250)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc200654250, 0x2b7ba0a94f30)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654200, 0xc20062d680, 0xc200654200, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200654200, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc




[MB-12072] error message for array_remove can be more informative Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
when I pass only one argument instead of 2 to array_remove I see a runtime error, perhaps it has to give me an error like 'parse error' or something like that?

cbq> select array_remove([1,2,3]) ;
PANIC: runtime error: index out of range.cbq>




[MB-12071] cbq-engine crashes when use negative numbers in array_range Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
run the command, it hangs:
cbq> select array_range(-20, 1, 2);
{
    "results": [

and then cbq-engine crashes

09:28:42.547108 cbq-engine started...
09:28:42.547249 version: 0.7.0
09:28:42.547261 datastore: dir:/tmp/data
fatal error: runtime: out of memory

goroutine 315 [running]:
[fp=0x2b1d69e84818] runtime.throw(0xcf8937)
/usr/local/go/src/pkg/runtime/panic.c:473 +0x67
[fp=0x2b1d69e84848] runtime.SysMap(0xc37eda0000, 0x85920000)
/usr/local/go/src/pkg/runtime/mem_linux.c:121 +0xec
[fp=0x2b1d69e84888] runtime.MHeap_SysAlloc(0x2b1d586b4000, 0x85920000)
/usr/local/go/src/pkg/runtime/zmalloc_linux_amd64.c:360 +0x142
[fp=0x2b1d69e848c8] MHeap_Grow(0x2b1d586b4000, 0x85920)
/usr/local/go/src/pkg/runtime/mheap.c:224 +0x55
[fp=0x2b1d69e84908] MHeap_AllocLocked(0x2b1d586b4000, 0x85914, 0x0)
/usr/local/go/src/pkg/runtime/mheap.c:108 +0x2fc
[fp=0x2b1d69e84930] runtime.MHeap_Alloc(0x2b1d586b4000, 0x85914, 0x100000000, 0x1)
/usr/local/go/src/pkg/runtime/mheap.c:77 +0x5e
[fp=0x2b1d69e84988] runtime.mallocgc(0x85913ae0, 0x100000000, 0xc200000001)
/usr/local/go/src/pkg/runtime/zmalloc_linux_amd64.c:58 +0x2fa
[fp=0x2b1d69e849c8] cnew(0x7455a0, 0x85913ae, 0x1)
/usr/local/go/src/pkg/runtime/zmalloc_linux_amd64.c:655 +0xee
[fp=0x2b1d69e849e8] runtime.cnewarray(0x7455a0, 0x85913ae)
/usr/local/go/src/pkg/runtime/zmalloc_linux_amd64.c:673 +0x38
[fp=0x2b1d69e84a00] makeslice1(0x739460, 0x6ada958, 0x85913ae, 0x2b1d69e84aa0)
/usr/local/go/src/pkg/runtime/slice.c:53 +0x45
[fp=0x2b1d69e84a30] growslice1(0x739460, 0xc313f27000, 0x6ada958, 0x6ada958, 0x6ada959, ...)
/usr/local/go/src/pkg/runtime/slice.c:212 +0x56
[fp=0x2b1d69e84a78] runtime.growslice(0x739460, 0xc313f27000, 0x6ada958, 0x6ada958, 0x1, ...)
/usr/local/go/src/pkg/runtime/slice.c:179 +0x9f
[fp=0x2b1d69e84b58] github.com/couchbaselabs/query/expression.(*ArrayRange).eval(0xc2007e07c0, 0xc2007e2960, 0x3, 0x3, 0xc200815180, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/func_array.go:950 +0x44b
[fp=0x2b1d69e84be0] github.com/couchbaselabs/query/expression.(*nAryBase).evaluate(0xc2007e07c0, 0xc20017fa10, 0xc2007e07c0, 0xc20016bd10, 0xc2007e0a40, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/n_ary.go:39 +0x19e
[fp=0x2b1d69e84c40] github.com/couchbaselabs/query/expression.(*ArrayRange).Evaluate(0xc2007e07c0, 0xc20016bd10, 0xc2007e0a40, 0xc20062b7b0, 0xc200815180, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/func_array.go:894 +0x78
[fp=0x2b1d69e84e20] github.com/couchbaselabs/query/execution.(*InitialProject).processTerms(0xc200815000, 0xc20016a700, 0xc2007e0a40, 0xc200815180, 0xc2006a06d8, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:100 +0x397
[fp=0x2b1d69e84e88] github.com/couchbaselabs/query/execution.(*InitialProject).processItem(0xc200815000, 0xc20016a700, 0xc2007e0a40, 0xc200815180, 0xd0b401, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:84 +0x4b1
[fp=0x2b1d69e84f08] github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:142 +0x25a
[fp=0x2b1d69e84f20] sync.(*Once).Do(0xc200815050, 0x2b1d69e84f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
[fp=0x2b1d69e84f60] github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815000, 0xc20062c640, 0xc200815000, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
[fp=0x2b1d69e84f98] github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200815000, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
[fp=0x2b1d69e84fa0] runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1223
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 1 [IO wait]:
net.runtime_pollWait(0x2b1d69381f00, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc200180230, 0xb, 0xc200136f90)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).accept(0xc2001801b0, 0x9bc7c8, 0x0, 0xc200136f90, 0xb, ...)
/usr/local/go/src/pkg/net/fd_unix.go:385 +0x2c1
net.(*TCPListener).AcceptTCP(0xc2000002d0, 0x562186, 0x2b1d6870fd00, 0x562186)
/usr/local/go/src/pkg/net/tcpsock_posix.go:229 +0x45
net.(*TCPListener).Accept(0xc2000002d0, 0xc20016c300, 0xc200000040, 0xc2001802d0, 0x0, ...)
/usr/local/go/src/pkg/net/tcpsock_posix.go:239 +0x25
net/http.(*Server).Serve(0xc20016c2b0, 0xc20015be40, 0xc2000002d0, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/server.go:1542 +0x85
net/http.(*Server).ListenAndServe(0xc20016c2b0, 0x0, 0xc200136360)
/usr/local/go/src/pkg/net/http/server.go:1532 +0x9e
github.com/couchbaselabs/query/server/http.(*HttpEndpoint).ListenAndServe(0xc20016c2a0, 0x80b6c0, 0xcfebd8)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_endpoint.go:36 +0x2a
main.main()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/main/main.go:61 +0x547

goroutine 2 [syscall]:

goroutine 261 [chan receive]:
github.com/couchbaselabs/query/server/http.(*HttpEndpoint).ServeHTTP(0xc20016c2a0, 0xc20015b480, 0xc20077a000, 0xc20015cc30)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_endpoint.go:46 +0xb3
net/http.serverHandler.ServeHTTP(0xc20016c2b0, 0xc20015b480, 0xc20077a000, 0xc20015cc30)
/usr/local/go/src/pkg/net/http/server.go:1517 +0x16c
net/http.(*conn).serve(0xc2001802d0)
/usr/local/go/src/pkg/net/http/server.go:1096 +0x765
created by net/http.(*Server).Serve
/usr/local/go/src/pkg/net/http/server.go:1564 +0x266

goroutine 5 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 6 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 7 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc200691910, 0x2b1d69d91ea0)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc2006918c0, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
github.com/couchbaselabs/query/server.(*Server).serviceRequest(0xc20016c1e0, 0xc20013bbe0, 0xc2007c5000)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:127 +0x412
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:80 +0x6c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 8 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 9 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 10 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 11 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 12 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 13 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 14 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 15 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 16 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 17 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 18 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 19 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 20 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 21 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 22 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 23 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 24 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 25 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 26 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 27 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 28 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 29 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 30 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 31 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 32 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 33 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 34 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 35 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 36 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 37 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 38 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 39 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 40 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 41 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 42 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 43 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 44 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 45 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 46 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 47 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 48 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 49 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 50 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 51 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 52 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 53 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 54 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 55 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 56 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 57 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 58 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 59 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 60 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 61 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 62 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 63 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 64 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 65 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 66 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 67 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 68 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 69 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 70 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 71 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 72 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 73 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 74 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 75 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 76 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 77 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 78 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 79 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 80 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 81 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 82 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 83 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 84 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 85 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 86 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 87 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 88 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 89 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 90 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 91 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 92 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 93 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 94 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 95 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 96 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 97 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 98 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 99 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 100 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 101 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 102 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 103 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 104 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 105 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 106 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 107 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 108 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 109 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 110 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 111 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 112 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 113 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 114 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 115 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 116 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 117 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 118 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 119 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 120 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 121 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 122 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 123 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 124 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 125 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 126 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 127 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 128 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 129 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 130 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 131 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 132 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 133 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 134 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 135 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 136 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 137 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 138 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 139 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 140 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 141 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 142 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 143 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 144 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 145 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 146 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 147 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 148 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 149 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 150 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 151 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 152 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 153 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 154 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 155 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 156 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 157 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 158 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 159 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 160 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 161 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 162 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 163 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 164 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 165 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 166 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 167 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 168 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 169 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 170 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 171 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 172 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 173 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 174 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 175 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 176 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 177 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 178 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 179 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 180 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 181 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 182 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 183 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 184 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 185 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 186 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 187 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 188 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 189 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 190 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 191 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 192 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 193 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 194 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 195 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 196 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 197 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 198 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 199 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 200 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 201 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 202 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 203 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 204 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 205 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 206 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 207 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 208 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 209 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 210 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 211 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 212 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 213 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 214 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 215 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 216 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 217 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 218 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 219 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 220 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 221 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 222 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 223 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 224 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 225 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 226 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 227 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 228 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 229 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 230 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 231 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 232 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 233 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 234 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 235 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 236 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 237 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 238 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 239 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 240 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 241 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 242 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 243 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 244 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 245 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 246 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 247 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 248 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 249 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 250 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 251 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 252 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 253 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 254 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 255 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 256 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 257 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 258 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 259 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 260 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 262 [IO wait]:
net.runtime_pollWait(0x2b1d69381e60, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2001802c0, 0xb, 0xc200136f90)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc200180240, 0xc200604000, 0x8000, 0x8000, 0x0, ...)
/usr/local/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc200000040, 0xc200604000, 0x8000, 0x8000, 0xb1, ...)
/usr/local/go/src/pkg/net/net.go:123 +0xc3
io.Copy(0xc2005b4390, 0xc200000318, 0xc2005b4090, 0xc200000040, 0x161, ...)
/usr/local/go/src/pkg/io/io.go:348 +0x1c6
net/http.func·004()
/usr/local/go/src/pkg/net/http/server.go:162 +0x66
created by net/http.(*conn).closeNotify
/usr/local/go/src/pkg/net/http/server.go:168 +0x1c6

goroutine 263 [chan receive]:
github.com/couchbaselabs/query/server/http.func·001()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:93 +0x39
created by github.com/couchbaselabs/query/server/http.newHttpRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:95 +0x587

goroutine 305 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc200691870, 0x2b1d69e91f70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200691820, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 282 [chan receive]:
github.com/couchbaselabs/query/server/http.func·001()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:93 +0x39
created by github.com/couchbaselabs/query/server/http.newHttpRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:95 +0x587

goroutine 301 [chan receive]:
github.com/couchbaselabs/query/server/http.func·001()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:93 +0x39
created by github.com/couchbaselabs/query/server/http.newHttpRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:95 +0x587

goroutine 306 [select]:
github.com/couchbaselabs/query/execution.func·010()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:78 +0x3fc
sync.(*Once).Do(0xc200180950, 0x2b1d69e8ef70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Parallel).RunOnce(0xc200180900, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:87 +0x6c
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 316 [semacquire]:
sync.runtime_Semacquire(0xc200815054)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc200815050)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc200815050, 0x2b1d69e8df30)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815000, 0xc20062c640, 0xc200815000, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200815000, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 308 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc2006919b0, 0x2b1d69e8cf70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200691960, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 309 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc200691a50, 0x2b1d69e8bf70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200691a00, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 310 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc200691af0, 0x2b1d69e8af70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200691aa0, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 312 [select]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2008152d0, 0x2b1d69e89f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815280, 0xc20062c600, 0xc200815280, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200815280, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 317 [semacquire]:
sync.runtime_Semacquire(0xc200815054)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc200815050)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc200815050, 0x2b1d69e88f30)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815000, 0xc20062c640, 0xc200815000, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200815000, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 313 [select]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2008153d0, 0x2b1d69e87f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815380, 0xc20062c600, 0xc200815380, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200815380, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 314 [select]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2008154d0, 0x2b1d69e86f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815480, 0xc20062c600, 0xc200815480, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200815480, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 307 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc2006917d0, 0x2b1d69e85f70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200691780, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 311 [select]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2008150d0, 0x2b1d69e93f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815080, 0xc20062c600, 0xc200815080, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200815080, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 303 [select]:
github.com/couchbaselabs/query/server/http.(*httpRequest).writeResults(0xc2007c5000, 0x1)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_response.go:68 +0x155
github.com/couchbaselabs/query/server/http.(*httpRequest).Execute(0xc2007c5000, 0xc200828060, 0xc200180501)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_response.go:43 +0xa7
created by github.com/couchbaselabs/query/server.(*Server).serviceRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:123 +0x337

goroutine 304 [select]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc200815150, 0x2b1d69ea1f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815100, 0xc20062c580, 0xc200815100, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*Stream).RunOnce(0xc200815100, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/stream.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 318 [semacquire]:
sync.runtime_Semacquire(0xc200815054)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc200815050)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc200815050, 0x2b1d69ea0f30)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815000, 0xc20062c640, 0xc200815000, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200815000, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc




[MB-12070] Append command should return the size of the document Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: memcached
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Major
Reporter: Patrick Varley Assignee: Trond Norbye
Resolution: Unresolved Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
Relates to
relates to MB-9817 Meta.size property Open

 Description   
It would be useful if the append command returned the current size of the object so a user know how big it is without the need to get it.




[MB-12069] [windows] Dependency management Created: 26/Aug/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sriram Melkote Assignee: Chris Hillery
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
The windows 3rd party binary dependencies should be moved from Google Drive to Amazon S3 or some other suitable location that allows access without a special client.

It will be good if the depot is versioned. If not, we should at least have a directory structure that mirrors the manifest file names of our various builds (i.e, rel-3.0.0)




[MB-12068] XDCR - only minority of items get replicated Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: 2.2.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Marek Obuchowicz Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Cluster "US" - 3x AWS m3.medium instances
Cluster "EU" - 1x AWS m3.medium instance
All running amzn-ami-pv-2014.03.1.x86_64-ebs
Cochbase server (latest community) installed from couchbase-server-2.2.0-837.x86_64.rpm

Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/couchbase_debug/eu.zip
https://s3.amazonaws.com/couchbase_debug/us.zip
Is this a Regression?: Unknown

 Description   
I have two clusters:
 - "US" with three nodes - there is one bucket, profiles (with 2 replicas)
 - "EU" with one node - there is one bucket, profiles_debug (without replicase)

I have setup XDCR from "US" cluster to "EU" cluster. Number of documents in "US" cluster is 2.35m (+ 4.7m replicas), but number of replicated documents to "EU" cluster is only 780k - exactly 1/3 of all documents. When I create new buckets in destination cluster and setup XDCR again, the situation doesn't change - new bucket after a while gets 1/3 of documents and no more.

In web console -> XDCR Replication on US cluster, status shows "Replicating - Last 10 errors". Last 10 errors state:
{code}
2014-08-26 09:39:14 [Vb Rep] Error replicating vbucket 765. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 649. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 593. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 567. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 552. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 521. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 502. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 454. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 439. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 417. Please see logs for details.
{code}

I have been using XDCR (bi-directional) in the past, when both US and EU clusters had only one node (no replicas within clusters). At that time it was working fine. Problems started, when I've added two nodes to US cluster, setup new bucket with 2 replicas and tried to use XDCR on this bucket.

List of steps we did, in chronological order:
 - create cluster US, 1 node
 - create cluster EU, 1 node
 - setup bucket A, no replicas, on both clusters
 - setup bi-directional XDCR A(US) <-> A(EU), default settings
 - add two nodes to US cluster
 - setup bucket B, 2 replicas, on cluster US
 - terminate XDCR A(US) <-> A(EU)
 - setup XDCR of bucket A(US) to bucket B(US), terminate it when all documents are copied.
 - delete bucket A(EU)
 - create bucket B(EU), 0 replicas
 - setup bi-directional XDCR B(US) <-> B(EU)
 - only 1/3 of documents from B(US) got replicated to B(EU)

Attached cbcollectInfo output from both clusters. As suggested by pfehre, created Jira issue. Please get in touch by email to get direct access to servers if it helps with investigation.

 Comments   
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
I'll need logs from all nodes.




Investigate other possible memory allocators that provide the better fragmentation management (MB-10496)

[MB-12067] Investigate explicit defragmentation of ep_engine buckets Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1, 3.0
Fix Version/s: techdebt-backlog
Security Level: Public

Type: Technical task Priority: Major
Reporter: Dave Rigby Assignee: Dave Rigby
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2014-08-26 at 17.59.09.png    

 Description   
As discussed in the parent (MB-10496) task, and in https://docs.google.com/document/d/1sgE9LFfT5ZD4FbSZqCuUtzOoLu5BFi1Kf63R9kYuAyY/edit# it has been demonstrated that there is significant memory usage overhead (over mem_used) due to the chunk-based memory allocation used by malloc - see attached graph.

We should investigate reducing this overhead by performing explicit defragmentation of ep_engine's in-memory hash table.




[MB-12066] [Offline Upgrade 2.1.1.766 -> 3.0.0-1174] items mismatch after upgrade Created: 26/Aug/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: cross-datacenter-replication, ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sangharsh Agarwal Assignee: Sangharsh Agarwal
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Ubuntu 12.04

Attachments: PNG File Screen Shot 2014-08-26 at 12.50.39 PM.png     PNG File Screen Shot 2014-08-26 at 12.54.36 PM.png    
Issue Links:
Gantt: start-finish
is triggered by MB-12075 Couchbase Server constantly shutting ... Resolved
Triage: Untriaged
Operating System: Ubuntu 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: [Source]
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12066/d30b70e7/10.3.3.239-8252014-2346-couch.tar.gz
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12066/e20435ef/10.3.3.239-8252014-2341-diag.zip
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12066/4b079374/10.3.3.240-8252014-2340-diag.zip
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12066/792d226b/10.3.3.240-8252014-2346-couch.tar.gz

[Destination]
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12066/d9c64bf9/10.3.3.218-8252014-2346-couch.tar.gz
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12066/e4f38323/10.3.3.218-8252014-2343-diag.zip
10.3.3.225 : https://s3.amazonaws.com/bugdb/jira/MB-12066/6f010183/10.3.3.225-8252014-2345-diag.zip
10.3.3.225 : https://s3.amazonaws.com/bugdb/jira/MB-12066/e5adac05/10.3.3.225-8252014-2346-couch.tar.gz
Is this a Regression?: Yes

 Description   
[Test Logs]
https://friendpaste.com/VPVWHuLxOZpWUx20pP37o

[Test Steps]
1. Create 2-2 Nodes Source and Destination Cluster.

Source Nodes 10.3.3.240 (Master), 10.3.3.239
Source Nodes 10.3.3.218 (Master), 10.3.3.225


2. Setup CAPI Mode XDCR

bucket: bucket0 (BiXDCR)
bucket: default (UniXDCR)

3. Load 1000 items on bucket0 bucket on the both the cluster.
4. Load 1000 items on default bucket on Source cluster.
5. Offline upgrade both the cluster with 3.0.0-1174-rel.
6. Modify XDCR Settings to use SSL on Source Cluster Only*****.
7. Load 1000 Items on bucket0 and default bucket on Source.
8. Verify items. Items mismatch on Destination Cluster. Items loaded after upgrade ( in Step-7) is not replicated to destination cluster.


[Test Error]
2014-08-25 23:40:08,126 - root - WARNING - Not Ready: curr_items 2000 == 3000 expected on '10.3.3.218:8091''10.3.3.225:8091', bucket0 bucket
2014-08-25 23:40:09,149 - root - WARNING - Not Ready: vb_active_curr_items 2000 == 3000 expected on '10.3.3.218:8091''10.3.3.225:8091', bucket0 bucket
2014-08-25 23:40:09,169 - root - WARNING - Not Ready: vb_replica_curr_items 2000 == 3000 expected on '10.3.3.218:8091''10.3.3.225:8091', bucket0 bucket

2014-08-25 23:40:10,189 - root - WARNING - Not Ready: curr_items 1000 == 2000 expected on '10.3.3.218:8091''10.3.3.225:8091', default bucket
2014-08-25 23:40:11,209 - root - WARNING - Not Ready: vb_active_curr_items 1000 == 2000 expected on '10.3.3.218:8091''10.3.3.225:8091', default bucket
2014-08-25 23:40:11,228 - root - WARNING - Not Ready: vb_replica_curr_items 1000 == 2000 expected on '10.3.3.218:8091''10.3.3.225:8091', default bucket


Issue occurred on Ubuntu VM.



 Comments   
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Cluster is Live for Investigation.

Source: 10.3.3.239, 10.3.3.240
Destination: 10.3.3.218, 10.3.3.225:
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Checked xdcr and info logs on 10.3.3.240 and founds lots of crashes:

[error_logger:error,2014-08-25T23:30:04.135,ns_1@10.3.3.240:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
  crasher:
    initial call: remote_clusters_info:-spawn_request_remote_bucket/5-fun-0-/0
    pid: <0.11425.0>
    registered_name: []
    exception error: no match of right hand side value false
      in function remote_clusters_info:do_with_mcd_to_couch_uri_dict/5 (src/remote_clusters_info.erl, line 1358)
      in call from remote_clusters_info:'-spawn_request_remote_bucket/5-fun-0-'/7 (src/remote_clusters_info.erl, line 702)
    ancestors: [remote_clusters_info,ns_server_sup,ns_server_cluster_sup,
                  <0.60.0>]
    messages: []
    links: [<0.324.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 28690
    stack_size: 27
    reductions: 118182
  neighbours:

[error_logger:error,2014-08-25T23:30:04.149,ns_1@10.3.3.240:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.11415.0> terminating
** Last message in was init
** When Server state == [{data,
                          [{"State",
                            {init_state,
                             {rep,
                              <<"30c42877cc3130e0225e567a98aacadb/bucket0/bucket0">>,
                              <<"bucket0">>,
                              <<"/remoteClusters/30c42877cc3130e0225e567a98aacadb/buckets/bucket0">>,
                              "capi",
                              [{max_concurrent_reps,16},
                               {checkpoint_interval,1800},
                               {doc_batch_size_kb,2048},
                               {failure_restart_interval,30},
                               {worker_batch_size,500},
                               {connection_timeout,180},
                               {worker_processes,4},
                               {http_connections,20},
                               {retries_per_request,2},
                               {optimistic_replication_threshold,256},
                               {socket_options,
                                [{keepalive,true},{nodelay,false}]},
                               {pause_requested,false},
                               {supervisor_max_r,25},
                               {supervisor_max_t,5}]},
                             1004,"capi",<0.1056.0>,<0.1057.0>,<0.1054.0>}}]}]
** Reason for termination ==
** {{badmatch,false},
    {gen_server,call,
        [remote_clusters_info,
         {get_remote_bucket,
             [{uuid,<<"30c42877cc3130e0225e567a98aacadb">>},
              {cert,
                  <<"-----BEGIN CERTIFICATE-----\nMIIC/jCCAeigAwIBAgIIE43leIvyau0wCwYJKoZIhvcNAQEFMCQxIjAgBgNVBAMT\nGUNvdWNoYmFzZSBTZXJ2ZXIgNTk2MmNkMDMwHhcNMTMwMTAxMDAwMDAwWhcNNDkx\nMjMxMjM1OTU5WjAkMSIwIAYDVQQDExlDb3VjaGJhc2UgU2VydmVyIDU5NjJjZDAz\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAn2C66HXyJDgz37O8N/Fq\nFFmrkknnIlxkvMKv4X1goGYKrx7uxhtAkCgzenISUb57IlzHx+rIvJ2DcbjQsDJo\n/F+cfEf7tYnRc/5mGw1r8sKHPQpZWahTc5KD1JFM7RxR+3vb3/n0AHvwN+6m6/w0\nRWFnrJrGlUwZ8t0Szod2ECAsr35k5lVxuiv7LfCTFS3+kpYy/eVRMZPaU9l23Nra\nYyAJEJCZ12RqRwYifZpmjjVRw32a9aGnSXJ5Ygiy3135FV0O47K48+lt1RG8lLjZ\nu33LUQf7cqn2wuplMUBhk2XZUxvUl/lF0GBUvm5PvtmR+r0H+YVGstUD1L8Q1VH8\nBwIDAQABozgwNjAOBgNVHQ8BAf8EBAMCAKQwEwYDVR0lBAwwCgYIKwYBBQUHAwEw\nDwYDVR0TAQH/BAUwAwEB/zALBgkqhkiG9w0BAQUDggEBAJQt+68WEL5tskVHrYgL\nGFcCEIlf29yrtzXHDIprcMhlNE6GiPCaV6BSgA7WH0ni4A9cLJzRfdUWUuLhT4iJ\nV04/qAVyQlYRuzrPC24znG2cfgJERkgdrePILRiCtInAHThfeFOJ8Z9jxjTdBCVh\nPrmsUYYXraN4YDVnDLyQZp/KIYYa5i/yZ7JGUL2mGpWItNs5hi0XJDTAtPCWU0EP\nV3beJqU8YZYrC3cE51A0je+92Bi2uAcmDmuboTXJQtT4Q8Y6WWdhCkkF8Wl1Ljo4\nfNosl8V6ss+AM5h63ntVZgPWkm5JOihvCGGMB91Ab2bpc01gWrBxxBj9oA/lJf/+\na7Q=\n-----END CERTIFICATE-----\n">>},
              {name,"cluster1"},
              {hostname,"10.3.3.218:8091"},
              {username,"Administrator"},
              {password,"password"}],
             "bucket0",false,30000},
         infinity]}}

Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Additionally, not able to open Admin Console on http://10.3.3.239:8091/index.html. Screen is attached.
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Crashes on 10.3.3.218:

[error_logger:error,2014-08-25T23:32:01.284,ns_1@10.3.3.218:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
  crasher:
    initial call: mochiweb_acceptor:init/3
    pid: <0.11931.0>
    registered_name: []
    exception error: no match of right hand side value {error,closed}
      in function mochiweb_http:request/2 (/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/mochiweb/mochiweb_http.erl, line 54)
    ancestors: [menelaus_web_ssl,ns_ssl_services_sup,menelaus_sup,
                  ns_server_sup,ns_server_cluster_sup,<0.60.0>]
    messages: [{ssl_closed,
                      {sslsocket,
                          {gen_tcp,#Port<0.8895>,tls_connection},
                          <0.12060.0>}}]
    links: [<0.3569.0>]
    dictionary: [{random_seed,{1410,4416,29898}}]
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 16778
  neighbours:
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
Do not have sufficient evidence for xdcr mismatch bug.
Comment by Aruna Piravi [ 26/Aug/14 ]
I understand keys are missing but the evidence as you can see from the GUI is this

C1
bucket0 3000
default 2000

C2
bucket0 2000
default 1000

You can see that the 1000 keys inserted after upgrade have not been replicated. Most likely due to MB-12075 that has caused master node in C1 to shutdown and restart every few secs. Pls check.
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
I'm going to bounce _every_ xdcr ticket that doesn't include required information.
Comment by Aruna Piravi [ 26/Aug/14 ]
1000 keys LoadTwo0-LoadTwo999 have not been replicated. We had discussed yesterday to include all the required artifacts. It's possible that since the nodes were unstable, Sangharsh could not retrieve the keys.
Comment by Aruna Piravi [ 26/Aug/14 ]
Fixing 12075 caused replication to complete without data loss. Marking this as duplicate of MB-12075.




[MB-12065] [Offline Upgrade 2.1.1-766 -> 3.0.0-1174] replica items mismatch after upgrade Created: 26/Aug/14  Updated: 27/Aug/14  Resolved: 26/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sangharsh Agarwal Assignee: Abhinav Dangeti
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Ubuntu 64 bit (12.04)

Issue Links:
Gantt: start-finish
is triggered by MB-12075 Couchbase Server constantly shutting ... Resolved
Triage: Untriaged
Operating System: Ubuntu 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: [Source]
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12065/40f0a9b2/10.3.3.239-8222014-2216-diag.zip
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12065/7584aa98/10.3.3.239-8222014-2222-couch.tar.gz
10.3.3.239 : https://s3.amazonaws.com/bugdb/jira/MB-12065/a15d63ee/10.3.3.239-diag.txt
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12065/802987e6/10.3.3.240-8222014-2222-couch.tar.gz
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12065/c555bf6e/10.3.3.240-diag.txt
10.3.3.240 : https://s3.amazonaws.com/bugdb/jira/MB-12065/c7c31c83/10.3.3.240-8222014-2214-diag.zip


[Destination]
10.3.3.199 : https://s3.amazonaws.com/bugdb/jira/MB-12065/85abaa8a/10.3.3.199-8222014-2220-diag.zip
10.3.3.199 : https://s3.amazonaws.com/bugdb/jira/MB-12065/99f3fb96/10.3.3.199-diag.txt.gz
10.3.3.199 : https://s3.amazonaws.com/bugdb/jira/MB-12065/c11ea857/10.3.3.199-8222014-2222-couch.tar.gz
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12065/27bf3a49/10.3.3.218-8222014-2219-diag.zip
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12065/402915cc/10.3.3.218-8222014-2222-couch.tar.gz
10.3.3.218 : https://s3.amazonaws.com/bugdb/jira/MB-12065/851c2f72/10.3.3.218-diag.txt.gz
Is this a Regression?: Unknown

 Description   
http://qa.hq.northscale.net/job/ubuntu_x64--36_01--XDCR_upgrade-P1/36/consoleFull

[Test]
./testrunner -i ubuntu_x64--36_01--XDCR_upgrade-P1.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,upgrade_version=3.0.0-1174-rel,initial_vbuckets=1024 -t xdcr.upgradeXDCR.UpgradeTests.offline_cluster_upgrade,initial_version=2.1.1-766-rel,sdata=False,bucket_topology=default:1>2;bucket0:1><2,upgrade_nodes=dest;src,use_encryption_after_upgrade=src


[Test Steps]
1. Create 2-2 Nodes Source and Destination Cluster.

Source Nodes 10.3.3.240 (Master), 10.3.3.239
Source Nodes 10.3.3.218 (Master), 10.3.3.199


2. Setup CAPI Mode XDCR

bucket: bucket0 (BiXDCR)
bucket: default (UniXDCR)

3. Load 1000 items on bucket0 bucket on the both the cluster.
4. Load 1000 items on default bucket on Source cluster.
5. Offline upgrade both the cluster with 3.0.0-1174-rel.
6. Modify XDCR Settings to use SSL on Source Cluster Only*****.
7. Load 1000 Items on bucket0 and default bucket on Source.
8. Verify items.

Replica items mismatch on Source Cluster itself.

[2014-08-22 22:11:14,388] - [task:459] WARNING - Not Ready: vb_replica_curr_items 1500 == 2000 expected on '10.3.3.240:8091''10.3.3.239:8091', default bucket
[2014-08-22 22:11:16,410] - [task:459] WARNING - Not Ready: vb_replica_curr_items 2500 == 3000 expected on '10.3.3.240:8091''10.3.3.239:8091', bucket0 bucket
[2014-08-22 22:11:19,432] - [task:459] WARNING - Not Ready: vb_replica_curr_items 1500 == 2000 expected on '10.3.3.240:8091''10.3.3.239:8091', default bucket





 Comments   
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Some updates from the test Logs:

1. After upgrade, we see the errors that test were not able to connect with 10.3.3.239:

[2014-08-22 22:08:44,291] - [data_helper:295] INFO - creating direct client 10.3.3.239:11210 bucket0
[2014-08-22 22:08:44,440] - [data_helper:295] INFO - creating direct client 10.3.3.240:11210 bucket0
[2014-08-22 22:08:45,351] - [task:772] INFO - Batch create documents done #: 0 with exp:0
[2014-08-22 22:08:45,691] - [task:772] INFO - Batch create documents done #: 1000 with exp:0
[2014-08-22 22:08:46,955] - [data_helper:295] INFO - creating direct client 10.3.3.239:11210 default
[2014-08-22 22:08:46,970] - [rest_client:750] ERROR - socket error while connecting to http://10.3.3.239:8091/pools/default/buckets/default?basic_stats=true error [Errno 111] Connection refused
[2014-08-22 22:08:47,974] - [rest_client:750] ERROR - socket error while connecting to http://10.3.3.239:8091/pools/default/buckets/default?basic_stats=true error [Errno 111] Connection refused
[2014-08-22 22:08:49,172] - [data_helper:295] INFO - creating direct client 10.3.3.240:11210 default
[2014-08-22 22:08:49,356] - [task:772] INFO - Batch create documents done #: 0 with exp:0
[2014-08-22 22:08:49,753] - [task:772] INFO - Batch create documents done #: 1000 with exp:0


2. Test is passed on CentOS too.
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
There were lot of logs, where buckets are going to shutdown here:

2014-08-22 22:08:30.085 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:30.085 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "default" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:32.731 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:32.862 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "default" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:32.915 mb_master:0:warning:message(ns_1@10.3.3.239) - Somebody thinks we're master. Not forcing mastership takover over ourselves
2014-08-22 22:08:32.962 menelaus_sup:1:info:web start ok(ns_1@10.3.3.239) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:08:32.997 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "default" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:33.035 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:35.795 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:35.925 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "default" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:35.992 mb_master:0:warning:message(ns_1@10.3.3.239) - Somebody thinks we're master. Not forcing mastership takover over ourselves
2014-08-22 22:08:36.033 menelaus_sup:1:info:web start ok(ns_1@10.3.3.239) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:08:36.069 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "default" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:36.073 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:08:39.216 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:39.292 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "default" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:08:39.368 mb_master:0:warning:message(ns_1@10.3.3.239) - Somebody thinks we're master. Not forcing mastership takover over ourselves
2014-08-22 22:08:39.404 menelaus_sup:1:info:web start ok(ns_1@10.3.3.239) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".



[ns_server.info.log]

[user:info,2014-08-22T22:08:00.713,ns_1@10.3.3.239:ns_memcached-default<0.12208.0>:ns_memcached:terminate:783]Shutting down bucket "default" on 'ns_1@10.3.3.239' for server shutdown
[ns_server:info,2014-08-22T22:08:00.769,ns_1@10.3.3.239:ns_memcached-default<0.12208.0>:ns_memcached:terminate:795]This bucket shutdown is not due to bucket deletion or reconfiguration. Doing nothing
[ns_server:info,2014-08-22T22:08:01.801,ns_1@10.3.3.239:ns_server_sup<0.13612.0>:dir_size:start_link:49]Starting quick version of dir_size with program name: i386-linux-godu
[ns_server:info,2014-08-22T22:08:01.816,ns_1@10.3.3.239:ns_config_rep<0.13631.0>:ns_config_rep:do_pull:343]Pulling config from: 'ns_1@10.3.3.240'

[user:warn,2014-08-22T22:08:01.838,ns_1@10.3.3.239:ns_server_sup<0.13612.0>:mb_master:check_master_takeover_needed:153]Somebody thinks we're master. Not forcing mastership takover over ourselves
[ns_server:info,2014-08-22T22:08:01.864,ns_1@10.3.3.239:ns_doctor<0.13655.0>:ns_doctor:update_status:235]The following buckets became ready on node 'ns_1@10.3.3.240': ["bucket0",
                                                               "default"]
[user:info,2014-08-22T22:08:01.879,ns_1@10.3.3.239:ns_server_sup<0.13612.0>:menelaus_sup:start_link:44]Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".
[ns_server:info,2014-08-22T22:08:01.880,ns_1@10.3.3.239:<0.13739.0>:mc_tcp_listener:init:24]mccouch is listening on port 11213
[ns_server:info,2014-08-22T22:08:01.882,ns_1@10.3.3.239:<0.13743.0>:ns_memcached_log_rotator:init:28]Starting log rotator on "/opt/couchbase/var/lib/couchbase/logs"/"memcached.log"* with an initial period of 39003ms
[ns_server:info,2014-08-22T22:08:01.900,ns_1@10.3.3.239:<0.13807.0>:compaction_new_daemon:spawn_scheduled_kv_compactor:468]Start compaction of vbuckets for bucket default with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2014-08-22T22:08:01.903,ns_1@10.3.3.239:janitor_agent-default<0.13805.0>:janitor_agent:read_flush_counter:1048]Loading flushseq failed: {error,enoent}. Assuming it's equal to global config.
[ns_server:info,2014-08-22T22:08:01.904,ns_1@10.3.3.239:ns_memcached-default<0.13784.0>:ns_memcached:handle_cast:675]Main ns_memcached connection established: {ok,#Port<0.9250>}
[ns_server:info,2014-08-22T22:08:01.904,ns_1@10.3.3.239:janitor_agent-bucket0<0.13811.0>:janitor_agent:read_flush_counter:1048]Loading flushseq failed: {error,enoent}. Assuming it's equal to global config.

Comment by Sangharsh Agarwal [ 26/Aug/14 ]
Trying to reproduce this issue to give you live cluster.
Comment by Sangharsh Agarwal [ 26/Aug/14 ]
There is one more issued logged MB-12066. Not sure of this issue is similar to MB-12066 as in this bug only replica items are mismatch.
Comment by Abhinav Dangeti [ 26/Aug/14 ]
I see from the diags of 10.3.3.240, that the node is continuously restarting after upgrade:

2014-08-22 22:14:52.811 menelaus_sup:1:info:web start ok(ns_1@10.3.3.239) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:14:52.860 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "default" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:14:52.885 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:14:54.853 ns_memcached:0:info:message(ns_1@10.3.3.240) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.240' for server shutdown
2014-08-22 22:14:55.256 ns_memcached:0:info:message(ns_1@10.3.3.240) - Shutting down bucket "default" on 'ns_1@10.3.3.240' for server shutdown
2014-08-22 22:14:56.166 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:14:56.452 menelaus_sup:1:info:web start ok(ns_1@10.3.3.240) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.240'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:14:56.509 ns_memcached:0:info:message(ns_1@10.3.3.240) - Bucket "default" loaded on node 'ns_1@10.3.3.240' in 0 seconds.
2014-08-22 22:14:56.510 ns_memcached:0:info:message(ns_1@10.3.3.240) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.240' in 0 seconds.
2014-08-22 22:14:56.538 ns_memcached:0:info:message(ns_1@10.3.3.239) - Shutting down bucket "default" on 'ns_1@10.3.3.239' for server shutdown
2014-08-22 22:14:57.705 menelaus_sup:1:info:web start ok(ns_1@10.3.3.239) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.239'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:14:57.782 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "bucket0" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:14:57.819 ns_memcached:0:info:message(ns_1@10.3.3.239) - Bucket "default" loaded on node 'ns_1@10.3.3.239' in 0 seconds.
2014-08-22 22:15:00.409 ns_memcached:0:info:message(ns_1@10.3.3.240) - Shutting down bucket "bucket0" on 'ns_1@10.3.3.240' for server shutdown
2014-08-22 22:15:00.693 ns_memcached:0:info:message(ns_1@10.3.3.240) - Shutting down bucket "default" on 'ns_1@10.3.3.240' for server shutdown
2014-08-22 22:15:00.917 menelaus_sup:1:info:web start ok(ns_1@10.3.3.240) - Couchbase Server has started on web port 8091 on node 'ns_1@10.3.3.240'. Version: "3.0.0-1174-rel-enterprise".
2014-08-22 22:15:00.972 ns_memcached:0:info:message(ns_1@10.3.3.240) - Bucket "default" loaded on node 'ns_1@10.3.3.240' in 0 seconds.

This can cause replicators to break, so I'm going to mark this as a duplicate of MB-12075.




[MB-12064] document the enterprise edition only features in documentation Created: 25/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Cihan Biyikoglu Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
documentation does not have a list of enterprise edition only features.
We should include a list of features that only work in enterprise edition here:
http://docs.couchbase.com/couchbase-manual-2.5/index.html

features are - rack/zone awareness and XDCR SSL communication.
thanks





[MB-12063] KV+XDCR System test : Between expiration and purging, getMeta() retrieves revID as 1 for deleted key from Source, same deleted key from Destination returns 2. Created: 25/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0.1, 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Aruna Piravi Assignee: Mike Wiederhold
Resolution: Unresolved Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: CentOS 6.x , build 3.0.0-1174-rel

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
--> Before expiration and after purging, all metadata match between source and destination clusters.
--> However, after expiration, there's some code that causes a deleted key at the source(!!!) to have seqno as 1. The seqno at destination is however 2, as expected.
--> The data below for uni-xdcr (C1 -> C2).
--> Not a recent regression, seen it once before in 3.0.0-9xx. Catching this bug totally depends on when I run the validation script after system test is completed. Expiration is usually set to 1 day and tombstone purge interval is 3 days on both source and destination. Once tombstones are purged, I don't see this mismatch. So I don't have a live cluster.

{'C1_location:': u'172.23.105.44', 'vb': 90, 'C2_node': u'172.23.105.54', 'C1_key_count': 19919, 'C2_key_count': 19919, 'missing_keys': 0}
RevID or CAS mismatch -
  172.23.105.44(C1): key:65ABEE18-153_100061 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902841111553483, 'flags': 0, 'expiration': 1408646731}
  172.23.105.54(C2): key:65ABEE18-153_100061 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902841111553484, 'flags': 0, 'expiration': 1408646731}
 RevID or CAS mismatch -
  172.23.105.44(C1): key:65ABEE18-153_100683 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902841111336520, 'flags': 0, 'expiration': 1408646731}
  172.23.105.54(C2): key:65ABEE18-153_100683 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902841111336521, 'flags': 0, 'expiration': 1408646731}
RevID or CAS mismatch -
  172.23.105.44(C1): key:65ABEE18-153_100713 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902841111837669, 'flags': 0, 'expiration': 1408646731}
  172.23.105.54(C2): key:65ABEE18-153_100713 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902841111837670, 'flags': 0, 'expiration': 1408646731}
 RevID or CAS mismatch -
  172.23.105.44(C1): key:65ABEE18-153_103240 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902843752129235, 'flags': 0, 'expiration': 1408646733}
  172.23.105.54(C2): key:65ABEE18-153_103240 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902843752129236, 'flags': 0, 'expiration': 1408646733}
 RevID or CAS mismatch -
  172.23.105.44(C1): key:65ABEE18-153_105170 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902847773405994, 'flags': 0, 'expiration': 1408646737}
  172.23.105.54(C2): key:65ABEE18-153_105170 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902847773405995, 'flags': 0, 'expiration': 1408646737}

Please let me know what/if you need in particular to diagnose this issue. Thanks!




 Comments   
Comment by Aruna Piravi [ 25/Aug/14 ]
Worth mentioning that if the same key gets recreated at Source with seq <= seqno(same key at destination), the create will/may not get propagated.
Comment by Aruna Piravi [ 25/Aug/14 ]
We have many xdcr functional tests with expiration, after which we compare revIDs, even of deleted items. We did not hit this particular bug. Venu did some unit tests and could not catch it either.

I'd like to perform the same system test on 2.5.1 to determine if this is a regression. Again system test runs itself runs for 12-15 hrs, we keep loading throughout the test, items keep expiring until the next 24 hrs, expiry pager runs 3 days from the start of the test and the validation script runs for half a day so it's all about timing.

Will get you the data files and start the system test.

Comment by Aruna Piravi [ 25/Aug/14 ]
http://172.23.105.44:8091/index.html
http://172.23.105.54:8091/index.html
Comment by Aruna Piravi [ 25/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12063/stdbucket.rtf --> keys that have this mismatch for bucket 'standardbucket'.

For a quick look -
https://s3.amazonaws.com/bugdb/jira/MB-12063/44_couch.tar (source)
https://s3.amazonaws.com/bugdb/jira/MB-12063/54_couch.tar (dest)
Comment by Aruna Piravi [ 25/Aug/14 ]
Also seeing some keys where revID is greater at source than dest. Pls look at vbuckets 15 and 28.

RevID or CAS mismatch -
  172.23.105.44(C1): key:6B67A321-142_4666321 metadata:{'deleted': 1, 'seqno': 4, 'cas': 13175771830561215, 'flags': 0, 'expiration': 1408649286}
  172.23.105.54(C2): key:6B67A321-142_4666321 metadata:{'deleted': 1, 'seqno': 3, 'cas': 13175771830561214, 'flags': 0, 'expiration': 1408649286}
RevID or CAS mismatch -
  172.23.105.44(C1): key:6B67A321-142_4666453 metadata:{'deleted': 1, 'seqno': 4, 'cas': 13175771778347790, 'flags': 0, 'expiration': 1408649286}
  172.23.105.54(C2): key:6B67A321-142_4666453 metadata:{'deleted': 1, 'seqno': 3, 'cas': 13175771778347789, 'flags': 0, 'expiration': 1408649286}
 RevID or CAS mismatch -
  172.23.105.44(C1): key:6B67A321-142_4674099 metadata:{'deleted': 1, 'seqno': 4, 'cas': 13175775983769867, 'flags': 0, 'expiration': 1408649290}
  172.23.105.54(C2): key:6B67A321-142_4674099 metadata:{'deleted': 1, 'seqno': 3, 'cas': 13175775983769866, 'flags': 0, 'expiration': 1408649290}
 RevID or CAS mismatch -
  172.23.105.44(C1): key:6B67A321-142_4674109 metadata:{'deleted': 1, 'seqno': 4, 'cas': 13175775977754745, 'flags': 0, 'expiration': 1408649290}
  172.23.105.54(C2): key:6B67A321-142_4674109 metadata:{'deleted': 1, 'seqno': 3, 'cas': 13175775977754744, 'flags': 0, 'expiration': 1408649290}
RevID or CAS mismatch -
  172.23.105.44(C1): key:6B67A321-142_4677328 metadata:{'deleted': 1, 'seqno': 4, 'cas': 13175778211679600, 'flags': 0, 'expiration': 1408649293}
  172.23.105.54(C2): key:6B67A321-142_4677328 metadata:{'deleted': 1, 'seqno': 3, 'cas': 13175778211679599, 'flags': 0, 'expiration': 1408649293}
Comment by Aruna Piravi [ 25/Aug/14 ]
cbcollect :

https://s3.amazonaws.com/bugdb/jira/MB-12063/source.tar
https://s3.amazonaws.com/bugdb/jira/MB-12063/dest.tar

Note: .44 is down now. Was up until this morning. .44 ran out of diskspace. The above script ran 2 days back.

I can start system test on 2.5.1 tomorrow morning if required. Pls let me know, thanks.
Comment by Wayne Siu [ 26/Aug/14 ]
Reviewed with Cihan. Potentially a RC2 candidate.
If the fix is contained and is ready by this week.
Need Dev's risk assessment.
Comment by Mike Wiederhold [ 26/Aug/14 ]
The logs have rolled over so I can't see exactly what happened. I think the reason that the cluster got into this state is because the expiry pager was run on one side of the cluster and not the other. This can easily explain the why destination would have a different sequence number and the source wouldn't. This happens because expiring a key will increment the rev sequence number and unfortunately I don't think there is anything we can do about this.

I am assuming that the same issue is happening when the source clusters seqno is incremented, but the destination seqno is not. In this case we should see the delete replicated to the other side and I cannot determine whether or not this happened because the logs are rolled over. I'm also not sure if you just checked those keys while keys were being propagated to the destination node. All of the traffic could have stopped, but the expiry pager might have kicked in during the verification phase.

I'm seeing a log message that I need to investigate further so I will leave this assigned to me.
Comment by Aruna Piravi [ 26/Aug/14 ]
 > I think the reason that the cluster got into this state is because the expiry pager was run on one side of the cluster and not the other.

That's possible but what can explain seeing 'deleted': 1 with 'seqno': 1? My understanding is - any key with deleted flag = 1 needs to have revid as atleast 2. RevID for any doc can be 1 only at the time of creation. If expired or deleted, revID should be incremented, right?
Comment by Aruna Piravi [ 27/Aug/14 ]
Also, this is C1 -> C2 (uni-xdcr). So seeing something like

172.23.105.44(C1): key:65ABEE18-153_100061 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902841111553483, 'flags': 0, 'expiration': 1408646731}
172.23.105.54(C2): key:65ABEE18-153_100061 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902841111553484, 'flags': 0, 'expiration': 1408646731}

would mean - expiry pager ran at C1('deleted':1) but did not increment revid which is clearly a bug.
Comment by Cihan Biyikoglu [ 27/Aug/14 ]
is there an ETA on the resolution? if this won't resolve in the next day, we need to push this one out.
thanks
Comment by Abhinav Dangeti [ 27/Aug/14 ]
Mike's already submitted the fixes:
http://review.couchbase.org/#/c/40996/
http://review.couchbase.org/#/c/40997/
Comment by Mike Wiederhold [ 28/Aug/14 ]
The above changes do not resolve this issue. When looking through the logs I saw two separate issues which needed to be fixed.
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
need a fix by EOD today so we can consider this for RC2 . otherwise we'll need to delay the RC
Comment by Sundar Sridharan [ 28/Aug/14 ]
Aruna, if possible, could you please list the operations that the test is doing since we need clues to reproduce and verify. thanks
Comment by Aruna Piravi [ 28/Aug/14 ]
Hi Sundar, as we discussed, here are the steps

Clusters
-----------
C1 : http://172.23.105.44:8091/
C2 : http://172.23.105.54:8091/

Steps
--------
1. Setup uni-xdcr on "standardbucket1" , load till active_resident_ratio = ~70 on standardbucket1
2. Access phase with 50% gets, 50%deletes for 3 hrs
3. Rebalance-out 1 node at cluster1
4. Rebalance-in 1 node at cluster1
5. Failover and remove node at cluster1
6. Failover and add-back node at cluster1
7. Rebalance-out 1 node at cluster2
8. Rebalance-in 1 node at cluster2
9. Failover and remove node at cluster2
10. Failover and add-back node at cluster2
11. Soft restart all nodes in cluster1 one by one

Run verification script after 15 hrs(after all items are expired).
Comment by Sundar Sridharan [ 28/Aug/14 ]
Aruna the toy build is couchbase-server-community_cent58-3.0.0-toy-sundar-x86_64_3.0.0-710-toy.rpm
Comment by Aruna Piravi [ 28/Aug/14 ]
System test started on the above toy build.

You can check
C1 : http://172.23.105.44:8091/
C2 : http://172.23.105.54:8091/
if you would like to watch the test.

It should be finished by tomorrow afternoon. If there are crashes, it would stop sooner. I will check back late night and tomorrow morning.
Comment by Aruna Piravi [ 28/Aug/14 ]
Sundar, there seems to a problem with the toy build. The buckets got created but did not get successfully loaded on all nodes in the cluster. Pls take a look at any of the clusters. All nodes in both clusters are in pending state.




[MB-12062] Documentation link for 3.0 build in admin UI goes to 2.5 docs Created: 25/Aug/14  Updated: 28/Aug/14  Resolved: 28/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0-Beta
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Don Pinto Assignee: Aleksey Kondratenko
Resolution: Fixed Votes: 0
Labels: Rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Please see description below.

 Comments   
Comment by Don Pinto [ 28/Aug/14 ]
Don needs to send email to marketing to carve out a link that we can control. Maybe a docs.couchbase.com subpath?
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
we need to clearly isolate the redirect to a URL we can control late in the release.
Lets point to http://docs.couchbase.com/redirect/documentation-3.0.0 in the console and ask for the web link to be a ridect to the top doc page for now. we'll add a checklist item to release to ensure we correct the redirect point before GA to the correct page.
thanks
-cihan
Comment by Matt Ingenthron [ 28/Aug/14 ]
nitpick, but using /redirect/ seems icky to me. it's definitely a nitpick though.

I might recommend http://docs.couchbase.com/release/couchbase-server/3.0.0

we'd reserve the /release/ namespace for these "persistent URLs" and then have the product name under it. It's nice and SEO happy and is a "hackable URL" meaning if a search dumps you on 3.0.0 but you know you want 3.0.1, it's easy to work out how to get there.
Comment by Don Pinto [ 28/Aug/14 ]
Today's URLs for docs have pre-built in them - see
http://docs.couchbase.com/prebuilt/couchbase-manual-3.0/beta-intro.html

I had asked for this to be changed before and it never happened. Having /release redirect to pre-built will be weird. Hence would favor redirect in this format -
http://docs.couchbase.com/redirect/couchbase-server/3.0.0
Comment by Don Pinto [ 28/Aug/14 ]
Hi Alk,

We would need your help to include this in the RC2 build for tomorrow. This is the documentation link in the UI which currently points to 2.5 docs.

Request to change that to point to - http://docs.couchbase.com/redirect/couchbase-server/3.0.0 (This is a redirect URL that we will then re-direct to the final documentation page).

Thank you for your help.

-Don
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
let make sure Amy and Ruth can add this URL on their as well otherwise Alk may need to make more changes. Don could you coordinate with Amy on this one.
thanks
-cihan
Comment by Don Pinto [ 28/Aug/14 ]
Cihan,

The redirects are happening at the domain level through our provider. Ray is controlling that but I'll check with Amy to make sure there are no URL collisions. Attached sub-task CBIT-1576 and Ray is on that.

Thanks,
Comment by Amy Kurtzman [ 28/Aug/14 ]
Hi Alk, Please get with me before you make the change. The URL Don requested above is incorrect. Thanks, Amy
Comment by Don Pinto [ 28/Aug/14 ]
Amy,

As discussed in the 3.0 meeting today (with Ruth), thats the re-directing URL we decided to go with. We do not want to go directly to docs.couchbase.com but to 3.0 specific content and I believe the final 3.0 introduction page URL in docs might change for GA. Thus we decided to make the product go to the redirect URL which could go to the final 3.0 introduction GA doc page when it is finalized.

Please sync up with Cihan if there is any confusion.

Thanks,

Comment by Amy Kurtzman [ 28/Aug/14 ]
I never said to send it to docs.couchbase.com and I understand that it is desired that it does to the 3.0 manual.
Comment by Don Pinto [ 28/Aug/14 ]
OK. Can you please provide the target 3.0 introduction page URL that the /redirect URL should point to?

Thanks,
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
http://review.couchbase.org/41059
Comment by Don Pinto [ 28/Aug/14 ]
Discussed with respective folks here - Alk, Ruth, Amy, and Ray.

We decided to go with a domain level link - http://www.couchbase.com/redirect/couchbase-server-docs/3.0.0 which will be in the UI.

Ray will setup a redirect through Contegix from http://www.couchbase.com/redirect/couchbase-server-docs/3.0.0 -> http://docs.couchbase.com/prebuilt/couchbase-manual-3.0/beta-intro.html

Later on, when we are going to GA 3.0, we will refresh the redirect target to point to the latest link docs team provides us with.

Thank you,
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
http://review.couchbase.org/41066




[MB-12061] Cluster name must have a max length Created: 25/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0-Beta
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Don Pinto Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2014-08-25 at 4.59.05 PM.png    
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
It is good to fix the length of the cluster to between [8 and 20] characters.






[MB-12060] should date_diff_str should difference 1 even if difference is less? Created: 25/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
select date_diff_str("2014-08-24T01:33:59", "2014-08-24T07:33:59", "day") ;
{
    "results": [
        {
            "$1": -1
        }
    ],
    "metrics": {
        elapsedTime: 2.925ms,
        executionTime: 2.444ms,
        resultCount: 1
    }
}

6 hours only of difference
but it shows 1 day, is it ok? i would expect 0 days




[MB-12059] date_add_str fn doesn't show correct result Created: 25/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
select date_add_str(clock_str(), 1, 'hour');
{
    "results": [
        {
            "$1": "240826-08-2426T08:246:20.7248-07:00"
        }
    ],
    "metrics": {
        elapsedTime: 2.899ms,
        executionTime: 2.535ms,
        resultCount: 1
    }
}
it shows date 240826-08-2426 which is impossible also 246 minutes doesn't seems to be right




[MB-12058] ip:18091 should redirect on https://ip:18091 Created: 24/Aug/14  Updated: 25/Aug/14  Resolved: 24/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Andrei Baranouski Assignee: Aleksey Kondratenko
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1974

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
unable to open url like 10.3.121.134:18091/index.html( or http://10.3.121.134:18091/index.html)


it's better to redirect on https because now loading 'ip:18091' stuck

 Comments   
Comment by Aleksey Kondratenko [ 24/Aug/14 ]
Unfortunately I'm not aware of anybody able to do that. You really have to specify https.
Comment by Andrei Baranouski [ 25/Aug/14 ]
I see, but the problem is that the request( ip:18091) hangs (we do not get any timeout error or 'page unavailable') and it is wrong
Comment by Aleksey Kondratenko [ 25/Aug/14 ]
you'll get this with _any_ https thing accessed as http




[MB-12057] apparent deadlock in ep-engine/bucket-engine (was: node_in is in pending state/ unable to restart cb service there/Rebalance exited with reason {not_all_nodes_are_ready_yet ) Created: 24/Aug/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Andrei Baranouski
Resolution: Fixed Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1174

Triage: Triaged
Is this a Regression?: Unknown

 Description   
steps:
1)run data load ~12hours on source cluster http://172.23.105.156/
2) then start replication for all 4 buckets on destination nodes(172.23.105.159, 172.23.105.160, 172.23.105.206)
3) almost immediately after step#2 add 172.23.105.207 to destination cluster and rebalance


Rebalance exited with reason {not_all_nodes_are_ready_yet,
['ns_1@172.23.105.207']}
ns_orchestrator002 ns_1@172.23.105.159 10:49:49 - Sun Aug 24, 2014
Started rebalancing bucket UserInfo ns_rebalancer000 ns_1@172.23.105.159 10:48:49 - Sun Aug 24, 2014
Starting rebalance, KeepNodes = ['ns_1@172.23.105.159','ns_1@172.23.105.160',
'ns_1@172.23.105.206','ns_1@172.23.105.207'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
ns_orchestrator004 ns_1@172.23.105.159 10:48:49 - Sun Aug 24, 2014
Control connection to memcached on 'ns_1@172.23.105.207' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
cmd_vocal_recv,
5,
[{file,
"src/mc_client_binary.erl"},
{line,
151}]},
{mc_client_binary,
select_bucket,
2,
[{file,
"src/mc_client_binary.erl"},
{line,
346}]},
{ns_memcached,
ensure_bucket,
2,
[{file,
"src/ns_memcached.erl"},
{line,
1269}]},
{ns_memcached,
handle_info,
2,
[{file,
"src/ns_memcached.erl"},
{line,
744}]},
{gen_server,
handle_msg,
5,
[{file,
"gen_server.erl"},
{line,
604}]},
{ns_memcached,
init,
1,
[{file,
"src/ns_memcached.erl"},
{line,
171}]},
{gen_server,
init_it,
6,
[{file,
"gen_server.erl"},
{line,
304}]},
{proc_lib,
init_p_do_apply,
3,
[{file,
"proc_lib.erl"},
{line,
239}]}]} (repeated 1 times) ns_memcached000 ns_1@172.23.105.207 10:44:42 - Sun Aug 24, 2014
Control connection to memcached on 'ns_1@172.23.105.207' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
cmd_vocal_recv,
5,
[{file,
"src/mc_client_binary.erl"},
{line,
151}]},
{mc_client_binary,
select_bucket,
2,
[{file,
"src/mc_client_binary.erl"},
{line,
346}]},
{ns_memcached,
ensure_bucket,
2,
[{file,
"src/ns_memcached.erl"},
{line,
1269}]},
{ns_memcached,
handle_info,
2,
[{file,
"src/ns_memcached.erl"},
{line,
744}]},
{gen_server,
handle_msg,
5,
[{file,
"gen_server.erl"},
{line,
604}]},
{ns_memcached,
init,
1,
[{file,
"src/ns_memcached.erl"},
{line,
171}]},
{gen_server,
init_it,
6,
[{file,
"gen_server.erl"},
{line,
304}]},
{proc_lib,
init_p_do_apply,
3,
[{file,
"proc_lib.erl"},
{line,
239}]}]} ns_memcached000 ns_1@172.23.105.207 10:43:56 - Sun Aug 24, 2014
Rebalance exited with reason {not_all_nodes_are_ready_yet,
['ns_1@172.23.105.207']}


trying to restart 172.23.105.207 node(firewall is turn off there):

[root@centos-64-x64 logs]# /etc/init.d/couchbase-server status
couchbase-server is running
[root@centos-64-x64 logs]# /etc/init.d/couchbase-server restart
Stopping couchbase-server
^C
BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution
a
                                                           [ OK ]
Starting couchbase-server [ OK ]
[root@centos-64-x64 logs]# /etc/init.d/couchbase-server status
couchbase-server is running
[root@centos-64-x64 logs]# /etc/init.d/couchbase-server stop
Stopping couchbase-serverNOTE: shutdown failed
{badrpc,nodedown}
                                                           [FAILED]
[root@centos-64-x64 logs]# /etc/init.d/couchbase-server start
couchbase-server is already started [WARNING]


cluster will be available a few hours
 

 Comments   
Comment by Andrei Baranouski [ 24/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12057/de39e575/172.23.105.159-8242014-1113-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12057/de39e575/172.23.105.160-8242014-1116-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12057/de39e575/172.23.105.206-8242014-1119-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12057/de39e575/172.23.105.207-8242014-1122-diag.zip
Comment by Aleksey Kondratenko [ 24/Aug/14 ]
Grabbing backtraces from memcached process on bad node might be very handy.
Comment by Andrei Baranouski [ 24/Aug/14 ]
[root@centos-64-x64 ~]# ps -ef| grep couch
root 3795 3469 0 12:17 pts/2 00:00:00 grep couch
498 27065 1 0 Aug20 ? 00:00:14 /opt/couchbase/lib/erlang/erts-5.10.4/bin/epmd -daemon
498 27102 1 0 Aug20 ? 00:01:05 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /opt/couchbase -- -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@127.0.0.1 -setcookie nocookie -noshell -noinput -noshell -noinput -run ns_babysitter_bootstrap -- -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" -ns_server pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie-ns-server" -ns_server enable_mlockall false
498 27136 27102 3 Aug20 ? 02:52:05 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 -- -root /opt/couchbase/lib/erlang -progname erl -- -home /opt/couchbase -- -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -nouser -run child_erlang child_start ns_bootstrap -- -smp enable -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
498 27171 27136 0 Aug20 ? 00:00:15 /opt/couchbase/lib/erlang/lib/os_mon-2.2.14/priv/bin/memsup
498 27172 27136 0 Aug20 ? 00:00:00 /opt/couchbase/lib/erlang/lib/os_mon-2.2.14/priv/bin/cpu_sup
498 27228 27102 0 Aug20 ? 00:17:42 /opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
498 31489 27136 0 10:36 ? 00:00:15 /opt/couchbase/lib/ns_server/erlang/lib/ns_server/priv/i386-linux-godu
[root@centos-64-x64 ~]# gdb -p 27102
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-64.el6_5.2)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 27102
Reading symbols from /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp...done.
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 27130]
[New LWP 27129]
[New LWP 27128]
[New LWP 27127]
[New LWP 27126]
[New LWP 27125]
[New LWP 27124]
[New LWP 27123]
[New LWP 27122]
[New LWP 27121]
[New LWP 27120]
[New LWP 27119]
[New LWP 27118]
[New LWP 27117]
[New LWP 27116]
[New LWP 27115]
[New LWP 27114]
[New LWP 27113]
[New LWP 27112]
[New LWP 27111]
[New LWP 27110]
[New LWP 27109]
[New LWP 27108]
[New LWP 27107]
[New LWP 27106]
[New LWP 27105]
[New LWP 27104]
[New LWP 27103]
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
0x00007f4f4c9614f3 in select () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install couchbase-server-3.0.0-1174.x86_64
(gdb) thread app all bt

Thread 29 (Thread 0x7f4f4af7f700 (LWP 27103)):
#0 0x00007f4f4ce2954d in read () from /lib64/libpthread.so.0
#1 0x0000000000550641 in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2916
#2 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178974f0) at pthread/ethread.c:106
#3 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f4f4a0ff700 (LWP 27104)):
#0 0x00007f4f4ce2643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005a7699 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1368
#2 0x0000000000463e3f in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:1788
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:938
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3286
#5 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975d0) at pthread/ethread.c:106
#6 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f4f4a57e700 (LWP 27105)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80138) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80138) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4bec9c40) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4bec9c40) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4bec9c40) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f4f496fe700 (LWP 27106)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80178) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80178) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4bec9d80) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4bec9d80) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4bec9d80) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f4f496dc700 (LWP 27107)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af801b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af801b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4bec9ec0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4bec9ec0) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4bec9ec0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f4f496ba700 (LWP 27108)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af801f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af801f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca000) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca000) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca000) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f4f49698700 (LWP 27109)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80238) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80238) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca140) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca140) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca140) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f4f49676700 (LWP 27110)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80278) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80278) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca280) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca280) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca280) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f4f49654700 (LWP 27111)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af802b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af802b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca3c0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca3c0) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca3c0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f4f49632700 (LWP 27112)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af802f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af802f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca500) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca500) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca500) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 19 (Thread 0x7f4f49610700 (LWP 27113)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80338) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80338) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca640) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca640) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca640) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f4f495ee700 (LWP 27114)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80378) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80378) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca780) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca780) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca780) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f4f495cc700 (LWP 27115)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af803b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af803b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4beca8c0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4beca8c0) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4beca8c0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f4f495aa700 (LWP 27116)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af803f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af803f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4becaa00) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4becaa00) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4becaa00) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f4f49588700 (LWP 27117)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80438) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80438) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4becab40) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4becab40) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4becab40) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f4f49566700 (LWP 27118)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80478) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80478) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4becac80) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4becac80) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4becac80) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f4f49544700 (LWP 27119)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af804b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af804b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4becadc0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4becadc0) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4becadc0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f4f49522700 (LWP 27120)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af804f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af804f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f4f4becaf00) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f4f4becaf00) at beam/erl_async.c:371
#5 async_main (arg=0x7f4f4becaf00) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff178975b0) at pthread/ethread.c:106
#7 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f4f4d913700 (LWP 27121)):
#0 0x00007f4f4ce2a09d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000054f158 in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2840
#2 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897520) at pthread/ethread.c:106
#3 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f4f48ebf700 (LWP 27122)):
#0 0x00007f4f4c95f253 in poll () from /lib64/libc.so.6
#1 0x000000000055a971 in check_fd_events (ps=0x7f4f4bdd4910, pr=0x7f4f48ebe320, len=0x7f4f48ebeb3c, utvp=<value optimized out>) at sys/common/erl_poll.c:2071
#2 erts_poll_wait_nkp (ps=0x7f4f4bdd4910, pr=0x7f4f48ebe320, len=0x7f4f48ebeb3c, utvp=<value optimized out>) at sys/common/erl_poll.c:2184
#3 0x000000000055d5c3 in erts_check_io_nkp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1183

#4 0x00000000004989d6 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a442340, rq=0x7f4f4a440b40) at beam/erl_process.c:2533
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a442340) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f4f484be700 (LWP 27123)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af805b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af805b8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a44c600, rq=0x7f4f4a440cc0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a44c600, rq=0x7f4f4a440cc0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a44c600) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f4f47abd700 (LWP 27124)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af805f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af805f8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a4568c0, rq=0x7f4f4a440e40) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a4568c0, rq=0x7f4f4a440e40) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a4568c0) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f4f470bc700 (LWP 27125)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80638) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80638) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a460b80, rq=0x7f4f4a440fc0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a460b80, rq=0x7f4f4a440fc0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a460b80) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f4f466bb700 (LWP 27126)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80678) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80678) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a46ae40, rq=0x7f4f4a441140) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a46ae40, rq=0x7f4f4a441140) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a46ae40) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 5 (Thread 0x7f4f45cba700 (LWP 27127)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af806b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af806b8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a475100, rq=0x7f4f4a4412c0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a475100, rq=0x7f4f4a4412c0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a475100) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f4f452b9700 (LWP 27128)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af806f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af806f8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a47f3c0, rq=0x7f4f4a441440) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a47f3c0, rq=0x7f4f4a441440) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a47f3c0) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f4f448b8700 (LWP 27129)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80738) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80738) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4f4a489680, rq=0x7f4f4a4415c0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4f4a489680, rq=0x7f4f4a4415c0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f4f4a489680) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#9 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f4f43eb7700 (LWP 27130)):
#0 0x00007f4f4c9652d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f4f4af80778) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f4f4af80778) at pthread/ethr_event.c:218
#3 0x0000000000498c8e in erts_tse_wait (unused=<value optimized out>) at beam/erl_threads.h:2710
#4 aux_thread (unused=<value optimized out>) at beam/erl_process.c:2272
#5 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fff17897600) at pthread/ethread.c:106
#6 0x00007f4f4ce22851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f4f4c96890d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f4f4dafa700 (LWP 27102)):
#0 0x00007f4f4c9614f3 in select () from /lib64/libc.so.6
#1 0x0000000000550a70 in erts_sys_main_thread () at sys/unix/sys.c:3059
---Type <return> to continue, or q <return> to quit---
#2 0x00000000004490b9 in erl_start (argc=56, argv=<value optimized out>) at beam/erl_init.c:1775
#3 0x00000000004273a9 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29
(gdb)
(gdb)
(gdb)
(gdb)
(gdb)
(gdb) q
A debugging session is active.

Inferior 1 [process 27102] will be detached.

Quit anyway? (y or n) Y
Detaching from program: /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp, process 27102
[root@centos-64-x64 ~]# gdb -p 27136
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-64.el6_5.2)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 27136
Reading symbols from /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp...done.

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected address (wrong library or version mismatch?)
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 27174]
[New LWP 27167]
[New LWP 27166]
[New LWP 27165]
[New LWP 27164]
[New LWP 27163]
[New LWP 27162]
[New LWP 27161]
[New LWP 27160]
[New LWP 27159]
[New LWP 27158]
[New LWP 27157]
[New LWP 27156]
[New LWP 27155]
[New LWP 27154]
[New LWP 27153]
[New LWP 27152]
[New LWP 27151]
[New LWP 27150]
[New LWP 27149]
[New LWP 27148]
[New LWP 27147]
[New LWP 27146]
[New LWP 27145]
[New LWP 27144]
[New LWP 27143]
[New LWP 27142]
[New LWP 27141]
[New LWP 27140]
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-3.2/priv/lib/crypto.so...done.
Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-3.2/priv/lib/crypto.so
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-3.2/priv/lib/crypto_callback.so...done.
Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-3.2/priv/lib/crypto_callback.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-961ad59-git/priv/lib/couch_icu_driver.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-961ad59-git/priv/lib/couch_icu_driver.so
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
0x00007f7bfcfdc4f3 in select () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install couchbase-server-3.0.0-1174.x86_64
(gdb) thread app all bt

Thread 30 (Thread 0x7f7bfb5ff700 (LWP 27140)):
#0 0x00007f7bfd4a454d in read () from /lib64/libpthread.so.0
#1 0x0000000000550641 in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2916
#2 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e070) at pthread/ethread.c:106
#3 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f7bfa37f700 (LWP 27141)):
#0 0x00007f7bfd4a143c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005a7699 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1368
#2 0x0000000000463e3f in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:1788
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:938
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3286
#5 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e150) at pthread/ethread.c:106
#6 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f7bfabfe700 (LWP 27142)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600138) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600138) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2380) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2380) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2380) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f7bf997e700 (LWP 27143)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600178) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600178) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb24c0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb24c0) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb24c0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f7bf973f700 (LWP 27144)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6001b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6001b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2600) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2600) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2600) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f7bf971d700 (LWP 27145)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6001f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6001f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2740) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2740) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2740) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f7bf96fb700 (LWP 27146)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600238) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600238) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2880) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2880) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2880) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f7bf96d9700 (LWP 27147)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600278) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600278) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb29c0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb29c0) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb29c0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f7bf96b7700 (LWP 27148)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6002b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6002b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2b00) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2b00) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2b00) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f7bf9695700 (LWP 27149)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6002f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6002f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2c40) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2c40) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2c40) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 20 (Thread 0x7f7bf9673700 (LWP 27150)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600338) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600338) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2d80) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2d80) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2d80) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f7bf9651700 (LWP 27151)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600378) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600378) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb2ec0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb2ec0) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb2ec0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f7bf962f700 (LWP 27152)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6003b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6003b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb3000) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb3000) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb3000) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f7bf960d700 (LWP 27153)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6003f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6003f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb3140) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb3140) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb3140) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f7bf95eb700 (LWP 27154)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600438) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600438) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb3280) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb3280) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb3280) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f7bf95c9700 (LWP 27155)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600478) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600478) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb33c0) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb33c0) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb33c0) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f7bf95a7700 (LWP 27156)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6004b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6004b8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb3500) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb3500) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb3500) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f7bf9585700 (LWP 27157)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6004f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6004f8) at pthread/ethr_event.c:218
#3 0x00000000004f847a in erts_tse_wait (arg=0x7f7bfabb3640) at beam/erl_threads.h:2710
#4 async_get (arg=0x7f7bfabb3640) at beam/erl_async.c:371
#5 async_main (arg=0x7f7bfabb3640) at beam/erl_async.c:492
#6 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e130) at pthread/ethread.c:106
#7 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f7bfdf8e700 (LWP 27158)):
#0 0x00007f7bfd4a509d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000054f158 in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2840
#2 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e0a0) at pthread/ethread.c:106
#3 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f7bf90ff700 (LWP 27159)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600578) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600578) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4e3680, rq=0x7f7bfc4e1e40) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4e3680, rq=0x7f7bfc4e1e40) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc4e3680) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f7bf86fe700 (LWP 27160)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6005b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6005b8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4ed940, rq=0x7f7bfc4e1fc0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4ed940, rq=0x7f7bfc4e1fc0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc4ed940) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f7bf7cfd700 (LWP 27161)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6005f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6005f8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4f7c00, rq=0x7f7bfc4e2140) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc4f7c00, rq=0x7f7bfc4e2140) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc4f7c00) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f7bf72fc700 (LWP 27162)):
#0 0x00007f7bfcfe3f03 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000555b6c in check_fd_events (ps=0x7f7bfc454910, pr=0x7f7bf72fb320, len=0x7f7bf72fbb3c, utvp=0x7f7bf72fbb20) at sys/common/erl_poll.c:2023
#2 erts_poll_wait_kp (ps=0x7f7bfc454910, pr=0x7f7bf72fb320, len=0x7f7bf72fbb3c, utvp=0x7f7bf72fbb20) at sys/common/erl_poll.c:2184
#3 0x00000000005589b3 in erts_check_io_kp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1183
#4 0x00000000004989d6 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc501ec0, rq=0x7f7bfc4e22c0) at beam/erl_process.c:2533
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc501ec0) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f7bf68fb700 (LWP 27163)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600678) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600678) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc50c180, rq=0x7f7bfc4e2440) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc50c180, rq=0x7f7bfc4e2440) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc50c180) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 6 (Thread 0x7f7bf5efa700 (LWP 27164)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6006b8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6006b8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc516440, rq=0x7f7bfc4e25c0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc516440, rq=0x7f7bfc4e25c0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc516440) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f7bf54f9700 (LWP 27165)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb6006f8) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb6006f8) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc520700, rq=0x7f7bfc4e2740) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc520700, rq=0x7f7bfc4e2740) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc520700) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f7bf4af8700 (LWP 27166)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600738) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600738) at pthread/ethr_event.c:218
#3 0x0000000000498345 in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f7bfc52a9c0, rq=0x7f7bfc4e28c0) at beam/erl_threads.h:2710
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f7bfc52a9c0, rq=0x7f7bfc4e28c0) at beam/erl_process.c:2354
#5 0x000000000049e7b3 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:7017
#6 0x00000000005311d0 in process_main () at beam/beam_emu.c:1198
#7 0x0000000000493f94 in sched_thread_func (vesdp=0x7f7bfc52a9c0) at beam/erl_process.c:5801
#8 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#9 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f7bf40f7700 (LWP 27167)):
#0 0x00007f7bfcfe02d9 in syscall () from /lib64/libc.so.6
#1 0x00000000005a9c25 in wait__ (e=0x7f7bfb600778) at pthread/ethr_event.c:92
#2 ethr_event_wait (e=0x7f7bfb600778) at pthread/ethr_event.c:218
#3 0x0000000000498c8e in erts_tse_wait (unused=<value optimized out>) at beam/erl_threads.h:2710
#4 aux_thread (unused=<value optimized out>) at beam/erl_process.c:2272
#5 0x00000000005a95d6 in thr_wrapper (vtwd=0x7fffebc4e180) at pthread/ethread.c:106
#6 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f7ba97aa700 (LWP 27174)):
#0 0x00007f7bfd4a4d2d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f7baa112c21 in terminatorLoop (args=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchdb/src/mapreduce/mapreduce_nif.cc:480
---Type <return> to continue, or q <return> to quit---
#2 0x00000000005a95d6 in thr_wrapper (vtwd=0x7f7bf90fead0) at pthread/ethread.c:106
#3 0x00007f7bfd49d851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f7bfcfe390d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f7bfe175700 (LWP 27136)):
#0 0x00007f7bfcfdc4f3 in select () from /lib64/libc.so.6
#1 0x0000000000550a70 in erts_sys_main_thread () at sys/unix/sys.c:3059
#2 0x00000000004490b9 in erl_start (argc=46, argv=<value optimized out>) at beam/erl_init.c:1775
#3 0x00000000004273a9 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29
Comment by Aleksey Kondratenko [ 24/Aug/14 ]
beam's backtraces are irrelevant
Comment by Andrei Baranouski [ 24/Aug/14 ]
gdb -p 27228
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-64.el6_5.2)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 27228
Reading symbols from /opt/couchbase/bin/memcached...done.

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected address (wrong library or version mismatch?)
Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0
Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so...done.
Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so
Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1
Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4
Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done.
Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.6
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 31578]
[New LWP 31577]
[New LWP 31576]
[New LWP 31575]
[New LWP 31574]
[New LWP 31573]
[New LWP 31572]
[New LWP 31571]
[New LWP 31570]
[New LWP 31569]
[New LWP 31568]
[New LWP 27271]
[New LWP 27270]
[New LWP 27269]
[New LWP 27268]
[New LWP 27267]
[New LWP 27266]
[New LWP 27265]
[New LWP 27242]
[New LWP 27241]
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so
Reading symbols from /opt/couchbase/lib/libdirutils.so.0.1.0...done.
Loaded symbols for /opt/couchbase/lib/libdirutils.so.0.1.0
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
0x00007f78968df0ad in pthread_join () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install couchbase-server-3.0.0-1174.x86_64
(gdb) thread app all bt

Thread 21 (Thread 0x7f78943df700 (LWP 27241)):
#0 0x00007f7895a7360d in read () from /lib64/libc.so.6
#1 0x00007f7895a09f68 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007f7895a0ba6e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007f7895a0014a in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007f78959fefa9 in fgets () from /lib64/libc.so.6
#5 0x00007f78943e08b1 in check_stdin_thread (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#6 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce0e0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#7 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f78937db700 (LWP 27242)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x7f78939de6e0, mutex=0x7f78939de6a0, ms=<value optimized out>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f78937ddd53 in logger_thead_main (arg=0x140e900) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:342
#3 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce080) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f7892bce700 (LWP 27265)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007f7892bd2db0 in lock_engines (name=0x13d23f0 "RevAB") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:356
#5 find_bucket (name=0x13d23f0 "RevAB") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:715
#6 0x00007f7892bd45d0 in handle_select_bucket (handle=0x7f7892dda840, cookie=0x5b51600, request=0x5b85000, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:3054
#7 bucket_unknown_command (handle=0x7f7892dda840, cookie=0x5b51600, request=0x5b85000, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:3206
#8 0x0000000000418701 in process_bin_unknown_packet (c=0x5b51600) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:2616
#9 process_bin_packet (c=0x5b51600) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5414
#10 complete_nread (c=0x5b51600) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5818
#11 conn_nread (c=0x5b51600) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7032
#12 0x000000000040b4bd in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x5b51600)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7305
#13 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4280, flags=<value optimized out>) at event.c:1308
#14 event_process_active (base=0x5ce4280, flags=<value optimized out>) at event.c:1375
#15 event_base_loop (base=0x5ce4280, flags=<value optimized out>) at event.c:1572
#16 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce140) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#17 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f78921cd700 (LWP 27266)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007f7892bd2db0 in lock_engines (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:356
#5 find_bucket (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:715
#6 0x00007f7892bd3e02 in handle_connect (cookie=0x5be5b00, type=<value optimized out>, event_data=<value optimized out>, cb_data=0x7f7892dda840)
---Type <return> to continue, or q <return> to quit---
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1291
#7 0x000000000040dcaf in perform_callbacks (sfd=124, parent_port=11210, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4500, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:248
#8 conn_new (sfd=124, parent_port=11210, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4500, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:915
#9 0x00000000004198ea in thread_libevent_process (fd=<value optimized out>, which=<value optimized out>, arg=0x5cb2af0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:312
#10 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4500, flags=<value optimized out>) at event.c:1308
#11 event_process_active (base=0x5ce4500, flags=<value optimized out>) at event.c:1375
#12 event_base_loop (base=0x5ce4500, flags=<value optimized out>) at event.c:1572
#13 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce150) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#14 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f78917cc700 (LWP 27267)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007f7892bd2db0 in lock_engines (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:356
#5 find_bucket (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:715
#6 0x00007f7892bd3e02 in handle_connect (cookie=0x5c76300, type=<value optimized out>, event_data=<value optimized out>, cb_data=0x7f7892dda840)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1291
#7 0x000000000040dcaf in perform_callbacks (sfd=126, parent_port=11210, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4780, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:248
#8 conn_new (sfd=126, parent_port=11210, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4780, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:915
#9 0x00000000004198ea in thread_libevent_process (fd=<value optimized out>, which=<value optimized out>, arg=0x5cb2be0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:312
#10 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4780, flags=<value optimized out>) at event.c:1308
#11 event_process_active (base=0x5ce4780, flags=<value optimized out>) at event.c:1375
#12 event_base_loop (base=0x5ce4780, flags=<value optimized out>) at event.c:1572
#13 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce160) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#14 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f7890dcb700 (LWP 27268)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007f7892bd2db0 in lock_engines (name=0x1052b980 "UserInfo") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:356
#5 find_bucket (name=0x1052b980 "UserInfo") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:715
#6 0x00007f7892bd45d0 in handle_select_bucket (handle=0x7f7892dda840, cookie=0x5c79000, request=0x5ca3000, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:3054
#7 bucket_unknown_command (handle=0x7f7892dda840, cookie=0x5c79000, request=0x5ca3000, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:3206
#8 0x0000000000418701 in process_bin_unknown_packet (c=0x5c79000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:2616
#9 process_bin_packet (c=0x5c79000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5414
#10 complete_nread (c=0x5c79000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5818
#11 conn_nread (c=0x5c79000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7032
#12 0x000000000040b4bd in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x5c79000)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7305
---Type <return> to continue, or q <return> to quit---
#13 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4a00, flags=<value optimized out>) at event.c:1308
#14 event_process_active (base=0x5ce4a00, flags=<value optimized out>) at event.c:1375
#15 event_base_loop (base=0x5ce4a00, flags=<value optimized out>) at event.c:1572
#16 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce170) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#17 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f78903ca700 (LWP 27269)):
#0 0x00007f78968e243c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f788d8b671d in wait (this=0xd525b00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:39
#2 EventuallyPersistentStore::initialize (this=0xd525b00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:346

#3 0x00007f788d8c342c in EventuallyPersistentEngine::initialize (this=0x21072a00, config=<value optimized out>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:2011
#4 0x00007f788d8c3786 in EvpInitialize (handle=0x21072a00,
    config_str=0x1c106003 "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_size=314572800;tap_keepalive=300;dbname=/data/MsgsCalls;allow_data_loss_during_shutdown=true;backend=couchdb;couch_bucket=MsgsCalls;couch_port=11213;ma"...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:135
#5 0x00007f7892bd325b in create_bucket_UNLOCKED (e=0x7f7892dda840, bucket_name=0x1052c840 "MsgsCalls", path=0x1c105fe0 "/opt/couchbase/lib/memcached/ep.so",
    config=0x1c106003 "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_size=314572800;tap_keepalive=300;dbname=/data/MsgsCalls;allow_data_loss_during_shutdown=true;backend=couchdb;couch_bucket=MsgsCalls;couch_port=11213;ma"..., e_out=0x0, msg=0x7f78903c9730 "", msglen=1024) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:857
#6 0x00007f7892bd3496 in handle_create_bucket (handle=0x7f7892dda840, cookie=0x5b4fe00, request=<value optimized out>, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:2841
#7 0x00007f7892bd4979 in bucket_unknown_command (handle=0x7f7892dda840, cookie=0x5b4fe00, request=0x5b6f000, response=0x40d2a0 <binary_response_handler>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:3193
#8 0x0000000000418701 in process_bin_unknown_packet (c=0x5b4fe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:2616
#9 process_bin_packet (c=0x5b4fe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5414
#10 complete_nread (c=0x5b4fe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:5818
#11 conn_nread (c=0x5b4fe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7032
#12 0x000000000040b4bd in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x5b4fe00)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7305
#13 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4c80, flags=<value optimized out>) at event.c:1308
#14 event_process_active (base=0x5ce4c80, flags=<value optimized out>) at event.c:1375
#15 event_base_loop (base=0x5ce4c80, flags=<value optimized out>) at event.c:1572
#16 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce180) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#17 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f788f9c9700 (LWP 27270)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007f7892bd2db0 in lock_engines (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:356
#5 find_bucket (name=0x13d20a8 "default") at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:715
#6 0x00007f7892bd3e02 in handle_connect (cookie=0x5a30400, type=<value optimized out>, event_data=<value optimized out>, cb_data=0x7f7892dda840)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1291
#7 0x000000000040dcaf in perform_callbacks (sfd=91, parent_port=11209, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4f00, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:248
#8 conn_new (sfd=91, parent_port=11209, init_state=0x415390 <conn_new_cmd>, event_flags=18, read_buffer_size=<value optimized out>, base=0x5ce4f00, timeout=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:915
#9 0x00000000004198ea in thread_libevent_process (fd=<value optimized out>, which=<value optimized out>, arg=0x5cb2eb0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:312
#10 0x00007f78970abd3c in event_process_active_single_queue (base=0x5ce4f00, flags=<value optimized out>) at event.c:1308
#11 event_process_active (base=0x5ce4f00, flags=<value optimized out>) at event.c:1375
---Type <return> to continue, or q <return> to quit---
#12 event_base_loop (base=0x5ce4f00, flags=<value optimized out>) at event.c:1572
#13 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce190) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#14 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f788efc8700 (LWP 27271)):
#0 0x00007f7895a80f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f78970c0376 in epoll_dispatch (base=0x5ce5180, tv=<value optimized out>) at epoll.c:404
#2 0x00007f78970abc44 in event_base_loop (base=0x5ce5180, flags=<value optimized out>) at event.c:1558
#3 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce1a0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f788e5c7700 (LWP 31568)):
#0 0x00007f7895a44b8d in nanosleep () from /lib64/libc.so.6
#1 0x00007f7895a79d64 in usleep () from /lib64/libc.so.6
#2 0x00007f788d8f40b5 in updateStatsThread (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/memory_tracker.cc:36
#3 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ce240) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f788b31f700 (LWP 31569)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x000000000041a236 in notify_io_complete (cookie=0x5c27800, status=ENGINE_SUCCESS) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:459
#5 0x00007f788d8ba536 in EventuallyPersistentEngine::notifyIOComplete(const void *, ._101) (this=0x621e000, cookie=0x5c27800, status=ENGINE_SUCCESS)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.h:478
#6 0x00007f788d8a43e3 in EventuallyPersistentStore::completeBGFetchMulti (this=0x5d4f8c0, vbId=<value optimized out>, fetchedItems=std::vector of length 4, capacity 4 = {...},
    startTime=1593905183966731) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:1620
#7 0x00007f788d88f0a2 in BgFetcher::doFetch (this=0x5d31ba0, vbId=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:90
#8 0x00007f788d88f817 in BgFetcher::run (this=0x5d31ba0, task=0x81161c0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:156
#9 0x00007f788d91b603 in BgFetcherTask::run (this=0xfffffffffffffe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:89

#10 0x00007f788d8f5b3a in ExecutorThread::run (this=0x5eba000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:110
#11 0x00007f788d8f6296 in launch_executor_thread (arg=0x5cb2e68) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#12 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cec50) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#13 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f788a91e700 (LWP 31570)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x000000000041a236 in notify_io_complete (cookie=0x5b03200, status=ENGINE_SUCCESS) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:459
#5 0x00007f788d8ba536 in EventuallyPersistentEngine::notifyIOComplete(const void *, ._101) (this=0x621e000, cookie=0x5b03200, status=ENGINE_SUCCESS)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.h:478
#6 0x00007f788d8a43e3 in EventuallyPersistentStore::completeBGFetchMulti (this=0x5d4f8c0, vbId=<value optimized out>, fetchedItems=std::vector of length 4, capacity 4 = {...},
    startTime=1593905182650801) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:1620
#7 0x00007f788d88f0a2 in BgFetcher::doFetch (this=0x5d32560, vbId=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:90
#8 0x00007f788d88f817 in BgFetcher::run (this=0x5d32560, task=0x7ccbc50) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:156
#9 0x00007f788d91b603 in BgFetcherTask::run (this=0xfffffffffffffe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:89
---Type <return> to continue, or q <return> to quit---
#10 0x00007f788d8f5b3a in ExecutorThread::run (this=0x5eba0e0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:110
#11 0x00007f788d8f6296 in launch_executor_thread (arg=0x5cb2e68) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#12 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cec70) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#13 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f7889f1d700 (LWP 31571)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x000000000041a236 in notify_io_complete (cookie=0x5b03500, status=ENGINE_SUCCESS) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:459
#5 0x00007f788d8ba536 in EventuallyPersistentEngine::notifyIOComplete(const void *, ._101) (this=0x621e000, cookie=0x5b03500, status=ENGINE_SUCCESS)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.h:478
#6 0x00007f788d8a43e3 in EventuallyPersistentStore::completeBGFetchMulti (this=0x5d4f8c0, vbId=<value optimized out>, fetchedItems=std::vector of length 1, capacity 1 = {...},
    startTime=1593905182702662) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:1620
#7 0x00007f788d88f0a2 in BgFetcher::doFetch (this=0x5d305b0, vbId=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:90
#8 0x00007f788d88f817 in BgFetcher::run (this=0x5d305b0, task=0x8115d60) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:156
#9 0x00007f788d91b603 in BgFetcherTask::run (this=0xfffffffffffffe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:89
#10 0x00007f788d8f5b3a in ExecutorThread::run (this=0x5eba1c0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:110
#11 0x00007f788d8f6296 in launch_executor_thread (arg=0x5cb2e68) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#12 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cec60) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#13 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f788951c700 (LWP 31572)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x000000000041a236 in notify_io_complete (cookie=0x5b05300, status=ENGINE_SUCCESS) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:459
#5 0x00007f788d8ba536 in EventuallyPersistentEngine::notifyIOComplete(const void *, ._101) (this=0x621e000, cookie=0x5b05300, status=ENGINE_SUCCESS)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.h:478
#6 0x00007f788d8a43e3 in EventuallyPersistentStore::completeBGFetchMulti (this=0x5d4f8c0, vbId=<value optimized out>, fetchedItems=std::vector of length 4, capacity 4 = {...},
    startTime=1593905193178279) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:1620
#7 0x00007f788d88f0a2 in BgFetcher::doFetch (this=0x5d32630, vbId=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:90
#8 0x00007f788d88f817 in BgFetcher::run (this=0x5d32630, task=0x7cc8730) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/bgfetcher.cc:156
#9 0x00007f788d91b603 in BgFetcherTask::run (this=0xfffffffffffffe00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:89
#10 0x00007f788d8f5b3a in ExecutorThread::run (this=0x5eba2a0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:110
#11 0x00007f788d8f6296 in launch_executor_thread (arg=0x5cb2e68) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#12 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cec80) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#13 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f7888b1b700 (LWP 31573)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x14370c0, mutex=0x1437088, ms=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f788d91d7b5 in TaskQueue::_doSleep (this=0x1437080, t=...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f788d92056f in TaskQueue::_fetchNextTask (this=0x1437080, t=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:98
#4 0x00007f788d92090e in TaskQueue::fetchNextTask (this=0x1437080, thread=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:142

#5 0x00007f788d8e397c in ExecutorPool::_nextTask (this=0x5eb1840, t=..., tick=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:214
#6 0x00007f788d8e3a4e in ExecutorPool::nextTask (this=0x5eb1840, t=..., tick=236 '\354') at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:229
#7 0x00007f788d8f597b in ExecutorThread::run (this=0x5eba380) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:78
---Type <return> to continue, or q <return> to quit---
#8 0x00007f788d8f6296 in launch_executor_thread (arg=0x14370c4) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#9 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cec90) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#10 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f788811a700 (LWP 31574)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x14370c0, mutex=0x1437088, ms=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f788d91d7b5 in TaskQueue::_doSleep (this=0x1437080, t=...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f788d92056f in TaskQueue::_fetchNextTask (this=0x1437080, t=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:98
#4 0x00007f788d92090e in TaskQueue::fetchNextTask (this=0x1437080, thread=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:142
#5 0x00007f788d8e397c in ExecutorPool::_nextTask (this=0x5eb1840, t=..., tick=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:214
#6 0x00007f788d8e3a4e in ExecutorPool::nextTask (this=0x5eb1840, t=..., tick=114 'r') at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:229
#7 0x00007f788d8f597b in ExecutorThread::run (this=0x5eba460) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:78
#8 0x00007f788d8f6296 in launch_executor_thread (arg=0x14370c4) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#9 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13ceca0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#10 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f7887719700 (LWP 31575)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x14370c0, mutex=0x1437088, ms=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f788d91d7b5 in TaskQueue::_doSleep (this=0x1437080, t=...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f788d92056f in TaskQueue::_fetchNextTask (this=0x1437080, t=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:98
#4 0x00007f788d92090e in TaskQueue::fetchNextTask (this=0x1437080, thread=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:142
#5 0x00007f788d8e397c in ExecutorPool::_nextTask (this=0x5eb1840, t=..., tick=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:214
#6 0x00007f788d8e3a4e in ExecutorPool::nextTask (this=0x5eb1840, t=..., tick=10 '\n') at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:229
#7 0x00007f788d8f597b in ExecutorThread::run (this=0x5eba540) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:78
#8 0x00007f788d8f6296 in launch_executor_thread (arg=0x14370c4) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#9 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cecb0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#10 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f7886d18700 (LWP 31576)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x14370c0, mutex=0x1437088, ms=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f788d91d7b5 in TaskQueue::_doSleep (this=0x1437080, t=...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f788d92056f in TaskQueue::_fetchNextTask (this=0x1437080, t=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:98
#4 0x00007f788d92090e in TaskQueue::fetchNextTask (this=0x1437080, thread=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:142
#5 0x00007f788d8e397c in ExecutorPool::_nextTask (this=0x5eb1840, t=..., tick=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:214
#6 0x00007f788d8e3a4e in ExecutorPool::nextTask (this=0x5eb1840, t=..., tick=187 '\273') at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:229
#7 0x00007f788d8f597b in ExecutorThread::run (this=0x5eba620) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:78
#8 0x00007f788d8f6296 in launch_executor_thread (arg=0x14370c4) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#9 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cecc0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#10 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f7886317700 (LWP 31577)):
#0 0x00007f78968e27bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f7897b2a9eb in cb_cond_timedwait (cond=0x1437380, mutex=0x1437348, ms=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f788d91d7b5 in TaskQueue::_doSleep (this=0x1437340, t=...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f788d92056f in TaskQueue::_fetchNextTask (this=0x1437340, t=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:98
#4 0x00007f788d92090e in TaskQueue::fetchNextTask (this=0x1437340, thread=..., toSleep=true) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:142
---Type <return> to continue, or q <return> to quit---
#5 0x00007f788d8e397c in ExecutorPool::_nextTask (this=0x5eb1840, t=..., tick=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:214
#6 0x00007f788d8e3a4e in ExecutorPool::nextTask (this=0x5eb1840, t=..., tick=191 '\277') at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:229
#7 0x00007f788d8f597b in ExecutorThread::run (this=0x5eba700) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:78
#8 0x00007f788d8f6296 in launch_executor_thread (arg=0x1437384) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#9 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cecd0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#10 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f7885916700 (LWP 31578)):
#0 0x00007f78968e5054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f78968e0388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f78968e0257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7897b2aaf9 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x000000000041a236 in notify_io_complete (cookie=0x5c77500, status=ENGINE_SUCCESS) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:459
#5 0x00007f788d8ba536 in EventuallyPersistentEngine::notifyIOComplete(const void *, ._101) (this=0x5d2c000, cookie=0x5c77500, status=ENGINE_SUCCESS)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.h:478
#6 0x00007f788d912c26 in UprConnMap::manageConnections (this=0x5d40000) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tapconnmap.cc:1107
#7 0x00007f788d91887f in ConnManager::run (this=0x1433310) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tapconnmap.cc:151
#8 0x00007f788d8f5b3a in ExecutorThread::run (this=0x5eba7e0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:110
#9 0x00007f788d8f6296 in launch_executor_thread (arg=0x5cb2aa8) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:34
#10 0x00007f7897b2a7ea in platform_thread_wrap (arg=0x13cece0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#11 0x00007f78968de851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f7895a8090d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f78983427e0 (LWP 27228)):
#0 0x00007f78968df0ad in pthread_join () from /lib64/libpthread.so.0
#1 0x0000000000419f74 in threads_shutdown () at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/thread.c:659
#2 0x000000000040f5f5 in main (argc=<value optimized out>, argv=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:8762
Comment by Aleksey Kondratenko [ 24/Aug/14 ]
memcached backtraces look dead-lock-ful
Comment by Mike Wiederhold [ 24/Aug/14 ]
It looks like all of the executor threads are running background fetches which are blocked from completing due to bucket creation. The bucket creation is waiting on warmup to complete, but it is not able to since there is no available executor thread to run it.
Comment by Sundar Sridharan [ 25/Aug/14 ]
Thanks Mike, you are right about the lack of reader threads. There is a quick fix for this issue, but instead of merging that, I wish to understand why one bucket's bgfetch completion notification should get blocked on a new bucket creation in memcached. Will try to update once I understand this better. thanks
Comment by Andrei Baranouski [ 25/Aug/14 ]
possible root of the problem lies in the http://www.couchbase.com/issues/browse/MB-12056
on this cluster I've played with the issue
Comment by Sundar Sridharan [ 25/Aug/14 ]
Andrei, I looked at MB-12056, from the first looks at least it appears like they are separate issues.
Comment by Sundar Sridharan [ 25/Aug/14 ]
Discussed this with Trond, the root cause is that the front-end thread is getting blocked by a ep-engine background thread due to wait for warmup in initialize() phase of bucket creation.
The fix is to change this behavior to make bucket creation non-blocking to avoid the deadlock seen here.
thanks
Comment by Sundar Sridharan [ 26/Aug/14 ]
fix uploaded for review at http://review.couchbase.org/#/c/40892 thanks
Comment by Wayne Siu [ 26/Aug/14 ]
Reviewed with PM/Cihan, this ticket is approved for rc2.
Comment by Sundar Sridharan [ 26/Aug/14 ]
fix has been merged commit id 46df358fadbd1f2b57996ad5546702b0e66731ad thanks




[MB-12056] double-click on "Create new Data bucket" produces "Unexpected server error, request logged." after first bucket creation Created: 24/Aug/14  Updated: 26/Aug/14

Status: Reopened
Project: Couchbase Server
Component/s: ns_server, UI
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Andrei Baranouski Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1174

Attachments: PNG File create_bucket.png    
Triage: Triaged
Is this a Regression?: Unknown

 Description   
[ns_server:error,2014-08-24T10:47:53.281,ns_1@127.0.0.1:<0.437.0>:menelaus_web:loop:170]Server error during processing: ["web request failed",
                                 {path,"/pools/default/buckets"},
                                 {type,error},
                                 {what,badarith},
                                 {trace,
                                  [{menelaus_web_buckets,interpret_ram_quota,
                                    4,
                                    [{file,"src/menelaus_web_buckets.erl"},
                                     {line,979}]},
                                   {menelaus_web_buckets,
                                    parse_bucket_params_without_warnings,3,
                                    [{file,"src/menelaus_web_buckets.erl"},
                                     {line,696}]},
                                   {menelaus_web_buckets,parse_bucket_params,
                                    2,
                                    [{file,"src/menelaus_web_buckets.erl"},
                                     {line,665}]},
                                   {menelaus_web_buckets,do_bucket_create,3,
                                    [{file,"src/menelaus_web_buckets.erl"},
                                     {line,533}]},
                                   {menelaus_web_buckets,
                                    handle_bucket_create,2,
                                    [{file,"src/menelaus_web_buckets.erl"},
                                     {line,564}]},
                                   {request_throttler,do_request,3,
                                    [{file,"src/request_throttler.erl"},
                                     {line,59}]},
                                   {menelaus_web,loop,2,
                                    [{file,"src/menelaus_web.erl"},
                                     {line,149}]},
                                   {mochiweb_http,headers,5,
                                    [{file,
                                      "/buildbot/build_slave/centos-5-x64-300-builder/build/build/couchdb/src/mochiweb/mochiweb_http.erl"},
                                     {line,94}]}]}]


 Comments   
Comment by Andrei Baranouski [ 24/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12056/logs.zip
Comment by Aleksey Kondratenko [ 25/Aug/14 ]
Need cbcollectinfos and not just logs
Comment by Andrei Baranouski [ 25/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12056/b37d4250/10.3.4.145-8252014-133-diag.zip
Comment by Aleksey Kondratenko [ 25/Aug/14 ]
http://review.couchbase.org/40889
Comment by Wayne Siu [ 26/Aug/14 ]
PM(Cihan) reviewed and agreed that the fix goes to 3.0.1.




[MB-12055] ns_janitor may lose replicas of nearly completed vbucket moves (was: {DCP} : Delta Recovery Impossible after re-try of graceful failover since in first attempt failed) Created: 22/Aug/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Parag Agarwal Assignee: Parag Agarwal
Resolution: Fixed Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 1186, centos 6x, 10.6.2.144-10.6.2.160

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
build 1186

 Scenario
1. Create a 7 Node cluster
2. Create default bucket with 200 K items
3. Graceful failover a node
4. Kill memcached of another node during graceful failover
5. Graceful failover the same node in step 3
6. Add-back the node with delta recovery
7. Hit Rebalance

We see the following warning:: "Fail Over Warning: Rebalance required, some data is not currently replicated!”

In Step 7, Rebalance fails for delta recovery. Says delta recovery is not possible. Although we see nodes in the cluster are in healthy state. This is true when we have 200 K items Vs 100K items where it passes.

I am attaching the logs for you to analyze. Since the above warning comes in both cases. Not sure about the internal state of the system which stops the add-back delta recovery.

Test fails for 2k items
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1436-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-1437-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-1439-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-1440-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1441-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1442-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1444-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1446-couch.tar.gz

Test passes for 1 K Items

https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1458-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-150-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-151-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-153-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-154-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-156-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-157-diag.zip

 Comments   
Comment by Aleksey Kondratenko [ 25/Aug/14 ]
http://review.couchbase.org/40886
Comment by Aleksey Kondratenko [ 25/Aug/14 ]
manifest updated here: http://review.couchbase.org/40888
Comment by Parag Agarwal [ 25/Aug/14 ]
The issues =is fixed for the scenario mentioned. Also tested it by killing 3 nodes, stopping 3 nodes, stop graceful failover.
Comment by Wayne Siu [ 26/Aug/14 ]
Reopening it for proper tagging (RC2).




[MB-12054] [windows] [2.5.1] cluster hang when flush beer-sample bucket Created: 22/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Thuan Nguyen Assignee: Raju Suravarjjala
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: windows server 2008 R2

Attachments: Zip Archive 172.23.107.124-8222014-1546-diag.zip     Zip Archive 172.23.107.125-8222014-1547-diag.zip     Zip Archive 172.23.107.126-8222014-1548-diag.zip     Zip Archive 172.23.107.127-8222014-1549-diag.zip    
Triage: Triaged
Operating System: Windows 64-bit
Is this a Regression?: Unknown

 Description   
Install couchbase server 2.5.1 on 4 nodes windows server 2008 R2 64-bit
Create a cluster of 4 nodes
Create beer-sample bucket
Enable flush in bucket setting.
Flush beer-sample bucket. Cluster became hang.




[MB-12053] memcached.log needs to capture crashes and aborts Created: 22/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: memcached
Affects Version/s: 2.5.1, 3.0
Fix Version/s: techdebt-backlog
Security Level: Public

Type: Bug Priority: Major
Reporter: Sundar Sridharan Assignee: Trond Norbye
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to MB-12035 Rationalize and 'fix' memcached / ep_... Open
Triage: Triaged
Is this a Regression?: Unknown

 Description   
memcached and ep-engine do not log crashes or aborts in memcached.log.
So critical runtime failures can sometimes get silently ignored.
This is made even worse if core files are not enabled on any machine and all crash information is lost.

It would be good to have the following..
1) Record a crash reason in memcached.log so the crash time is placed in context with other log messages
2) Dump backtraces for all the threads
3) Dump stats for all buckets - since state of the system at the point of the crash can be most relevant.
4) Allow the crash dump infrastructure to be invoked by an external process in case a hang is detected.

 Comments   
Comment by Dave Rigby [ 22/Aug/14 ]
Related / possibly some overlap with MB-12035
Comment by Dave Rigby [ 22/Aug/14 ]
Also, I was talking to Trond (and others in Mcr) about adding Google Brakepad support (https://code.google.com/p/google-breakpad/) to memcached.exe which would allow the equivalent of core dumps but at a significantly reduced size (few KB...MB instead of the multi-GB cores we typically see from memcached).
Comment by Trond Norbye [ 26/Aug/14 ]
We have ongoing work on this from the Manchester team by Adam and Mark. I'll grab the ownership of the bug report until we've sorted out the direction.




[MB-12052] add stale=false semantic changes to release notes Created: 22/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0, 3.0-Beta
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Matt Ingenthron Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Need to release note the stale=false semantic changes.
This doesn't seem to be in current release notes, though MB-11589 seems loosely related.

Please use/adapt from the following text:
Starting with the 3.0 release, the "stale" view query argument "false" has been enhanced so it will consider all document changes which have been received at the time the query has been received. This means that use of the `durability requirements` or `observe` feature to block for persistence in application code before issuing the `false` stale query is no longer needed. It is recommended that you remove all such application level checks after completing the upgrade to the 3.0 release.

- - -

Ruth: assigning this to you to work out the right way to work the text into the release notes. This probably goes with a change in a different MB.




Mac version update check is incorrectly identifying newest version (MB-10214)

[MB-12051] Update the Release_Server job on Jenkins to include updating the file (membasex.xml) and the download URL Created: 22/Aug/14  Updated: 22/Aug/14

Status: Open
Project: Couchbase Server
Component/s: build
Affects Version/s: 2.0.1, 2.2.0, 2.1.1, 2.5.0
Fix Version/s: 3.0
Security Level: Public

Type: Technical task Priority: Blocker
Reporter: Wayne Siu Assignee: Chris Hillery
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We can update the Release_Server job on Jenkins to create an updated version of this XML file from a template, and upload it to S3.




[MB-12050] Set LANG=C on cbcollect_info. Created: 22/Aug/14  Updated: 22/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Minor
Reporter: Patrick Varley Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: supportability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
There have been a number of cases where we have got the couchbase.log in another language unfortunately most of the support team only speaks English.

I believe if we set the LANG=C it should return English.





[MB-12049] Doc get_misses is wrong Created: 22/Aug/14  Updated: 22/Aug/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Patrick Varley Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: supportability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: http://docs.couchbase.com/couchbase-manual-2.2/#ongoing-monitoring-and-maintenance

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
A get_misses means couchbase does not have the doc.

Our documentation suggests that it means the doc is not in memory:

"""
Cache Misses ( get_misses )

Ideally this should be low, and certainly lower than get_hits. Increasing or high values mean that data that your application expects to be stored is not in memory.
"""

Instead of get_misses maybe this should be ep_bg_fetched?

It is worth explaining what get_misses mean as the name is misleading.




[MB-12048] View engine 2.5 to 3.0 index file upgrade Created: 22/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sarath Lakshman Assignee: Ketaki Gangal
Resolution: Fixed Votes: 0
Labels: RC2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Gantt: start-finish
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
View engine 2.5 index files are not compatible with 3.0 index files. Hence, it requires index rebuild for 3.0.
We need a method that renames index files to new compatible filenames (with signature) and append new header.

 Comments   
Comment by Volker Mische [ 22/Aug/14 ]
I created a hacky script that still involves some manual steps, though i think it works in general.

Though I'm hitting a bigger issue with DCP. It expectes that you send the correct partition/vBucket version with your request whenever you don't start indexing from scratch. The problem is that this information is persisted on disk, hence in the index header. When we offline upgrade from 2.x to 3.0 we don't know which partition that server might have, hence we can't save the correct one.

Currently the only sane way I see is changing the DCP semantics and making it possible to resume from a certain seq number with sending {0, 0} as partition version (currently you need to send the correct partition version in case you want to resume).
Comment by Sriram Melkote [ 22/Aug/14 ]
Folks - we need to do this as an automatic feature (i.e., fully automated without needing any manual steps) or not at all. Let's talk with EP engine folks to spec this.
Comment by Sarath Lakshman [ 22/Aug/14 ]
I am guessing it can be done as part of installation postscript which checks the current version and performs upgrade of existing files.
Eg. RPM and Deb has a way to specify post scripts.
Comment by Volker Mische [ 22/Aug/14 ]
Sarath, yes, that would be a way.
Comment by Volker Mische [ 22/Aug/14 ]
Siri, I misunderstood you comment as you probably did mine. The manual steps are only needed atm to verify my idea works. The final result will be a script that can be run without any manual steps.

My misunderstanding was that I thought you talk about "online upgrade", but you didn't really say that.
Comment by Sriram Melkote [ 22/Aug/14 ]
Ketaki, can we add a test to detect this situation? The test must fail until we fix this issue.
Comment by Ketaki Gangal [ 22/Aug/14 ]
Hi Siri,

We run automation tests which do offline upgrades from 2.X to 3.X, what these tests dont check is whether index is rebuilt /not.
https://github.com/couchbase/testrunner/blob/master/conf/py-newupgrade.conf#L39

I ll update the tests to add a check for index-rebuild verification.

Sarath: Can you provide details on how to check if indexes are rebuilt/not.


Comment by Sarath Lakshman [ 23/Aug/14 ]
I can think of a very easy way where you can have a timeout for stale=false after Couchbase is up soon after warmup. Run a stale=false and it should run immediately. Since index rebuild is going on, it would take more time.
Comment by Sriram Melkote [ 25/Aug/14 ]
Product Managers, please note:

This is UPGRADE step. We'll need to discuss how we'll handle upgrade logic in the product before we decide on closure of this issue.

+Anil, Ilam, Cihan
Comment by Cihan Biyikoglu [ 25/Aug/14 ]
lets pick this up on the daily synup tomorrow.
thanks
-cihan
Comment by Volker Mische [ 26/Aug/14 ]
I think I found a nice solution. Now it's an online upgrade:

http://review.couchbase.org/40914
http://review.couchbase.org/40915
http://review.couchbase.org/40916
Comment by Sriram Melkote [ 27/Aug/14 ]
This has been discussed earlier this week and is a 3.0 approved exception.




[MB-12046] CBTransfer showing missing items when items are present after topology change with graceful failover+ full recovery with nodes crashing Created: 21/Aug/14  Updated: 27/Aug/14  Resolved: 22/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: tools
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Parag Agarwal Assignee: Bin Cui
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1184

Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.144-8212014-1657-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.144-8212014-176-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.145-8212014-1658-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.145-8212014-176-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.146-8212014-1659-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.146-8212014-176-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.147-8212014-171-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.147-8212014-176-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.148-8212014-172-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.148-8212014-177-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.149-8212014-173-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.149-8212014-177-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.150-8212014-174-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12046/10.6.2.150-8212014-177-couch.tar.gz
Is this a Regression?: Unknown

 Description   
Scenario

1. Create 7 node cluster
2. Create default bucket and add 100K items
3. Graceful failover 1 node
4. During Graceful failover, kill memcached of 3 other nodes, this fails graceful failover
5. Restart Graceful failover and let it run to completion
6. Full recover the failed over node and rebalance
7. During rebalance, kill memcached of 3 other nodes, this fails rebalance
8. Restart Rebalance and run it to completion

After Step 8, we collect data using cbtransfer and compare it to the one we had in step 2. We see missing keys.

Note that there are no mutations running from step 3 to step 8. We always read from couch store after the queues have been drained and replication is complete. Also, before we run cbtransfer, we verified item counts and verified data items as well

This seems like a bug in cbtransfer

Missing keys

failover97727
 failover96541
 failover19942
 failover72566
 failover98994
 failover21107
 failover17597
 failover47535
 failover58469
 failover47247
 failover79250
 failover95182
 failover48606
 failover885
 failover98366
 failover72214
 failover24016
 failover74124
 failover51288
 failover41177
 failover47925
 failover19220
 failover6008
 failover40281
 failover94916
 failover20361
 failover29410
 failover29800
 failover61528
 failover90103
 failover73072
 failover17817
 failover46753
 failover27955
 failover91997
 failover25502
 failover99672
 failover32149
 failover19552
 failover34279
 failover26723
 failover16113
 failover79522
 failover96951
 failover11737
 failover15332
 failover70253
 failover78036
 failover20413
 failover45200
 failover13192
 failover14154
 failover31368
 failover88099
 failover44684
 failover49460
 failover25882
 failover62699
 failover12486
 failover81678
 failover23632
 failover15850
 failover27237
 failover505
 failover11045
 failover49312
 failover94496
 failover95760
 failover24186
 failover10941
 failover84769
 failover72976
 failover77295
 failover20993
 failover15440
 failover12516
 failover277
 failover38589
 failover92636
 failover22844
 failover72384
 failover73700
 failover95012
 failover82459
 failover22326
 failover87548
 failover87958
 failover98414
 failover78744
 failover23140
 failover27545
 failover10223
 failover61938
 failover3119
 failover14626
 failover79932
 failover74656
 failover12896
 failover41605
 failover93322
 failover42424
 failover92144
 failover99090
 failover94274
 failover91365
 failover77867
 failover44066
 failover24764
 failover40311
 failover38809
 failover20803
 failover68259
 failover54209
 failover76163
 failover70931
 failover6198
 failover44714
 failover18734
 failover51318
 failover43642
 failover98584
 failover49870
 failover43130
 failover82849
 failover41795
 failover28104
 failover13002
 failover10551
 failover76611
 failover91807
 failover17407
 failover22454
 failover91587
 failover67618
 failover29580
 failover16661
 failover915
 failover90671
 failover495
 failover12906
 failover9449
 failover42356
 failover97055
 failover98804
 failover77477
 failover29990
 failover40463
 failover42834
 failover45390
 failover29362
 failover9859
 failover57028
 failover38419
 failover77305
 failover17987
 failover71035
 failover76781
 failover45962
 failover71747
 failover75820
 failover18046
 failover91417
 failover20583
 failover12264
 failover25492
 failover64439
 failover94886
 failover70521
 failover28676
 failover48796
 failover45572
 failover16083
 failover25912
 failover49282
 failover64829
 failover96233
 failover26051
 failover38999
 failover99100
 failover3089
 failover48174
 failover5229
 failover21097
 failover93450
 failover37058
 failover25270
 failover46021
 failover93840
 failover62709
 failover28094
 failover40873
 failover13770
 failover58879
 failover90093
 failover88109
 failover73690
 failover67788
 failover17375
 failover94506
 failover75342
 failover54399
 failover52139
 failover75430
 failover21675a

 Comments   
Comment by Parag Agarwal [ 22/Aug/14 ]
does not repro with 1186, marking it for closure




[MB-12044] Enable on/off from the Couchbase UI where it pre-loads a random document while editing a view Created: 21/Aug/14  Updated: 21/Aug/14

Status: Open
Project: Couchbase Server
Component/s: UI
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Major
Reporter: Larry Liu Assignee: Anil Kumar
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
When I see a random document displayed initially at the top of the page used for editing a View, then I would think it might be a sampled document FROM THAT VIEW. But it’s not. It’s random and may have nothing to do with that view. Can that “initial load” of a random document be turned off if you don’t want to see it?

To be clear, I DO want that window to be visible after I click Show Results for the view and click on one of the resulting document links. Those documents are returned from the view, so it makes sense there, but it’s the immediate load of an unrelated document that doesn’t make sense.




[MB-12043] cbq crash after trying to delete a key Created: 21/Aug/14  Updated: 21/Aug/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
cbq> delete from my_bucket KEYS ['query-testa7480c4-0'];
PANIC: Expected plan.Operator instead of <nil>..




[MB-12042] Memcached crashes with Segmentation fault during bucket deletion Created: 21/Aug/14  Updated: 26/Aug/14  Resolved: 22/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Meenakshi Goel Assignee: Meenakshi Goel
Resolution: Fixed Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1174-rel, Debian 7.0

Attachments: Text File crash_log.txt     Text File test_log.txt    
Triage: Triaged
Is this a Regression?: Yes

 Description   
Jenkins Ref Link:
http://qa.hq.northscale.net/job/debian_x64--90_04--observe_tests-P0/1/consoleFull

Test Failed:
./testrunner -i yourfile.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,GROUP=ALL -t observe.observetest.ObserveTests.test_observe_basic_data_load_delete,items=100,sasl_buckets=1,standard_buckets=1,rebalance=out,GROUP=P1

1. Note that failed test will not deterministically reproduce the crash.
2. Same test and some other tests found to be failed on CentOS and Ubuntu but found no core dumps.
http://qa.hq.northscale.net/job/centos_x64--44_04--observe-P0/125/console

[user:info,2014-08-21T5:17:57.340,ns_1@10.3.5.154:<0.750.0>:ns_orchestrator:handle_info:483]Rebalance exited with reason {unexpected_exit,
                              {'EXIT',<0.32255.122>,
                               {{{badmatch,{error,closed}},
                                 {gen_server,call,
                                  ['ns_memcached-standard_bucket0',
                                   {set_vbucket,610,replica},
                                   180000]}},
                                {gen_server,call,
                                 [{'janitor_agent-standard_bucket0',
                                   'ns_1@10.3.5.154'},
                                  {if_rebalance,<0.14541.122>,
                                   {dcp_takeover,'ns_1@10.3.5.155',634}},
                                  infinity]}}}}

[error_logger:error,2014-08-21T5:17:57.340,ns_1@10.3.5.154:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
  crasher:
    initial call: ns_single_vbucket_mover:-set_initial_vbucket_state/6-fun-0-/0
    pid: <0.32266.122>
    registered_name: []
    exception error: {bulk_set_vbucket_state_failed,
                      [{'ns_1@10.3.5.154',
                        {'EXIT',
                         {{{badmatch,{error,closed}},
                           {gen_server,call,
                            ['ns_memcached-standard_bucket0',
                             {set_vbucket,610,replica},
                             180000]}},
                          {gen_server,call,
                           [{'janitor_agent-standard_bucket0',
                             'ns_1@10.3.5.154'},
                            {if_rebalance,<0.14541.122>,
                             {update_vbucket_state,610,replica,passive,
                              'ns_1@10.3.5.155'}},
                            infinity]}}}}]}
      in function janitor_agent:bulk_set_vbucket_state/4 (src/janitor_agent.erl, line 411)
    ancestors: [<0.32268.122>,<0.14541.122>,<0.14465.122>,<0.750.0>,
                  mb_master_sup,mb_master,ns_server_sup,ns_server_cluster_sup,
                  <0.60.0>]
    messages: []
    links: [<0.32268.122>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 3407
  neighbours:
[ns_server:info,2014-08-21T5:17:57.344,ns_1@10.3.5.154:<0.32684.122>:diag_handler:log_all_tap_and_checkpoint_stats:125]logging tap & checkpoint stats
[error_logger:error,2014-08-21T5:17:57.342,ns_1@10.3.5.154:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
  crasher:
    initial call: ns_memcached:init/1
    pid: <0.18706.119>
    registered_name: []
    exception exit: {badmatch,{error,closed}}
      in function gen_server:init_it/6 (gen_server.erl, line 328)
    ancestors: ['single_bucket_sup-default',<0.18679.119>]
    messages: []
    links: [<0.18720.119>,<0.18726.119>,<0.18744.119>,<0.279.0>,
                  <0.18696.119>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 75113
    stack_size: 27
    reductions: 632988
  neighbours:
    neighbour: [{pid,<0.18744.119>},
                  {registered_name,[]},
                  {initial_call,{erlang,apply,['Argument__1','Argument__2']}},
                  {current_function,{gen,do_call,4}},
                  {ancestors,['ns_memcached-default',
                              'single_bucket_sup-default',<0.18679.119>]},
                  {messages,[]},
                  {links,[<0.18706.119>,#Port<0.174708>]},
                  {dictionary,[]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,6772},
                  {stack_size,23},
                  {reductions,525127}]
    neighbour: [{pid,<0.18726.119>},
                  {registered_name,[]},
                  {initial_call,{erlang,apply,['Argument__1','Argument__2']}},
                  {current_function,{gen,do_call,4}},
                  {ancestors,['ns_memcached-default',
                              'single_bucket_sup-default',<0.18679.119>]},
                  {messages,[]},
                  {links,[<0.18706.119>,#Port<0.174706>]},
                  {dictionary,[]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,28690},
                  {stack_size,23},
                  {reductions,3329}]
                  {reductions,3329}]
    neighbour: [{pid,<0.18720.119>},
                  {registered_name,[]},
                  {initial_call,{erlang,apply,['Argument__1','Argument__2']}},
                  {current_function,{gen,do_call,4}},
                  {ancestors,['ns_memcached-default',
                              'single_bucket_sup-default',<0.18679.119>]},
                  {messages,[]},
                  {links,[<0.18706.119>,#Port<0.174713>]},
                  {dictionary,[]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,46422},
                  {stack_size,23},
                  {reductions,2151481}]

Please refer attached crash_log.txt.
Core is available at 10.3.5.155:/tmp/backup_crash/21_08_2014_05_45
Uploading Logs

 Comments   
Comment by Meenakshi Goel [ 21/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12042/f9ad56ee/10.3.5.154-8212014-520-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12042/c7d38635/10.3.5.155-8212014-522-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12042/fd4f484f/10.3.2.146-8212014-524-diag.zip
Comment by Anil Kumar [ 21/Aug/14 ]
Venu please look at it and work with Sundar to verify the test case.
Comment by Sundar Sridharan [ 21/Aug/14 ]
Looks like the crash is in warmup callback in the midst of bucket creation and due to a missing vbucket entry. Can we have the list of exact steps for manual reproduction? thanks
Comment by Venu Uppalapati [ 21/Aug/14 ]
The test case invokes rebalance of nodes. Investigating if there is race between bucket creation and rebalance.
Comment by Meenakshi Goel [ 22/Aug/14 ]
Steps to Reproduce:
1. Create 3 buckets default, sasl_bucket and standard_bucket
2. Rebalance-in 2 nodes
3. Load data 100 items
4. Wait for item to get persist on disk
5. Run re-balance out and Observe in parallel
6. Create View and Run query on view with stale=false

While querying on standard_bucket test fails with timeout error wherein only 100 items are there and timeout is 120 seconds.
In tearDown below steps are occurring during which crash is occurring after rebalance failure (Please note that as mentioned in description crash doesn't happen every time):
1. Memcached is being killed using killall -9 memcached
2. Stop Rebalance
3. Bucket deletion

Attached is the test output. test_log.txt
Comment by Sundar Sridharan [ 22/Aug/14 ]
Does not seem to reproduce with cluster_run easily.
Comment by Sundar Sridharan [ 22/Aug/14 ]
fix ready for merge at http://review.couchbase.org/#/c/40849/ thanks
Comment by Sundar Sridharan [ 22/Aug/14 ]
fix is merged. can you please verify the same. thanks
Comment by Meenakshi Goel [ 26/Aug/14 ]
Verified with 3.0.0-1191-rel and observed no memcached Crash.

http://qa.hq.northscale.net/job/debian_x64--90_04--observe_tests-P0/3/console




[MB-12041] Disabling access.log on multiple buckets results in node failing to become available Created: 21/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Brent Woodruff Assignee: Abhinav Dangeti
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2014-08-26 at 8.53.15 AM.png     PNG File Screen Shot 2014-08-26 at 8.53.29 AM.png     PNG File Screen Shot 2014-08-26 at 8.58.37 AM.png     PNG File Screen Shot 2014-08-26 at 8.58.43 AM.png    
Issue Links:
Dependency
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
On review of a customer ticket today during support discussions, this particular issue was brought up. It is unclear from subsequent discussions in that ticket whether this issue was addressed and fixed.

Steps to reproduce:

* Initialize a Couchbase node with more than one bucket

* Disable the access.log on *both* buckets using the following command for each bucket:

wget -O- --user=Administrator --password=password --post-data='ns_bucket:update_bucket_props("bucket1", [{extra_config_string, "alog_path="}]).' http://localhost:8091/diag/eval

wget -O- --user=Administrator --password=password --post-data='ns_bucket:update_bucket_props("bucket2", [{extra_config_string, "alog_path="}]).' http://localhost:8091/diag/eval

where 'bucket1' and 'bucket2' are the bucket names.

* Restart the node and observe the following errors in the logs:

memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
memcached<0.89.0>: Unsupported key: <^A>

* Note that the node remains pending and never becomes available

 Comments   
Comment by Abhinav Dangeti [ 21/Aug/14 ]
I don't see the node failing to become available.

Started couchbase server with 2 buckets:
  1 Fri Aug 22 10:36:25.628702 PDT 3: (default) Trying to connect to mccouch: "127.0.0.1:13000"
  2 Fri Aug 22 10:36:25.628978 PDT 3: (default) Connected to mccouch: "127.0.0.1:13000"
  3 Fri Aug 22 10:36:25.644502 PDT 3: (No Engine) Bucket default registered with low priority
  4 Fri Aug 22 10:36:25.644528 PDT 3: (No Engine) Spawning 4 readers, 4 writers, 1 auxIO, 1 nonIO threads
  5 Fri Aug 22 10:36:25.646178 PDT 3: (default) metadata loaded in 982 usec
  6 Fri Aug 22 10:36:25.646205 PDT 3: (default) Enough number of items loaded to enable traffic
  7 Fri Aug 22 10:36:25.646559 PDT 3: (default) warmup completed in 1052 usec
  8 Fri Aug 22 10:36:33.495128 PDT 3: (default) Shutting down tap connections!
  9 Fri Aug 22 10:36:33.495174 PDT 3: (default) Shutting down dcp connections!
 10 Fri Aug 22 10:36:33.496244 PDT 3: (No Engine) Unregistering last bucket default
 11 Fri Aug 22 10:36:41.791797 PDT 3: (bucket1) Trying to connect to mccouch: "127.0.0.1:13000"
 12 Fri Aug 22 10:36:41.791932 PDT 3: (bucket1) Connected to mccouch: "127.0.0.1:13000"
 13 Fri Aug 22 10:36:41.800241 PDT 3: (No Engine) Bucket bucket1 registered with low priority
 14 Fri Aug 22 10:36:41.800273 PDT 3: (No Engine) Spawning 4 readers, 4 writers, 1 auxIO, 1 nonIO threads
 15 Fri Aug 22 10:36:41.801437 PDT 3: (bucket1) metadata loaded in 719 usec
 16 Fri Aug 22 10:36:41.801450 PDT 3: (bucket1) Enough number of items loaded to enable traffic
 17 Fri Aug 22 10:36:41.801593 PDT 3: (bucket1) warmup completed in 761 usec
 18 Fri Aug 22 10:36:46.922063 PDT 3: (bucket2) Trying to connect to mccouch: "127.0.0.1:13000"
 19 Fri Aug 22 10:36:46.922191 PDT 3: (bucket2) Connected to mccouch: "127.0.0.1:13000"
 20 Fri Aug 22 10:36:46.931024 PDT 3: (No Engine) Bucket bucket2 registered with low priority
 21 Fri Aug 22 10:36:46.932154 PDT 3: (bucket2) metadata loaded in 715 usec
 22 Fri Aug 22 10:36:46.932170 PDT 3: (bucket2) Enough number of items loaded to enable traffic
 23 Fri Aug 22 10:36:46.932314 PDT 3: (bucket2) warmup completed in 776 usec

Loaded 1000 items in each, and restarted node, after setting the alog_path to NULL, in the same way mentioned.
  1 Fri Aug 22 10:38:08.372050 PDT 3: (bucket2) Trying to connect to mccouch: "127.0.0.1:13000"
  2 Fri Aug 22 10:38:08.372307 PDT 3: (bucket2) Connected to mccouch: "127.0.0.1:13000"
  3 Fri Aug 22 10:38:08.382418 PDT 3: (No Engine) Bucket bucket2 registered with low priority
  4 Fri Aug 22 10:38:08.382445 PDT 3: (No Engine) Spawning 4 readers, 4 writers, 1 auxIO, 1 nonIO threads
  5 Fri Aug 22 10:38:08.434024 PDT 3: (bucket1) Trying to connect to mccouch: "127.0.0.1:13000"
  6 Fri Aug 22 10:38:08.434205 PDT 3: (bucket1) Connected to mccouch: "127.0.0.1:13000"
  7 Fri Aug 22 10:38:08.445064 PDT 3: (No Engine) Bucket bucket1 registered with low priority
  8 Fri Aug 22 10:38:08.481732 PDT 3: (bucket2) metadata loaded in 98 ms
  9 Fri Aug 22 10:38:08.507847 PDT 3: (bucket2) warmup completed in 124 ms
 10 Fri Aug 22 10:38:08.540342 PDT 3: (bucket1) metadata loaded in 92 ms
 11 Fri Aug 22 10:38:08.553951 PDT 3: (bucket1) warmup completed in 106 ms

[10:37:46] abhinav: ~/Documents/couchbase30/ep-engine $ ./management/cbstats localhost:12000 all -b bucket1 | grep alog
 ep_alog_block_size: 4096
 ep_alog_path:
 ep_alog_sleep_time: 1440
 ep_alog_task_time: 10
[10:38:50] abhinav: ~/Documents/couchbase30/ep-engine $ ./management/cbstats localhost:12000 all -b bucket2 | grep alog
 ep_alog_block_size: 4096
 ep_alog_path:
 ep_alog_sleep_time: 1440
 ep_alog_task_time: 10

I do see the duplicate entry warning, but that I'm guessing is because we set alog_path again after initializing it to the default value, in which case it would overwrite.
Comment by Abhinav Dangeti [ 22/Aug/14 ]
I tried your scenario with the latest 3.0 and then with 2.5.1, and noted similar behavior.
Can you point me to the build, with which you saw this issue, or perhaps the logs from where you hit the issue?
Comment by Abhinav Dangeti [ 25/Aug/14 ]
I also merged a change that will let the user enable/disable access log generation during run time:
http://review.couchbase.org/#/c/40884/
Comment by Brent Woodruff [ 26/Aug/14 ]
Hi Abhinav,

Apologies for the delay in getting back to you regarding a build that exhibits this issue. I believe it was the 2.5.1 release, since that was what the customer had just upgraded to in the originating issue. I was running that release on my Mac testing the commands provided by engineering to disable the access.log. I was able to reproduce this issue again today.

Attached screenshots of before and after these commands:

$./cbstats localhost:11210 config -b default |grep alog_path
 ep_alog_path: /Users/brent/Library/Application Support/Couchbase/var/lib/couchdb/default/access.log

$./cbstats localhost:11210 config -b beer-sample |grep alog_path
 ep_alog_path: /Users/brent/Library/Application Support/Couchbase/var/lib/couchdb/beer-sample/access.log

$ wget -O- --user=Administrator --password=couchbase --post-data='ns_bucket:update_bucket_props("default", [{extra_config_string, "alog_path="}]).' http://localhost:8091/diag/eval
# wget output removed

$ wget -O- --user=Administrator --password=couchbase --post-data='ns_bucket:update_bucket_props("beer-sample", [{extra_config_string, "alog_path="}]).' http://localhost:8091/diag/eval
# wget output removed

# Restarted Couchbase

$./cbstats localhost:11210 config -b default |grep alog_path
 ep_alog_path:

$./cbstats localhost:11210 config -b beer-sample |grep alog_path
# no output
Comment by Brent Woodruff [ 26/Aug/14 ]
Note: In the first screen shot showing the buckets, both buckets were availalbe, I just did not think to get the green indicator since my goal was just to show that there was more than one bucket configured.
Comment by Abhinav Dangeti [ 26/Aug/14 ]
Brent, I did not see this issue when I used a couchbase instance started by cluster_run. However I do see this issue with a mac build for some reason.
Note that this issue is not because of having multiple buckets in the cluster. I see this issue with a single bucket as well.

The duplicate entry warning showed just once while in cluster_run. With the build however, the babysitter logs are flooded with those messages:

37899 memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
37900 memcached<0.89.0>: Unsupported key: <Vû^F^A>
37901 memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
37902 memcached<0.89.0>: Unsupported key: <<86>^C^G^A>
37903 memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
37904 memcached<0.89.0>: Unsupported key: <öê^F^A>
37905 memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
37906 memcached<0.89.0>: Unsupported key: <&ó^F^A>
37907 memcached<0.89.0>: WARNING: Found duplicate entry for "alog_path"
37908 memcached<0.89.0>: Unsupported key: <Vû^F^A>
...
Comment by Abhinav Dangeti [ 26/Aug/14 ]
However, this issue already seems to be resolved in 3.0. Verified with 3.0.0-1175-rel.
Comment by Abhinav Dangeti [ 27/Aug/14 ]
Please re-open if you see this with 3.0




[MB-12040] cbrestore backing up linux and restoring on OSX does not Created: 21/Aug/14  Updated: 28/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: tools
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Facundo Farias Assignee: Bin Cui
Resolution: Won't Fix Votes: 0
Labels: backup, customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: OSX

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
I am using the build Version 2.5.1-1083-rel.

We do have daily backups of our cluster on production. I took one of those, and after a whole morning dealing with it, I realized that I was on OSX so I had to use this instructions: http://docs.couchbase.com/couchbase-manual-2.0/#backup-and-restore-between-mac-os-x-and-other-platforms.

So, basically I did what the example shows:

shell> cbrestore /linuxbackup/today memcached://mac:11211 -b test -B test

And magically, ir works. The thing is that, the data it is on the default bucket, instead of the one that I pointed on the command line option, so the documents were imported correctly, but on the wrong bucket.

{code}
  [####################] 100.0% (11067/estimated 11067 msgs)
bucket: test, msgs transferred...
       : total | last | per sec
 byte : 4464406 | 4464406 | 2203418.8
2014-08-21 13:18:58,718: mt warning: cannot restore bucket design on a memached destination
done
{code}


 Comments   
Comment by Bin Cui [ 21/Aug/14 ]
Since the default vbucket number is different between linux and osx, you have to use -x rehash=1 during restore.
Comment by Facundo Farias [ 22/Aug/14 ]
Okay. I will try, but can you also update the documentation about it!?
Thanks.
Comment by Facundo Farias [ 26/Aug/14 ]
I've tried with the rehash parameter, but it is still doing the restore on the default bucket.
Comment by Bin Cui [ 26/Aug/14 ]
restore tool is not smart enough to find out the right bucket for you. By default, it always restores to default bucket.
However, you can use "-B bucketname" option for the destination bucket. Use "-h", you can find out all the possible options.
Comment by Facundo Farias [ 28/Aug/14 ]
That's the thing. I've been using the "-B test" option as I put on the description, but it is always restoring the data on the default bucket.




[MB-12039] Document the maximum length of a bucket name Created: 21/Aug/14  Updated: 21/Aug/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 2.2.0, 2.5.0, 2.5.1, 3.0, 3.0-Beta
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Major
Reporter: Ian McCloy Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
As per MB-5641 since Couchbase Server 2.2 the bucket name has been limited to 100 characters in the UI to prevent problems.

Can you ensure the following documentation pages include the information,
"The bucket name can only contain characters in range A-Z, a-z, 0-9 as well as underscore, period, dash and percent symbols and can only be a maximum of 100 characters in length."

http://docs.couchbase.com/couchbase-manual-2.5/cb-admin/#couchbase-admin-web-console-data-buckets-createedit
http://docs.couchbase.com/prebuilt/couchbase-manual-3.0/UI/ui-data-buckets.html
http://docs.couchbase.com/prebuilt/couchbase-manual-3.0/Misc/limits.html
http://docs.couchbase.com/couchbase-manual-2.5/cb-rest-api/#creating-and-editing-buckets
http://docs.couchbase.com/prebuilt/couchbase-manual-3.0/REST/rest-bucket-create.html






[MB-12038] build go programs in ns_server instead of shipping pre-built binaries (was: [windows] F-Secure flagging binary) Created: 21/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 2.5.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Sriram Melkote Assignee: Wayne Siu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
A popular virus scanner, F-Secure is deleting a file installed by us, generate_cert.exe.

We need to analyze why this is happening.

See: https://groups.google.com/forum/#!topic/couchbase/E3QvNolCknQ


 Comments   
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
What you want me to do? Those crappy antiviruses are known to have false positives occasionally.
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
And btw our build machines _are_ clean.
Comment by Sriram Melkote [ 27/Aug/14 ]
You could compile it latest go on Windows, and the resulting binary does not trip any virus scanners
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
I don't compile anything on windows. This binaries are built on GNU/Linux machine using go cross compiler. I can rebuild using more recent go but it's unclear if it's really going to help.
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
And surely if build folks are ready to build our go stuff in ns_server I'm ok if some fan of cmake can make it happen.
Comment by Sriram Melkote [ 27/Aug/14 ]
Yes, let's request build team to add ability to compile Go code. We'll need it soon enough, independent of this bug anyway.




[MB-12037] ns_server may lose replicas on stopped rebalance/graceful failover (was: {DCP} : Delta Recovery Impossible after re-try of graceful failover since in first attempt failed) Created: 21/Aug/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Closed
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Parag Agarwal Assignee: Aleksey Kondratenko
Resolution: Fixed Votes: 0
Labels: RC2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 10.6.2.144-10.6.2.150
centos 6x
1174

Triage: Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8202014-2226-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8202014-2227-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8202014-2227-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8202014-2227-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8202014-2227-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8202014-2228-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8202014-2228-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/1174.log.tar.gz
Is this a Regression?: Unknown

 Description   
Scenario
1. Create a 7 Node cluster
2. Create default bucket with 100 K items
3. Graceful failover a node
4. Kill memcached of another node during graceful failover
5. Graceful failover the same node in step 3
6. Add-back the node with delta recovery
7. Hit Rebalance

In Step 7, Rebalance fails for delta recovery. Says delta recovery is not possible. Although we see nodes in the cluster are in healthy state.

We see the following warning:: "Fail Over Warning: Rebalance required, some data is not currently replicated!”

Seems like the delta recovery will not work in this condition, unless we rebalance the cluster. Also, I was able to cancel the delta recovery and do full recovery.

Opening the bug to follow-up on the issue. Attaching logs and data files



 Comments   
Comment by Aleksey Kondratenko [ 21/Aug/14 ]
I was able to reproduce it easily. There's indeed something wrong with restarting graceful failover which impacts delta recovery.
Comment by Aleksey Kondratenko [ 21/Aug/14 ]
And predictably happens with any stop/restart of graceful failover.
Comment by Parag Agarwal [ 21/Aug/14 ]
When the warning is showing (" Rebalance required, some data is not currently replicated!"), we don't expect delta recovery to succeed and this should be correct behavior? Asking since we will have to document it as well

Comment by Aleksey Kondratenko [ 21/Aug/14 ]
Warning has nothing to do with that. And warning is valid. Midway into graceful failover your're not balanced indeed.
Comment by Aleksey Kondratenko [ 21/Aug/14 ]
manifest updated here: http://review.couchbase.org/40811

fix merged here: http://review.couchbase.org/40803
Comment by Parag Agarwal [ 22/Aug/14 ]
Tested
Comment by Parag Agarwal [ 22/Aug/14 ]
Test Run:: http://qa.hq.northscale.net/job/centos_x64--02_01--Rebalance-In/6/console
Comment by Parag Agarwal [ 22/Aug/14 ]
Saw the issue again for the following scenario with build 1186 1

 Scenario
1. Create a 7 Node cluster
2. Create default bucket with 200 K items
3. Graceful failover a node
4. Kill memcached of another node during graceful failover
5. Graceful failover the same node in step 3
6. Add-back the node with delta recovery
7. Hit Rebalance

We see the following warning:: "Fail Over Warning: Rebalance required, some data is not currently replicated!”

In Step 7, Rebalance fails for delta recovery. Says delta recovery is not possible. Although we see nodes in the cluster are in healthy state. This is true when we have 200 K items Vs 100K items where it passes.

I am attaching the logs for you to analyze. Since the above warning comes in both cases. Not sure about the internal state of the system which stops the add-back delta recovery.

Test fails for 2k items
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1436-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-1437-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-1439-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-1445-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-1440-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1441-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1442-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1446-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1444-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1446-couch.tar.gz

Test passes for 1 K Items

https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-1458-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.144-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-150-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.145-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-151-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.146-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-153-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.147-8222014-159-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.148-8222014-154-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.149-8222014-156-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-1510-couch.tar.gz
https://s3.amazonaws.com/bugdb/jira/MB-12037/10.6.2.150-8222014-157-diag.zip
Comment by Parag Agarwal [ 22/Aug/14 ]
I think the logs were not uploaded, will add them again
Comment by Parag Agarwal [ 22/Aug/14 ]
fixed the logs
Comment by Aleksey Kondratenko [ 22/Aug/14 ]
Please open new ticket for new instance of the issue
Comment by Parag Agarwal [ 22/Aug/14 ]
http://www.couchbase.com/issues/browse/MB-12055
Comment by Wayne Siu [ 26/Aug/14 ]
Reviewed with PM/Cihan. Approved for RC2.




[MB-12036] cbbackup throws error and exits with exception Created: 20/Aug/14  Updated: 26/Aug/14  Resolved: 20/Aug/14

Status: Resolved
Project: Couchbase Server
Component/s: tools
Affects Version/s: 3.0, 3.0-Beta
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Ashvinder Singh Assignee: Bin Cui
Resolution: Fixed Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: All OSes

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Tested with build: 3.0.0-1174.rel

Setup: 3-Node cluster with XDCR between another 3-node cluster.

The cbbackup fails when try to take full backup of the cluster.
Command executed: /opt/couchbase/bin/cbbackup http://Administrator:password@172.23.106.71:8091 /tmp/backup -m full

Output from Cbbackup>>

vb_937:abs_high_seqno 0
vb_756:abs_high_seqno 0
vb_982:purge_seqno 0
vb_813:high_seqno 0
vb_921:high_seqno 0
vb_1008:uuid 54486033539771
vb_976:uuid 21079925921752


Exception in thread w2:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/couchbase/lib/python/pump.py", line 286, in run_worker
    source_map, sink_map, self.ctl, curx).run()
  File "/opt/couchbase/lib/python/pump.py", line 344, in run
    rv_batch, batch = self.source.provide_batch()
  File "/opt/couchbase/lib/python/pump_dcp.py", line 113, in provide_batch
    rv, dcp_conn = self.get_dcp_conn()
  File "/opt/couchbase/lib/python/pump_dcp.py", line 395, in get_dcp_conn
    self.setup_dcp_streams()
  File "/opt/couchbase/lib/python/pump_dcp.py", line 475, in setup_dcp_streams
    if int(vbid) not in self.node_vbucket_map:
ValueError: invalid literal for int() with base 10: '751:abs'


The last build when cbbackup was working: 3.0.0-1157. The build 3.0.0-1158 when cbbackup was broken.

 Comments   
Comment by Bin Cui [ 20/Aug/14 ]
http://review.couchbase.org/#/c/40767/
Comment by Wayne Siu [ 26/Aug/14 ]
Reviewed by PM/Cihan, this ticket is approved for RC2.




[MB-12035] Rationalize and 'fix' memcached / ep_engine logging Created: 20/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket, memcached
Affects Version/s: 3.0
Fix Version/s: techdebt-backlog
Security Level: Public

Type: Bug Priority: Major
Reporter: Dave Rigby Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: memcached
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to MB-12053 memcached.log needs to capture crashe... Open
Is this a Regression?: Yes