[MB-12285] Update elastic search documentation with configuration info on the document type feature Created: 01/Oct/14  Updated: 01/Oct/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Subhashni Balakrishnan Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
A new feature has been added to the elastic search plugin which allows setting type for couchbase documents.
Refer MB-12284 for the request.

A sample shown below uses regular expression selector for the plugin to determine the type of the couchbase document from its key. The following needs to be on the elastic search config file. Default is config/elasticsearch.yml.

couchbase.typeSelector: org.elasticsearch.transport.couchbase.capi.RegexTypeSelector
To set the document type as review, use a regular expression of the couchbase document keys.
couchbase.documentTypes.review: review-\d$

Parent field can be configured by using
couchbase.documentTypeParentFields.review: doc.bookId

And routing can be configured as
couchbase.documentTypeRoutingFields.review: doc.bookId
 
where bookId is a couchbasedocument key and the plugin requires it to be a string.

Related info:
https://github.com/couchbaselabs/elasticsearch-transport-couchbase/issues/6
https://github.com/couchbaselabs/elasticsearch-transport-couchbase/wiki/Supporting-More-Than-One-Document-Type





[MB-12283] XDCR@next release - initialize replication manager in main.go Created: 30/Sep/14  Updated: 30/Sep/14  Resolved: 30/Sep/14

Status: Resolved
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: feature-backlog
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Xiaomei Zhang Assignee: Yu Sui
Resolution: Done Votes: 0
Labels: sprint2_xdcr
Remaining Estimate: 8h
Time Spent: Not Specified
Original Estimate: 8h

Epic Link: XDCR next release




[MB-12282] XDCR@next release - End to end testing on AWS Created: 30/Sep/14  Updated: 30/Sep/14

Status: In Progress
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: feature-backlog
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Xiaomei Zhang Assignee: Yu Sui
Resolution: Unresolved Votes: 0
Labels: sprint2_xdcr
Remaining Estimate: 40h
Time Spent: Not Specified
Original Estimate: 40h

Epic Link: XDCR next release

 Description   
Goals:
1. end to end with two cluster, each cluster with one node
2. set up replication with filter, change filter
3. enable log level per replication
4. rough performance comparison




[MB-12281] XDCR@next release - filter part Created: 30/Sep/14  Updated: 30/Sep/14

Status: Open
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: feature-backlog
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Xiaomei Zhang Assignee: Xiaomei Zhang
Resolution: Unresolved Votes: 0
Labels: sprint2_xdcr
Remaining Estimate: 32h
Time Spent: Not Specified
Original Estimate: 32h

Epic Link: XDCR next release




[MB-12280] XDCR@next release - Distributed Metadata Integration Created: 30/Sep/14  Updated: 30/Sep/14

Status: Open
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: feature-backlog
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Xiaomei Zhang Assignee: Xiaomei Zhang
Resolution: Unresolved Votes: 0
Labels: sprint2_xdcr
Remaining Estimate: 24h
Time Spent: Not Specified
Original Estimate: 24h

Epic Link: XDCR next release




[MB-12279] 86% regression in 95th percentile GET latency on Windows 2012 Created: 30/Sep/14  Updated: 01/Oct/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Venu Uppalapati Assignee: Venu Uppalapati
Resolution: Unresolved Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Yes

 Description   
86% regression in 95th percentile GET latency on Windows 2012

 Comments   
Comment by Venu Uppalapati [ 30/Sep/14 ]
will assign owner after attaching logs.




[MB-12278] Azure Marketplace Image - Enterprise Edition and Community Edition - provide VHD to PM Created: 30/Sep/14  Updated: 30/Sep/14

Status: Open
Project: Couchbase Server
Component/s: cloud
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Major
Reporter: Anil Kumar Assignee: Wei-Li Liu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Need VHD for 3.0.0 GA

 Comments   
Comment by Wei-Li Liu [ 30/Sep/14 ]
Anil: Please send me Azure Account so that I can work on creating VHD




[MB-12277] [3.0.1 ubuntu]: Bucket goes in Pending state on setting memcached -t option Created: 30/Sep/14  Updated: 01/Oct/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Meenakshi Goel Assignee: Meenakshi Goel
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.1-1354-rel

Triage: Triaged
Operating System: Ubuntu 64-bit
Is this a Regression?: Yes

 Description   
Jenkins Ref Link:
http://qa.sc.couchbase.com/job/ubuntu_x64--47_01--connection-P1/55/consoleFull

Test to Reproduce:
python testrunner.py -i ini-file -t connectionstests.ConnectionTests.test_memcahed_t_option

Command:
[ns_config:update_key({node, N, memcached}, fun (PList) -> lists:keystore(verbosity, 1, PList, {verbosity, '-t 8'}) end) || N <- ns_node_disco:nodes_wanted()].

Test Logs:
2014-09-30 02:47:18 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.24:8091: True content: [ok] command: [ns_config:update_key({node, N, memcached}, fun (PList) -> lists:keystore(verbosity, 1, PList, {verbosity, '-t 8'}) end) || N <- ns_node_disco:nodes_wanted()].
2014-09-30 02:47:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. wait some time before restart ...
2014-09-30 02:47:23 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.24 with username : root password : couchbase ssh_key:
2014-09-30 02:47:23 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.24
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: sudo cat /proc/cpuinfo
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: df -Th
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: sudo cat /proc/meminfo
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: hostname
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: hostname -d
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: ls /tmp | grep dump
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2014-09-30 02:47:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: /etc/init.d/couchbase-server stop
2014-09-30 02:47:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2014-09-30 02:47:26 | INFO | MainProcess | test_thread | [remote_util.log_command_output] * Stopped couchbase-server
2014-09-30 02:47:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: /etc/init.d/couchbase-server start
2014-09-30 02:47:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2014-09-30 02:47:35 | INFO | MainProcess | test_thread | [remote_util.log_command_output] * Started couchbase-server
2014-09-30 02:47:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: ls /tmp | grep dump
2014-09-30 02:47:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully


Live Node:
172.23.107.24

**Please note that test passed successfully with 3.0.0-1209-rel.
http://qa.sc.couchbase.com/job/ubuntu_x64--47_01--connection-P1/51/consoleFull

Uploading Logs

 Comments   
Comment by Meenakshi Goel [ 30/Sep/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12277/f9ad56ee/172.23.107.24-diag.zip
Comment by Chiyoung Seo [ 01/Oct/14 ]
Can you please grab the gdb backtrace when the bucket is in pending state?




[MB-12276] Maintenance patch install scripts need to check existing version info for compatibility before updating files Created: 29/Sep/14  Updated: 29/Sep/14

Status: Open
Project: Couchbase Server
Component/s: build
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Gokul Krishnan Assignee: Chris Hillery
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The Maintenance patch update/install scripts should check the existing version info for compatibility before updating files. This will prevent customers from installing MPs on older/incompatible versions.




[MB-12275] Maintenance Patches need to update UI/Logs with version change when applied Created: 29/Sep/14  Updated: 29/Sep/14

Status: Open
Project: Couchbase Server
Component/s: build
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Gokul Krishnan Assignee: Chris Hillery
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Per the last discussion with the hotfix team, it was universally agreed that the UI & logs needs to display the Maintenance Patch level information when applied. At the moment, only the manifest file contains that information, but isn't propagated to the UI or Logs. This needs to be rectified for the next MP.




[MB-12274] 2.5.1 Maintenance Patch-2 Release Created: 29/Sep/14  Updated: 30/Sep/14

Status: Open
Project: Couchbase Server
Component/s: build
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Gokul Krishnan Assignee: Chris Hillery
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to MB-12238 Infinite timeout on outgoing xmem req... Open
relates to MB-12241 Fix memory leak in vbucket deletion Open
relates to MB-12156 time of check/time of use race in dat... Resolved
relates to MB-10437 XDCR replication rate drops almost to... Closed
relates to MB-11873 [XDCR] Mutations are not replicated f... Closed

 Description   
Creating an MB to track the release of 2.5.1 MP2, initial list of MB's that need to pulled into this MP are below:

CBSE-1399 -> MB-10437, MB-12238, MB-11873
CBSE-1371 -> MB-12241




[MB-12273] Memory leak in vbucket deletion caused by bg fetch items not being deleted. Created: 29/Sep/14  Updated: 29/Sep/14  Resolved: 29/Sep/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Major
Reporter: Patrick Varley Assignee: Mike Wiederhold
Resolution: Fixed Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
When a bucket flush happens items that are currently being bg fetched, will not be deleted and will consume memory.

 Comments   
Comment by Patrick Varley [ 29/Sep/14 ]
Can you check I got all the detail right here? I think it affect all older versions of Couchbase.

Thanks,
Patrick
Comment by Mike Wiederhold [ 29/Sep/14 ]
Yeah, this looks right. I believe this issue exists in older versions of Couchbase, but I have verified that it doesn't exist in 3.0.




[MB-12272] [ubuntu]DCP upgrade exited with reason badmatch (offline upgrade 2.0.0->3.0.1) Created: 29/Sep/14  Updated: 29/Sep/14

Status: Open
Project: Couchbase Server
Component/s: None
Affects Version/s: 3.0.1
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Iryna Mironava Assignee: Iryna Mironava
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.1-1326-rel

Triage: Untriaged
Operating System: Ubuntu 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/bugdb/jira/MB-12272/dca3bf89/10.3.3.138-9232014-1728-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12272/dca3bf89/10.6.2.80-9232014-1729-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12272/dca3bf89/10.6.2.73-9232014-1728-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12272/dca3bf89/10.6.2.83-9232014-1730-diag.zip
Is this a Regression?: Unknown

 Description   
test:
newupgradetests.MultiNodesUpgradeTests.offline_cluster_upgrade,initial_version=2.0.0-1976-rel,nodes_init=4,during-ops=start_index,ddocs-num=3,upgrade_version=3.0.1-1326-rel,initial_vbuckets=512,get-cbcollect-info=True,GROUP=P0,timeout=300

{u'node': u'ns_1@10.3.3.138', u'code': 2, u'text': u'DCP upgrade exited with reason {{badmatch,\n {error,{failed_nodes,[\'ns_1@10.3.3.138\']}}},\n [{dcp_upgrade,handle_call,3,\n [{file,"src/dcp_upgrade.erl"},{line,65}]},\n {gen_server,handle_msg,5,\n [{file,"gen_server.erl"},{line,585}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,239}]}]}\n', u'shortText': u'message', u'serverTime': u'2014-09-23T17:27:36.292Z', u'module': u'ns_orchestrator', u'tstamp': 1411518456292, u'type': u'info'}
[2014-09-23 17:27:40,830] - [rest_client:2034] ERROR - {u'node': u'ns_1@10.3.3.138', u'code': 2, u'text': u'Rebalance exited with reason bad_replicas\n', u'shortText': u'message', u'serverTime': u'2014-09-23T17:27:35.942Z', u'module': u'ns_orchestrator', u'tstamp': 1411518455942, u'type': u'info'}
[2014-09-23 17:27:40,831] - [rest_client:2034] ERROR - {u'node': u'ns_1@10.3.3.138', u'code': 2, u'text': u"Bad replicators after rebalance:\nMissing = [{'ns_1@10.6.2.80','ns_1@10.3.3.138',256},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',257},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',258},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',259},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',260},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',261},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',262},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',263},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',264},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',265},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',266},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',267},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',268},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',269},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',270},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',271},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',272},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',273},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',274},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',275},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',276},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',277},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',278},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',279},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',280},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',281},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',282},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',283},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',284},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',285},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',286},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',287},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',288},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',289},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',290},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',291},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',292},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',293},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',294},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',295},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',296},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',297},\n {'ns_1@10.6.2.80','ns_1@10.3.3.138',298},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',341},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',342},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',343},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',344},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',345},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',346},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',347},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',348},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',349},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',350},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',351},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',352},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',353},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',354},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',355},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',356},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',357},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',358},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',359},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',360},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',361},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',362},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',363},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',364},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',365},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',366},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',367},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',368},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',369},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',370},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',371},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',372},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',373},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',374},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',375},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',376},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',377},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',378},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',379},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',380},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',381},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',382},\n {'ns_1@10.6.2.80','ns_1@10.6.2.83',383}]\nExtras = []", u'shortText': u'message', u'serverTime': u'2014-09-23T17:27:35.937Z', u'module': u'ns_rebalancer', u'tstamp': 1411518455937, u'type': u'info'}

 Comments   
Comment by Parag Agarwal [ 29/Sep/14 ]
Seems like a Bad Replicator issue " u'code': 2, u'text': u"Bad replicators after rebalance"
Comment by Aleksey Kondratenko [ 29/Sep/14 ]
There's a number of auth errors like this:

[error_logger:info,2014-09-23T17:27:36.263,ns_1@10.3.3.138:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
          supervisor: {local,'ns_vbm_sup-default'}
             started: [{pid,<0.1022.0>},
                       {name,
                           {new_child_id,
                               [257,258,259,260,261,262,263,264,265,266,267,
                                268,269,270,271,272,273,274,275,276,277,278,
                                279,280,281,282,283,284,285,286,287,288,289,
                                290,291,292,293,294,295,296,297,298],
                               'ns_1@10.6.2.80'}},
                       {mfargs,
                           {ebucketmigrator_srv,start_link,
                               [{{"10.6.2.80",11209},
                                 {"10.3.3.138",11209},
                                 [{on_not_ready_vbuckets,
                                      #Fun<tap_replication_manager.1.103303975>},
                                  {username,"default"},
                                  {password,get_from_config},
                                  {vbuckets,
                                      [257,258,259,260,261,262,263,264,265,
                                       266,267,268,269,270,271,272,273,274,
                                       275,276,277,278,279,280,281,282,283,
                                       284,285,286,287,288,289,290,291,292,
                                       293,294,295,296,297,298]},
                                  {set_to_pending_state,false},
                                  {takeover,false},
                                  {suffix,"ns_1@10.3.3.138"},
                                  {src_node,'ns_1@10.6.2.80'},
                                  {dst_node,'ns_1@10.3.3.138'}]}]}},
                       {restart_type,temporary},
                       {shutdown,60000},
                       {child_type,worker}]

[error_logger:error,2014-09-23T17:27:36.267,ns_1@10.3.3.138:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
  crasher:
    initial call: ebucketmigrator_srv:init/1
    pid: <0.1022.0>
    registered_name: []
    exception error: no match of right hand side value
                     {memcached_error,auth_error,<<"Auth failure">>}
      in function mc_replication:connect/1 (src/mc_replication.erl, line 43)
      in call from mc_replication:connect/1 (src/mc_replication.erl, line 49)
      in call from ebucketmigrator_srv:init/1 (src/ebucketmigrator_srv.erl, line 502)
    ancestors: ['ns_vbm_sup-default','single_bucket_sup-default',<0.529.0>]
    messages: []
    links: [<0.560.0>,#Port<0.7621>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 1064
  neighbours:

Which makes no sense given that default bucket is supposed to have no password at all. I'll need live system in order to investigate this.
Comment by Aleksey Kondratenko [ 29/Sep/14 ]
Also seeing this:

[ns_server:info,2014-09-23T17:27:36.291,babysitter_of_ns_1@127.0.0.1:<0.77.0>:ns_port_server:log:169]memcached<0.77.0>: Tue Sep 23 17:27:36.090684 PDT 3: 63: Invalid username/password combination
memcached<0.77.0>: Tue Sep 23 17:27:36.128035 PDT 3: 63: Invalid username/password combination
memcached<0.77.0>: Tue Sep 23 17:27:36.207143 PDT 3: 63: Invalid username/password combination
memcached<0.77.0>: Tue Sep 23 17:27:36.237060 PDT 3: 63: Invalid username/password combination
memcached<0.77.0>: Tue Sep 23 17:27:36.281951 PDT 3: 63: Invalid username/password combination




[MB-12271] rebalance failed during partially online upgrade cluster 3.0.0->3.0.1 Created: 29/Sep/14  Updated: 01/Oct/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Mike Wiederhold
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.1-1326-rel

Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/bugdb/jira/MB-12271/dca3bf89/172.27.33.10-9292014-2013-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12271/dca3bf89/172.27.33.11-9292014-2016-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12271/dca3bf89/172.27.33.12-9292014-2014-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12271/dca3bf89/172.27.33.13-9292014-2017-diag.zip
Is this a Regression?: Unknown

 Description   
test to reproduce
newupgradetests.MultiNodesUpgradeTests.online_upgrade_rebalance_in_with_ops,initial_version=3.0.0-1209-rel,sasl_buckets=1,standard_buckets=1,items=10000,nodes_init=2,nodes_in=1,nodes_out=1,ddocs-num=1,upgrade_version=3.0.1-1326-rel,initial_vbuckets=512

1) 2 nodes cluster of 3.0.0, 3 buckets
2) remove 1 3.0.0 node and add one of 3.0.1. Rebalance failed
Bad replicators after rebalance:
Missing = [{'ns_1@172.27.33.11','ns_1@172.27.33.10',366},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',370},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',380},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',381},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',384},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',385},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',386},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',387},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',388},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',389},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',390},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',391},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',392},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',393},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',394},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',395},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',396},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',397},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',398},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',399},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',400},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',401},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',402},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',403},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',404},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',405},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',406},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',407},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',408},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',409},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',410},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',411},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',412},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',413},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',414},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',415},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',416},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',417},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',418},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',419},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',420},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',421},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',422},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',423},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',424},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',425},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',426},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',427},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',428},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',429},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',430},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',431},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',432},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',433},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',434},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',435},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',436},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',437},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',438},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',439},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',440},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',441},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',442},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',443},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',444},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',445},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',446},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',447},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',448},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',449},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',450},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',451},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',452},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',453},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',454},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',455},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',456},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',457},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',458},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',459},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',460},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',461},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',462},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',463},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',464},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',465},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',466},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',467},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',468},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',469},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',470},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',471},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',472},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',473},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',474},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',475},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',476},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',477},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',478},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',479},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',480},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',481},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',482},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',483},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',484},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',485},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',486},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',487},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',488},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',489},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',490},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',491},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',492},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',493},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',494},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',495},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',496},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',497},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',498},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',499},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',500},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',501},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',502},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',503},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',504},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',505},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',506},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',507},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',508},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',509},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',510},
{'ns_1@172.27.33.11','ns_1@172.27.33.10',511}]
Extras = []

will attach cbcollects

 Comments   
Comment by Aleksey Kondratenko [ 29/Sep/14 ]
Appears to be caused by this:

Mon Sep 29 20:30:33.657017 MSK 3: (standard_bucket0) Notified the completion of checkpoint persistence for vbucket 374, id 108, cookie 0xbae6f00
Mon Sep 29 20:30:33.658158 MSK 3: (standard_bucket0) Notified the completion of checkpoint persistence for vbucket 339, id 120, cookie 0xbae6000
Mon Sep 29 20:30:35.713443 MSK 3: (standard_bucket0) DCP (Consumer) eq_dcpq:replication:ns_1@172.27.33.11->ns_1@172.27.33.10:standard_bucket0 - Disconnecting because noop message has no been received for 40 seconds
Mon Sep 29 20:30:36.329697 MSK 3: (standard_bucket0) Notified the completion of checkpoint persistence for vbucket 357, id 120, cookie 0xba59800
Mon Sep 29 20:30:36.330611 MSK 3: (standard_bucket0) Notified the completion of checkpoint persistence for vbucket 364, id 117, cookie 0xba5b300
Mon Sep 29 20:30:36.335938 MSK 3: (standard_bucket0) Notified the completion of checkpoint persistence for vbucket 378, id 62, cookie 0xbaa5b00


Because on my side I see:

[ns_server:debug,2014-09-29T20:30:34.369,ns_1@172.27.33.10:<0.4028.0>:ns_vbucket_mover:spawn_workers:326]Got actions: []
[rebalance:debug,2014-09-29T20:30:34.370,ns_1@172.27.33.10:<0.19034.0>:janitor_agent:handle_call:812]Going to wait for persistence of seqno 100 in vbucket 320
[ns_server:info,2014-09-29T20:30:34.654,ns_1@172.27.33.10:ns_config_rep<0.296.0>:ns_config_rep:do_pull:343]Pulling config from: 'ns_1@172.27.33.11'

[ns_server:debug,2014-09-29T20:30:35.723,ns_1@172.27.33.10:dcp_consumer_conn-standard_bucket0-ns_1@172.27.33.11<0.4673.0>:dcp_proxy:handle_info:88]Socket #Port<0.9579> was closed. Closing myself. State = {state,
                                                          #Port<0.9579>,
                                                          {consumer,
                                                           "replication:ns_1@172.27.33.11->ns_1@172.27.33.10:standard_bucket0",
                                                           'ns_1@172.27.33.10',
Comment by Aleksey Kondratenko [ 29/Sep/14 ]
Both log fragments are from node .10 which is receiver of all missing vbuckets.




[MB-12270] Cbbackup 2.5.1 Run Directly Against File System May Fail w/OS Error 2 If Files Are Concurrently Compacted Created: 29/Sep/14  Updated: 30/Sep/14

Status: Open
Project: Couchbase Server
Component/s: tools
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Morrie Schreibman Assignee: Bin Cui
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: 40h
Time Spent: Not Specified
Original Estimate: 40h

Attachments: Text File cbbackup_error.out.txt     Text File ls_begin.out.txt     Text File ls_end.out.txt    
Issue Links:
Relates to
Triage: Untriaged
Operating System: Centos 64-bit
Is this a Regression?: No

 Description   
Customer is running cbbackup directly against the bucket datafiles in order to avoid TAP vbucket dumps; cbbackups are being executed this way in order to avoid significant changes to bucket residency percentages which negatively affect application performance. The general form of the command being used is:
.
    cbbackup couchstore-files:///&lt;bucket data path> <backup_dir> -u <admin> -p <password>
.

Customer is performing these backups against very large nodes. Customer reports that the cbbackups invariable fail on an OS error2 exception (file not found). Further investigation shows that the missing file is present when cbbackup begins but gets compacted at some point while the backup is running with the result that there is a different version of the data file in play when cbbackup attempts to back it up. Attached find a listing of the directory as it exists when cbbackup starts, the error message and a second listing of the directory taken after the missing file exception has been trapped. Clearly it shows that the missing file "166.couch.2029" exists when cbbackup starts but has been replaced by file "166.couch.2030" by the time cbbackup tries to read it.

While an apparent workaround might be to suspend compaction while they are doing backups, that might result in other performance problems and also cause file system exhaustion. So we would like to see if there can be a code fix for this.

Attached: directory listings before and after OS 2 missing file exception showing changed incarnation and the error message itself.
  

 Comments   
Comment by Steve Yen [ 30/Sep/14 ]
> Customer is running cbbackup directly against the bucket datafiles in order to avoid TAP vbucket
> dumps; cbbackups are being executed this way in order to avoid significant changes to bucket
> residency percentages which negatively affect application performance.

another idea for this issue is to improve DCP to avoid radically changing the residency percentages, so that customer can use DCP based cbbackup.




[MB-12269] Tunable memory causes issues at 100mb bucket size Created: 26/Sep/14  Updated: 29/Sep/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Perry Krug Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2014-09-26 at 3.33.34 PM.png    
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
I created a bucket with only 100mb of RAM per node on a 4-node cluster running 3.0-1209

I then attempted to put about 800mb of data into that bucket using XDCR from another cluster.

I observed that the small bucket quickly filled up, the mem_used went above the high_water_mark and it started throwing tmp_ooms. I also noticed that the "metadata in RAM" was showing 0 whereas the "user data in RAM" was showing >200mb. I also noticed that the "active vbucket user data in RAM" is only ~600kb, yet the total is >200mb (see screenshot)

There seem to be a few number of ejections going on, but I would have expected much more data to be ejected at once in order to make space. I'm also concerned about the memory accounting.

I totally realize this may be worse because the data bucket is so small...but I also want to make sure that it's not a bigger problem that we only see at this very low level.

Logs are at:
https://s3.amazonaws.com/cb-customers/perry/tunablememory/collectinfo-2014-09-26T223047-ns_1%40ec2-54-176-96-196.us-west-1.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/tunablememory/collectinfo-2014-09-26T223047-ns_1%40ec2-54-193-227-17.us-west-1.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/tunablememory/collectinfo-2014-09-26T223047-ns_1%40ec2-54-219-244-195.us-west-1.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/tunablememory/collectinfo-2014-09-26T223047-ns_1%40ec2-54-241-198-57.us-west-1.compute.amazonaws.com.zip

 Comments   
Comment by Perry Krug [ 29/Sep/14 ]
In a similar test, I'm seeing problems even at 500mb bucket size. Basically I'm running the following pillow-fight load against a single node with "metadata ejection" turned on:
cbc-pillowfight -h ec2-54-177-0-243.us-west-1.compute.amazonaws.com -b default -C HTTP -m 4096 -M 4096 -I 1000000 --loop -n

And seeing tmp_ooms being returned. Less than at 100mb bucket size, but still would not expect any
Comment by Chiyoung Seo [ 29/Sep/14 ]
You can still get tmp_oom errors even with the full ejection mode if the client does write-heavy load or initial load. Disk is always slower than the frontend operations in this case.
Comment by Perry Krug [ 29/Sep/14 ]
Yes of course, but it would seem that the ejection process is not running fast enough since it comes up and down in spikes.

I'm seeing a relatively small disk write queue, certainly well below what needs to be held in RAM...
Comment by Chiyoung Seo [ 29/Sep/14 ]
We can't expect the item will be ejected immediately after the persistence. The item pager has to visit and scan each vbucket (1024 in this case as there is only one node). This will take some time even if it is in-memory scanning.




[MB-12268] Carrier publication in 3.0 causes upto 200ms delay after first nmv is received during a rebalance Created: 26/Sep/14  Updated: 29/Sep/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket, ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Perry Krug Assignee: Abhinav Dangeti
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Doing some load testing with 3.0 exposed a potentially serious performance issue during rebalance.

When a client (in this case libcouchbase via pillowfight) receives the first "not my vbucket", it took nearly 200ms before it was able to get a config via carrier publication that had the correct vbucket in it.

From the application's perspective, 2.5.1 had only a few 'ms' delay on each missed operation during rebalance, and with 3.0 it is receiving up to 200ms. We need further investigation, but it seems like a regression.

Logs from a pillowfight run:
1 15408ms [I0] {3376} [INFO] (confmon - L:169) Setting new configuration. Received via CCCP
   2 15408ms [I0] {3376} [INFO] (bootstrap - L:135) Got async step callback..
   3 15408ms [I0] {3376} [DEBUG] (retryq - L:305) Adding PKT=0x7f4318067b90 to retry queue. Try count=1
   4 15408ms [I0] {3376} [TRACE] (retryq - L:167) Next tick in 14 ms
   5 15408ms [I0] {3376} [DEBUG] (bootstrap - L:38) Instance configured!
   6 15408ms [I0] {3376} [INFO] (newconfig - L:59) Config Diff: [ vBuckets Modified=1 ], [Sequence Changed=0]
   7 15408ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-204-236-170-174.us-west-1.compute.amazonaws.com:11210 (0xc51100). OldIndex=0. NewIndex=0
   8 15408ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-116-129.us-west-1.compute.amazonaws.com:11210 (0xc52090). OldIndex=1. NewIndex=1
   9 15408ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-171-142.us-west-1.compute.amazonaws.com:11210 (0xc530a0). OldIndex=2. NewIndex=2
  10 15408ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-213-125.us-west-1.compute.amazonaws.com:11210 (0xc540b0). OldIndex=3. NewIndex=3
  11 15408ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-177-26-252.us-west-1.compute.amazonaws.com:11210 (0xc550c0). OldIndex=4. NewIndex=4
  12 15408ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-219-88-7.us-west-1.compute.amazonaws.com:11210 (0xc560d0). OldIndex=5. NewIndex=5
  13 15424ms [I0] {3376} [WARN] (server - L:125) <ec2-54-176-116-129.us-west-1.compute.amazonaws.com:11210> (SRV=0xc52090,IX=1) NOT_MY_VBUCKET. Packet=0x7f4318067b90 (S=31359). VBID=550
  14 15426ms [I0] {3376} [INFO] (confmon - L:161) Not applying configuration received via CCCP. No changes detected
  15 15426ms [I0] {3376} [DEBUG] (retryq - L:305) Adding PKT=0x7f4318070370 to retry queue. Try count=2
  16 15426ms [I0] {3376} [TRACE] (retryq - L:167) Next tick in 29 ms
  17 15460ms [I0] {3376} [WARN] (server - L:125) <ec2-54-176-116-129.us-west-1.compute.amazonaws.com:11210> (SRV=0xc52090,IX=1) NOT_MY_VBUCKET. Packet=0x7f4318070370 (S=31359). VBID=550
  18 15461ms [I0] {3376} [INFO] (confmon - L:169) Setting new configuration. Received via CCCP
  19 15461ms [I0] {3376} [INFO] (bootstrap - L:135) Got async step callback..
  20 15461ms [I0] {3376} [DEBUG] (retryq - L:305) Adding PKT=0x7f43180f4c20 to retry queue. Try count=3
  21 15461ms [I0] {3376} [TRACE] (retryq - L:167) Next tick in 44 ms
  22 15461ms [I0] {3376} [DEBUG] (bootstrap - L:38) Instance configured!
  23 15461ms [I0] {3376} [INFO] (newconfig - L:59) Config Diff: [ vBuckets Modified=1 ], [Sequence Changed=0]
  24 15461ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-204-236-170-174.us-west-1.compute.amazonaws.com:11210 (0xc51100). OldIndex=0. NewIndex=0
  25 15462ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-116-129.us-west-1.compute.amazonaws.com:11210 (0xc52090). OldIndex=1. NewIndex=1
  26 15462ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-171-142.us-west-1.compute.amazonaws.com:11210 (0xc530a0). OldIndex=2. NewIndex=2
  27 15462ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-213-125.us-west-1.compute.amazonaws.com:11210 (0xc540b0). OldIndex=3. NewIndex=3
  28 15462ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-177-26-252.us-west-1.compute.amazonaws.com:11210 (0xc550c0). OldIndex=4. NewIndex=4
  29 15462ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-219-88-7.us-west-1.compute.amazonaws.com:11210 (0xc560d0). OldIndex=5. NewIndex=5
  30 15507ms [I0] {3376} [WARN] (server - L:125) <ec2-54-176-116-129.us-west-1.compute.amazonaws.com:11210> (SRV=0xc52090,IX=1) NOT_MY_VBUCKET. Packet=0x7f43180f4c20 (S=31359). VBID=550
  31 15509ms [I0] {3376} [INFO] (confmon - L:161) Not applying configuration received via CCCP. No changes detected
  32 15509ms [I0] {3376} [DEBUG] (retryq - L:305) Adding PKT=0x7f4318067c10 to retry queue. Try count=4
  33 15509ms [I0] {3376} [TRACE] (retryq - L:167) Next tick in 59 ms
  34 15570ms [I0] {3376} [WARN] (server - L:125) <ec2-54-176-116-129.us-west-1.compute.amazonaws.com:11210> (SRV=0xc52090,IX=1) NOT_MY_VBUCKET. Packet=0x7f4318067c10 (S=31359). VBID=550
  35 15572ms [I0] {3376} [INFO] (confmon - L:169) Setting new configuration. Received via CCCP
  36 15572ms [I0] {3376} [INFO] (bootstrap - L:135) Got async step callback..
  37 15572ms [I0] {3376} [DEBUG] (retryq - L:305) Adding PKT=0x7f43180f50d0 to retry queue. Try count=5
  38 15572ms [I0] {3376} [TRACE] (retryq - L:167) Next tick in 74 ms
  39 15572ms [I0] {3376} [DEBUG] (bootstrap - L:38) Instance configured!
  40 15572ms [I0] {3376} [INFO] (newconfig - L:59) Config Diff: [ vBuckets Modified=1 ], [Sequence Changed=0]
  41 15572ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-204-236-170-174.us-west-1.compute.amazonaws.com:11210 (0xc51100). OldIndex=0. NewIndex=0
  42 15572ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-116-129.us-west-1.compute.amazonaws.com:11210 (0xc52090). OldIndex=1. NewIndex=1
  43 15572ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-171-142.us-west-1.compute.amazonaws.com:11210 (0xc530a0). OldIndex=2. NewIndex=2
  44 15572ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-213-125.us-west-1.compute.amazonaws.com:11210 (0xc540b0). OldIndex=3. NewIndex=3
  45 15572ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-177-26-252.us-west-1.compute.amazonaws.com:11210 (0xc550c0). OldIndex=4. NewIndex=4
  46 15572ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-219-88-7.us-west-1.compute.amazonaws.com:11210 (0xc560d0). OldIndex=5. NewIndex=5
  47 15648ms [I0] {3376} [WARN] (server - L:125) <ec2-54-176-116-129.us-west-1.compute.amazonaws.com:11210> (SRV=0xc52090,IX=1) NOT_MY_VBUCKET. Packet=0x7f43180f50d0 (S=31359). VBID=550
  48 15649ms [I0] {3376} [INFO] (confmon - L:169) Setting new configuration. Received via CCCP
  49 15649ms [I0] {3376} [INFO] (bootstrap - L:135) Got async step callback..
  50 15649ms [I0] {3376} [DEBUG] (retryq - L:305) Adding PKT=0x7f43180704c0 to retry queue. Try count=6
  51 15649ms [I0] {3376} [TRACE] (retryq - L:167) Next tick in 89 ms
  52 15649ms [I0] {3376} [DEBUG] (bootstrap - L:38) Instance configured!
  53 15649ms [I0] {3376} [INFO] (newconfig - L:59) Config Diff: [ vBuckets Modified=1 ], [Sequence Changed=0]
  54 15649ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-204-236-170-174.us-west-1.compute.amazonaws.com:11210 (0xc51100). OldIndex=0. NewIndex=0
  55 15649ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-116-129.us-west-1.compute.amazonaws.com:11210 (0xc52090). OldIndex=1. NewIndex=1
  56 15649ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-171-142.us-west-1.compute.amazonaws.com:11210 (0xc530a0). OldIndex=2. NewIndex=2
  57 15650ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-176-213-125.us-west-1.compute.amazonaws.com:11210 (0xc540b0). OldIndex=3. NewIndex=3
  58 15650ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-177-26-252.us-west-1.compute.amazonaws.com:11210 (0xc550c0). OldIndex=4. NewIndex=4
  59 15650ms [I0] {3376} [INFO] (newconfig - L:148) Reusing server ec2-54-219-88-7.us-west-1.compute.amazonaws.com:11210 (0xc560d0). OldIndex=5. NewIndex=5

 Comments   
Comment by Perry Krug [ 26/Sep/14 ]
Unfortunately I wasn't able to gather the server logs before my environment shut down. Can you let me know if we log verbosely enough about the cccp update for the logs to be useful?

This isn't an edge case, it happens everytime with 3.0 so I think it should be easy enough to reproduce:
-4 nodes, beer-sample bucket
-any amount of pillowfight load seems to exhibit this behavior, I've been using: cbc-pillowfight -U couchbase://ec2-54-219-88-7.us-west-1.compute.amazonaws.com/beer-sample?error_thresh_delay=0.5 --num-threads 1 --min-size 1024 --max-size 1024 -r 50 -Q 1 -I 300000 -n --loop

Let me know if you need another reproduction for server logs
Comment by Mark Nunberg [ 26/Sep/14 ]
Also bug me if you need help understanding something in the client logs
Comment by Abhinav Dangeti [ 29/Sep/14 ]
Perrry, no there isn't any logging in this respect on memcached/ep-engine side, or the ns_server side.
I added a debug log, in the ep-engine, whenever the cluster configuration is updated by the ns_server: http://review.couchbase.org/#/c/41746.
If possible, could you increase the verbosity, with the new build, once its ready and get the logs.
Comment by Abhinav Dangeti [ 29/Sep/14 ]
Perry, I noted the delay in receiving the updated cluster configuration on 3.0.1. However, I found that these 200ms and sometimes 200+ ms delays in getting the right carrier configuration is not only seen in 3.0.1, but in 2.5.1 also. The following are pillow fight logs from 2.5.1:

48855ms [I0] {4355} [WARN] (server - L:125) <localhost:12002> (SRV=0x7f860b000f60,IX=1) NOT_MY_VBUCKET. Packet=0x7f860c0c57a0 (S=872661). VBID=492
48856ms [I0] {4355} [INFO] (confmon - L:161) Not applying configuration received via CCCP. No changes detected
48856ms [I0] {4355} [DEBUG] (retryq - L:305) Adding PKT=0x7f8609f07260 to retry queue. Try count=1
48856ms [I0] {4355} [TRACE] (retryq - L:167) Next tick in 14 ms
48871ms [I0] {4355} [WARN] (server - L:125) <localhost:12002> (SRV=0x7f860b000f60,IX=1) NOT_MY_VBUCKET. Packet=0x7f8609f07260 (S=872661). VBID=492
48871ms [I0] {4355} [INFO] (confmon - L:161) Not applying configuration received via CCCP. No changes detected
48871ms [I0] {4355} [DEBUG] (retryq - L:305) Adding PKT=0x7f8609c11470 to retry queue. Try count=2
48871ms [I0] {4355} [TRACE] (retryq - L:167) Next tick in 29 ms
48902ms [I0] {4355} [WARN] (server - L:125) <localhost:12002> (SRV=0x7f860b000f60,IX=1) NOT_MY_VBUCKET. Packet=0x7f8609c11470 (S=872661). VBID=492
48902ms [I0] {4355} [INFO] (confmon - L:161) Not applying configuration received via CCCP. No changes detected
48902ms [I0] {4355} [DEBUG] (retryq - L:305) Adding PKT=0x7f860b2055c0 to retry queue. Try count=3
48902ms [I0] {4355} [TRACE] (retryq - L:167) Next tick in 44 ms
48948ms [I0] {4355} [WARN] (server - L:125) <localhost:12002> (SRV=0x7f860b000f60,IX=1) NOT_MY_VBUCKET. Packet=0x7f860b2055c0 (S=872661). VBID=492
48949ms [I0] {4355} [INFO] (confmon - L:161) Not applying configuration received via CCCP. No changes detected
48949ms [I0] {4355} [DEBUG] (retryq - L:305) Adding PKT=0x7f860b205560 to retry queue. Try count=4
48949ms [I0] {4355} [TRACE] (retryq - L:167) Next tick in 59 ms
49010ms [I0] {4355} [WARN] (server - L:125) <localhost:12002> (SRV=0x7f860b000f60,IX=1) NOT_MY_VBUCKET. Packet=0x7f860b205560 (S=872661). VBID=492
49011ms [I0] {4355} [INFO] (confmon - L:161) Not applying configuration received via CCCP. No changes detected
49011ms [I0] {4355} [DEBUG] (retryq - L:305) Adding PKT=0x7f860b200660 to retry queue. Try count=5
49011ms [I0] {4355} [TRACE] (retryq - L:167) Next tick in 74 ms
49087ms [I0] {4355} [WARN] (server - L:125) <localhost:12002> (SRV=0x7f860b000f60,IX=1) NOT_MY_VBUCKET. Packet=0x7f860b200660 (S=872661). VBID=492
49089ms [I0] {4355} [INFO] (confmon - L:169) Setting new configuration. Received via CCCP

The updated cluster configuration is available on the ep-engine only once the configuration is actually updated by the ns_server.

Clearly this isn't a regression. Would you agree?
Comment by Perry Krug [ 29/Sep/14 ]
It would seem that you are correct. My understanding from Mark is that this delay, combined with some new logic in the 2.4.2 libcouchbase is what contributed to a much more noticeable effect in the latest versions.

Either way, it seems like something we need to address no?
Comment by Mark Nunberg [ 29/Sep/14 ]
After digging into this issue a bit more, it would seem we are getting outdated configs. This wasn't as prominent in earlier versions of the library because the library would end up using the fast-forward vbucket map OR randomly select a node. This would increase throughput, at the cost of increased network I/O.

With the advent of CCCP where the client is directly given a new config which it is supposed to use, the library does not inspect the fast forward map, because it trusts that the config provided to it is the current config. This is especially true in this log where we see that the client _does_ get new configurations, but none of those configurations actually contain the up-to-date data, which is actually found within the ffmap.

If I patch the library to inspect the fast forward map and use the old < 2.4 "not my vbucket" behavior, the performance goes back to normal. However I'm not sure if this is intended behavior or design.
Comment by Abhinav Dangeti [ 29/Sep/14 ]
ns_server does not update the cluster configuration until all the checkpoints from the old master have been persisted in the new master.
So if there was a constant load running at during rebalance, the client could receive a NOT_MY_VBUCKET response until all the delta mutations have been persisted on the new master.

I would say, this is more or less the expected behavior.
Comment by Mark Nunberg [ 29/Sep/14 ]
I think it's reasonable that there be a document describing exactly how the client is expected to locate a node, and how the client is expected to interpret embedded configs within not-my-vbucket responses.




[MB-12267] 95% GET Latency Regression Created: 26/Sep/14  Updated: 26/Sep/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0, 3.0-Beta
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Thomas Anderson Assignee: Thomas Anderson
Resolution: Unresolved Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: test cluster ares: 4 node; CentOS 6.5; 24 core/64GB ram;2.3GHz; RAID 10 HDD. separate client, 10 core/60GB/2.8GHz;

Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: 2.5.1:: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_251-1083_f35_access
3.0.0:: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_300-1209_6b1_access
3.0.1:: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_301-1330_e1e_access
Is this a Regression?: Yes

 Description   
test for 95% GET latency - 1Bucket,150M documentsx2K/doc, DGM, 6Kops, 20%cache miss ration.
regression of ~15% compared to 2.5.1

 Comments   
Comment by Chiyoung Seo [ 26/Sep/14 ]
Can you please provide the hardware configurations, such as number of cores, disk type (e.g., SSD, HDD, RAID SSD, RAID HDD), physical RAM size?
Comment by Thomas Anderson [ 26/Sep/14 ]
rerun with 3.0.1-1330. regression in 3.0.0 all builds compared with 3.5.1 may have been resolved, as this latest build 1330 shows no regression. will lower priority leave assigned for performance to retest with subsequent 3.0.1 builds.. its priority is the need to recheck and examine. no known change targetted this improved behavior.





[MB-12266] Batch size for View Query is not tuned correctly Created: 26/Sep/14  Updated: 30/Sep/14

Status: Open
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Thomas Anderson Assignee: Sriram Melkote
Resolution: Unresolved Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos6.

Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: 3.0.0-1209::http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=leto_ssd_300-1209_ab3_access
2.5.1-1083::http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=leto_ssd_251-1083_aa9_access
cluster cbcollectinfo details for 3.0.1-1330::
http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=leto_ssd_301-1330_a48_access
http://ci.sc.couchbase.com/job/leto/610/artifact/172.23.100.29.zip
http://ci.sc.couchbase.com/job/leto/610/artifact/172.23.100.30.zip
http://ci.sc.couchbase.com/job/leto/610/artifact/172.23.100.31.zip
http://ci.sc.couchbase.com/job/leto/610/artifact/172.23.100.32.zip
Is this a Regression?: Yes

 Description   
We need to tune the batch-size for the view-query.

 Comments   
Comment by Volker Mische [ 29/Sep/14 ]
Where's the regression? I fail to see it.

BTW, you can display several results on a single page: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=leto_ssd_300-1209_ab3_access&snapshot=leto_ssd_251-1083_aa9_access
Comment by Thomas Anderson [ 29/Sep/14 ]
the regression is that i 2.5.1 the View Query Thruput is 2189/queries per sec; while 3.0.0-1209 is 1639. or a drop in Qps of 24%.
note: this level of Qps slowdown is recorded across all 3.x builds. from the combined graphs, latency for 2.5.1 averages ~10ms, and 3.x avgs. ~12.5ms.
also noted, that latency (especially the SET case) increases much greater than linear for the last 1/10th of a percent of the total workload, most of the differential between releases. what is happening in that last 45 seconds of of a 3600 second test run. all this while seeing a 20x reduction in disk update time.
Comment by Volker Mische [ 30/Sep/14 ]
Thomas, guess you got the numbers from ShowFast [1] "View Query" -> "Troughput".

When was the 2.5.1 run performed? Would you mind running it again in case it wasn't recently. I remember hardware changes (SSDs) which could change the results.

I fail to see what this has to do with the batch size of the indexing (I guess that's what you mean, else I don't know what you mean with "batch-size" for the view-query.

The reason for less throughput is mostly caused by DCP as getting the needed information straight from disk is faster than getting it through TCP.

[1]: http://showfast.sc.couchbase.com/#/timeline




[MB-12265] Enhance cbbackup test suite on performance and footprint measurement Created: 25/Sep/14  Updated: 25/Sep/14

Status: Open
Project: Couchbase Server
Component/s: test-execution
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Bin Cui Assignee: Wayne Siu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
See CBSE-1407.

We missed the test coverage for cbbackup tool on the following areas:
1. On DGM scenario, watch the backup throughput performance
2. On DGM scenario, watch the memory consumption for cbbackup process.




[MB-12263] collect_info should try 11209 if failure on 11210 Created: 25/Sep/14  Updated: 25/Sep/14

Status: Open
Project: Couchbase Server
Component/s: tools
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Minor
Reporter: James Mauss Assignee: Todd Greenstein
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
If there are issues/failures with connecting to memcached on 11210, the collect_info script should also try 11209.

This will allow us to collect logs and memcached stats if there are too many open connections on 11210.




[MB-12262] More updater stats Created: 25/Sep/14  Updated: 25/Sep/14

Status: Open
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Patrick Varley Assignee: Sriram Melkote
Resolution: Unresolved Votes: 0
Labels: supportability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We have some great information about what the updater has updated, however we have no details on how much work it did to do those updates i.e: how many documents it read, how big are those documents were and how much data it wrote. As a result it is hard to tell if there is a performance issue with the updater itself or the hardware underneath.

Here are some suggestions on new stats that I think might be useful (I do not know the code so these might be silly suggestions):

time spent reading data from disk
time spent processing in the v8 engine
time spent writing data to disk
amount of data read from disk
amount of data written to disk
number of documents read/processed

What the current stats look like:

[couchdb:info,2014-09-20T21:04:07.105,ns_1@10.0.0.1:<0.17156.6231>:couch_log:info:39]Set view `Patrick`, main group `_design/dev_test`, updater finished
Indexing time: 0.119 seconds
Blocked time: 0.000 seconds
Inserted IDs: 1
Deleted IDs: 4
Inserted KVs: 1
Deleted KVs: 1
Cleaned KVs: 0
# seqs done: 5




[MB-12261] Stop server on failover Created: 25/Sep/14  Updated: 26/Sep/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 2.5.1
Fix Version/s: bug-backlog
Security Level: Public

Type: Improvement Priority: Major
Reporter: James Mauss Assignee: Anil Kumar
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
It would be nice to have the option of having the processes stop on a node when it is failed over.

The problem we have seen is that an app uses an array of hosts to connect to and since the array is instantiated at start time of our app it includes the now removed node which replies to requests from the application even though it now not part of the cluster.

 Comments   
Comment by Patrick Varley [ 26/Sep/14 ]
@James, it would be good to get some more detail on this one? When I tested this with LCB it behaved the way I expect.
I had similar issue with a node that is not apart of a cluster see MB-11875.




[MB-12260] Secondary Indexes (2i) exposed as a replacement to Views for lookup indexes Created: 25/Sep/14  Updated: 25/Sep/14  Resolved: 25/Sep/14

Status: Closed
Project: Couchbase Server
Component/s: clients, secondary-index
Affects Version/s: cbq-DP3
Fix Version/s: sherlock
Security Level: Public

Type: Bug Priority: Major
Reporter: Cihan Biyikoglu Assignee: Sriram Melkote
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
goal; 2i is a slide in replacement for views for the lookup index case.
this is a placeholder item tracking the work needed for this goal.

open items:
- can we validate that we won't take away bandwidth from 2i in n1ql or some other team beyond SDK team.
  Next step: detail for the API. Is the impact contained?
- can we validate that 2i is a slide in replacement for views
  Next step: which specific use case can we replace? Is it a slide in replacement for existing apps or just simpler porting?
- detail how and when we would recommend view vs secondary index?

 Comments   
Comment by Cihan Biyikoglu [ 25/Sep/14 ]
dup 12237




[MB-12259] data loss if reboot right after online upgrade 2.0.1->3.0.1 Created: 25/Sep/14  Updated: 30/Sep/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket, ns_server
Affects Version/s: 3.0.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Iryna Mironava Assignee: Mike Wiederhold
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.1-1326-rel


 Description   
test to reproduce:
-t newupgradetests.MultiNodesUpgradeTests.online_upgrade_rebalance_in_out,initial_version=2.0.1-170-rel,initial_build_type=community,reboot_cluster=true,upgrade_version=3.0.1-1326-rel,initial_vbuckets=512

1)initial vbuckets number= 512, 2 nodes with 2.0.1, 1 default bucket, 1000 items
2)added two nodes with 3.0.1, after that removed 2.0.1 nodes
3)reboot both nodes, wait for warmup
4)2 items are lost: upgrade45 and upgrade927 keys, 261 vbucket
vbucket is empty
[root@kiwi-r112 default]# /opt/couchbase/bin/couch_dbinfo 261.couch.1
DB Info (261.couch.1) - header at 12288
   file format version: 11
   update_seq: 0
   no documents
   B-tree size: 0 bytes
   total disk size: 12.0 kB
[root@kiwi-r112 default]#

will attach cbcollects(.10, .12 -are 2.0.1 nodes, .11, .13 - 3.0.1)

https://s3.amazonaws.com/bugdb/jira/MB-12259/dca3be89/172.27.33.10-9252014-1827-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dca3be89/172.27.33.11-9252014-1829-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dca3be89/172.27.33.12-9252014-1828-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dca3be89/172.27.33.13-9252014-1830-diag.zip

 Comments   
Comment by Aleksey Kondratenko [ 26/Sep/14 ]
Upgrading to blocker.

Do you know if items "survived" rebalance ? I.e. we need to know if they got lost during rebalance or during restart.

In logs I see successful and completely normal shutdown followed by startup. So ep-engine was certainly supposed to preserve all items.
Comment by Aleksey Kondratenko [ 26/Sep/14 ]
Notably rebalance was TAP while indexes used DCP to consume docs on 3.x nodes. Could be relevant.
Comment by Aleksey Kondratenko [ 26/Sep/14 ]
I need you to reproduce the test while checking for items _after every step_.

And please consider doing it all the time.
Comment by Iryna Mironava [ 28/Sep/14 ]
stats after each step:
1) stats before upgrade - 1000 items, 1000 items in replica
172.27.33.12
 curr_items: 500
 curr_items_tot: 1000
 curr_temp_items: 0
 ep_access_scanner_num_items: 0
 ep_chk_max_items: 5000
 ep_diskqueue_items: 0
 ep_items_rm_from_checkpoints: 0
 ep_total_del_items: 0
 ep_total_new_items: 1000
 ep_uncommitted_items: 0
 ep_warmup_min_items_threshold: 100
 vb_active_curr_items: 500
 vb_active_num_ref_items: 500
 vb_pending_curr_items: 0
 vb_pending_num_ref_items: 0
 vb_replica_curr_items: 500
 vb_replica_num_ref_items: 500

172.27.33.10
 curr_items: 500
 curr_items_tot: 1000
 curr_temp_items: 0
 ep_access_scanner_num_items: 0
 ep_chk_max_items: 5000
 ep_diskqueue_items: 0
 ep_items_rm_from_checkpoints: 0
 ep_total_del_items: 0
 ep_total_new_items: 1000
 ep_uncommitted_items: 0
 ep_warmup_min_items_threshold: 100
 vb_active_curr_items: 500
 vb_active_num_ref_items: 500
 vb_pending_curr_items: 0
 vb_pending_num_ref_items: 0
 vb_replica_curr_items: 500
 vb_replica_num_ref_items: 500

2) after rebalance in 3.0.1 nodes - 1000 items, 1000 of replica
 
172.27.33.12
 curr_items: 267
 curr_items_tot: 507
 curr_temp_items: 0
 ep_access_scanner_num_items: 0
 ep_chk_max_items: 5000
 ep_diskqueue_items: 0
 ep_items_rm_from_checkpoints: 44
 ep_total_del_items: 0
 ep_total_new_items: 1000
 ep_uncommitted_items: 0
 ep_warmup_min_items_threshold: 100
 vb_active_curr_items: 267
 vb_active_num_ref_items: 267
 vb_pending_curr_items: 0
 vb_pending_num_ref_items: 0
 vb_replica_curr_items: 240
 vb_replica_num_ref_items: 240

172.27.33.13
 curr_items: 232
 curr_items_tot: 486
 curr_temp_items: 0
 ep_access_scanner_num_items: 0
 ep_chk_max_items: 500
 ep_diskqueue_items: 0
 ep_items_rm_from_checkpoints: 481
 ep_total_del_items: 0
 ep_total_new_items: 486
 ep_uncommitted_items: 0
 ep_warmup_min_items_threshold: 100
 vb_active_curr_items: 232
 vb_pending_curr_items: 0
 vb_replica_curr_items: 254

172.27.33.10
 curr_items: 269
 curr_items_tot: 515
 curr_temp_items: 0
 ep_access_scanner_num_items: 0
 ep_chk_max_items: 5000
 ep_diskqueue_items: 0
 ep_items_rm_from_checkpoints: 44
 ep_total_del_items: 0
 ep_total_new_items: 1000
 ep_uncommitted_items: 0
 ep_warmup_min_items_threshold: 100
 vb_active_curr_items: 269
 vb_active_num_ref_items: 269
 vb_pending_curr_items: 0
 vb_pending_num_ref_items: 0
 vb_replica_curr_items: 246
 vb_replica_num_ref_items: 246

172.27.33.11
 curr_items: 232
 curr_items_tot: 492
 curr_temp_items: 0
 ep_access_scanner_num_items: 0
 ep_chk_max_items: 500
 ep_diskqueue_items: 0
 ep_items_rm_from_checkpoints: 464
 ep_total_del_items: 0
 ep_total_new_items: 492
 ep_uncommitted_items: 0
 ep_warmup_min_items_threshold: 100
 vb_active_curr_items: 232
 vb_pending_curr_items: 0
 vb_replica_curr_items: 260

3) rebalance out 2.0.1 - 1000 items, but 996 of replica, still can see all items


172.27.33.13
 curr_items: 500
 curr_items_tot: 996
 curr_temp_items: 0
 ep_access_scanner_num_items: 0
 ep_chk_max_items: 500
 ep_diskqueue_items: 3
 ep_items_rm_from_checkpoints: 1007
 ep_total_del_items: 0
 ep_total_new_items: 993
 ep_uncommitted_items: 0
 ep_warmup_min_items_threshold: 100
 vb_active_curr_items: 500
 vb_pending_curr_items: 0
 vb_replica_curr_items: 496

172.27.33.11
 curr_items: 500
 curr_items_tot: 1000
 curr_temp_items: 0
 ep_access_scanner_num_items: 0
 ep_chk_max_items: 500
 ep_diskqueue_items: 6
 ep_items_rm_from_checkpoints: 990
 ep_total_del_items: 0
 ep_total_new_items: 994
 ep_uncommitted_items: 0
 ep_warmup_min_items_threshold: 100
 vb_active_curr_items: 500
 vb_pending_curr_items: 0
 vb_replica_curr_items: 500

4) after warm up: 996 items and 996 of replica

172.27.33.13
 curr_items: 500
 curr_items_tot: 996
 curr_temp_items: 0
 ep_access_scanner_num_items: 0
 ep_chk_max_items: 500
 ep_diskqueue_items: 0
 ep_items_rm_from_checkpoints: 0
 ep_total_del_items: 0
 ep_total_new_items: 0
 ep_uncommitted_items: 0
 ep_warmup_min_items_threshold: 100
 vb_active_curr_items: 500
 vb_pending_curr_items: 0
 vb_replica_curr_items: 496

172.27.33.11
 curr_items: 496
 curr_items_tot: 996
 curr_temp_items: 0
 ep_access_scanner_num_items: 0
 ep_chk_max_items: 500
 ep_diskqueue_items: 0
 ep_items_rm_from_checkpoints: 0
 ep_total_del_items: 0
 ep_total_new_items: 0
 ep_uncommitted_items: 0
 ep_warmup_min_items_threshold: 100
 vb_active_curr_items: 496
 vb_pending_curr_items: 0
 vb_replica_curr_items: 500

new logs :
https://s3.amazonaws.com/bugdb/jira/MB-12259/dc4r5e89/172.27.33.10-9282014-174-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dc4r5e89/172.27.33.11-9282014-175-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dc4r5e89/172.27.33.12-9282014-174-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dc4r5e89/172.27.33.13-9282014-176-diag.zip
Comment by Aleksey Kondratenko [ 29/Sep/14 ]
Stats is not helpful enough way. I need to know if items actually exist. You could for example backup .couch files. In addition to just doing plain GETs to see if everything is present or not.
Comment by Aleksey Kondratenko [ 29/Sep/14 ]
Given that presented evidence (however poor) suggests that data was lost after rebalance I'm going to pass it to ep-engine team.
Comment by Iryna Mironava [ 30/Sep/14 ]
vbuckets after rebalance out 2.5 nodes(.11 and .13):
https://s3.amazonaws.com/bugdb/jira/MB-12259/dca3be89/step3_11.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dca3be89/step3_13.zip
vbuckets after dcp rebalance
https://s3.amazonaws.com/bugdb/jira/MB-12259/dca3be89/step4_11.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dca3be89/step4_13.zip
vbuckets after warmup
https://s3.amazonaws.com/bugdb/jira/MB-12259/dca3be89/step5_11.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dca3be89/step5_13.zip

vbucket 292 had 2 items after rebalance out.

new cbcollects:
https://s3.amazonaws.com/bugdb/jira/MB-12259/dcft6e89/172.27.33.10-9302014-1928-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dcft6e89/172.27.33.11-9302014-1929-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dcft6e89/172.27.33.12-9302014-1928-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12259/dcft6e89/172.27.33.13-9302014-1930-diag.zip




[MB-12257] n1ql + 3.0.0 primary index doesn't work after reload bucket or loading more items Created: 25/Sep/14  Updated: 29/Sep/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP3
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Iryna Mironava Assignee: Manik Taneja
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: build 3.0.0-1209
n1ql version : dp3

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
after installing couchbase I run create primary index on my_bucket
and start running queries
if I add items and run same query it keeps old result
If I delete bucket and then add bucket with same name again - for every query i see error: {u'code': 5000, u'message': u'Bucket default not found.', u'caller': u'view_index:200', u'key': u'Internal Error'}

how can I update the index? or how can I delete it and create again? if bucket is deleted should the index be deleted also?

 Comments   
Comment by Gerald Sangudi [ 25/Sep/14 ]
Test blocker.
Comment by Manik Taneja [ 26/Sep/14 ]
In general we need a mechanism that informs the query engine whenever a bucket or an index is removed either via the UI or through another instance of a query engine. I guess these set of issues will be addressed once we have a query metadata store in place.

For now the best we can do, is to have a long poll function that checks the pool/buckets at a preconfigured frequency. Results may be inconsistent until the next pass of the pool refresh thread.
Comment by Manik Taneja [ 29/Sep/14 ]
As for the new items not showing up in the query, this is because by default the views engine uses stale=update_after which means that the views will be updated right after a query is made. So the next time you run the same query you should see the updated results.




[MB-12256] No 32bit support for 3.x on RedHat CentOS Ubuntu Linux Created: 25/Sep/14  Updated: 25/Sep/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Ian McCloy Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
The documentation pages for 3.x are incorrect.

http://docs.couchbase.com/prebuilt/couchbase-manual-3.0/Install/RHEL-install-intro.html
http://docs.couchbase.com/prebuilt/couchbase-manual-3.0/Install/Ubuntu-install.html

Both pages list 32bit and 64bit as supported, PM's documents state that only 64bit will be supported from 3.x onwards and indeed the beta is only available as 64bit.

http://www.couchbase.com/download#beta

 Comments   
Comment by Anil Kumar [ 25/Sep/14 ]
Ruth this is "blocker" needs to be fixed for 3.0GA.
Comment by Ruth Harris [ 25/Sep/14 ]
Fixed. in 3 sections: supported platforms, RHEL/CentOS, and Ubuntu

BTW,Ceej indicated that Debian 7 is supported. Te PM wiki page says 2-3 months after 3.0. Please clarify..... is Debian 7 64-bit in 3.0???
Comment by Anil Kumar [ 25/Sep/14 ]
I have fixed the wiki http://hub.internal.couchbase.com/confluence/display/PM/Couchbase+Server+-+Supported+Platforms. Debian 7 is supported in 3.0.




[MB-12255] trigger sanity test when new build available Created: 24/Sep/14  Updated: 25/Sep/14  Resolved: 25/Sep/14

Status: Resolved
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Task Priority: Major
Reporter: Thuan Nguyen Assignee: Phil Labee
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos 6, ubuntu 12.04 and windows 2008


 Description   
Phil,
Could you trigger these jobs when new build available?

Centos 6.x: http://qa.sc.couchbase.com/job/CouchbaseServer-SanityTest-4Nodes-Centos64-openssl_1/ (token: rpm-64)
Ubuntu 12.04: http://qa.sc.couchbase.com/job/CouchbaseServer-SanityTest-4Nodes-Ubuntu64-openssl_1/ (token: rpm-64)
Windows 2008: http://qa.sc.couchbase.com/job/CouchbaseServer-SanityTest-4Node-Windows_2008_x64/ (token: exe-64)

 Comments   
Comment by Phil Labee [ 25/Sep/14 ]
change to buildbot-internal: Buildbot/master.cfg

    http://review.couchbase.org/#/c/41648/

merged into github:

    * b1f7dc2585dd2f6fe0cacf5e87db4cb2f9a1f644 2014-09-25 10:07:35 -0700: MB-12255: call sanity tests after 3.0.1 builds
Comment by Phil Labee [ 25/Sep/14 ]
These tests ran on 3.0.0-1338

    centos: http://builds.hq.northscale.net:8010/builders/centos-6-x64-301-builder/builds/134/steps/trigger%20jenkins%20%5B%201%20%5D/logs/stdio

    ubuntu: http://builds.hq.northscale.net:8010/builders/ubuntu-1204-x64-301-builder/builds/130/steps/trigger%20jenkins%20%5B%201%20%5D/logs/stdio

    windows: http://builds.hq.northscale.net:8010/builders/server-30-win-x64-301/builds/138/steps/start%20test/logs/stdio




[MB-12254] Test effect of access log generation on front end ops Created: 24/Sep/14  Updated: 26/Sep/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Task Priority: Major
Reporter: Abhinav Dangeti Assignee: Venu Uppalapati
Resolution: Unresolved Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to MB-12117 Access log generation holds onto lock... Resolved

 Description   
A change was made on the engine side, that ensures we do not hold on to hash table locks when we add new entries into the mutation log.
http://review.couchbase.org/#/c/41572/

We will need to test the performance improvement this would show in terms of front end operations per sec.




[MB-12253] Add CentOS 7 as supported platform Created: 24/Sep/14  Updated: 25/Sep/14

Status: Open
Project: Couchbase Server
Component/s: build
Affects Version/s: feature-backlog
Fix Version/s: feature-backlog
Security Level: Public

Type: Improvement Priority: Major
Reporter: Anil Kumar Assignee: Anil Kumar
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to MB-11671 couchbase.log needs alternitives to n... Resolved

 Description   
We need to add support for Centos7

 Comments   
Comment by Phil Labee [ 24/Sep/14 ]
Do you mean both centos7 and ubuntu 14.04? The summary and description say different things.
Comment by Cihan Biyikoglu [ 24/Sep/14 ]
Centos7.




[MB-12251] Improve test coverage of spatial views Created: 24/Sep/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0
Fix Version/s: sherlock
Security Level: Public

Type: Task Priority: Major
Reporter: Volker Mische Assignee: Volker Mische
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Improve test coverage of the spatial views. The area to improve is the range queries with wildcards and the handling of geometries in a multi-dimensional context.




[MB-12250] Performance tests for largeDB support with full ejection Created: 24/Sep/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase Server
Component/s: None
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Cihan Biyikoglu Assignee: Anil Kumar
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
placeholder for the improvement on performance test for largeDB support

 Comments   
Comment by Cihan Biyikoglu [ 24/Sep/14 ]
Anil, pls add the test details for the tests we need for 3.0.1




[MB-12249] Revive spatial unit tests Created: 24/Sep/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0
Fix Version/s: sherlock
Security Level: Public

Type: Bug Priority: Major
Reporter: Volker Mische Assignee: Volker Mische
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
The unit tests for the spatial views were broken after the merge into 3.0. Make them work again.




[MB-12248] meta shows only ids for couchbase bucket with base64 items Created: 24/Sep/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
we also had flags and cas earlier
cbq> select meta(default) from default;
{
    "signature": {
        "$1": "object"
    },
    "results": [
        {
            "$1": {
                "id": "tuq0"
            }
        },
        {
            "$1": {
                "id": "tuq1"
            }
        },
        {
            "$1": {
                "id": "tuq2"
            }
        },
        {
            "$1": {
                "id": "tuq3"
            }
        },
        {
            "$1": {
                "id": "tuq4"
            }
        },
        {
            "$1": {
                "id": "tuq5"
            }
        },
        {
            "$1": {
                "id": "tuq6"
            }
        },
        {
            "$1": {
                "id": "tuq7"
            }
        },
        {
            "$1": {
                "id": "tuq8"
            }
        },
        {
            "$1": {
                "id": "tuq9"
            }
        }
    ],
    "state": "success",
    "metrics": {
        "elapsedTime": "28.142ms",
        "executionTime": "28.073ms",
        "resultCount": 10
    }
}





[MB-12247] [Windows]: view engine failed to index doc in query Created: 24/Sep/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Meenakshi Goel Assignee: Harsha Havanur
Resolution: Unresolved Votes: 0
Labels: windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.1-1313-rel

Triage: Triaged
Operating System: Windows 64-bit
Is this a Regression?: Yes

 Description   
Jenkins Ref Link:
http://qa.hq.northscale.net/job/win_2008_x64--69_05_view_query_extended-P1/9/consoleFull

Test to Reproduce:
python testrunner.py -i inifile -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_start_stop_rebalance_out_incremental

Steps to repro:
1. Start load data
2. wait data for persistence
3. start rebalance out
4. stop rebalance
5. Start querying

****************** Error report *********************
Failure message is: Exception: DEBUG INFO: [{'msg': 'missing ids from memcached', 'details': ["view engine failed to index doc [cd30122b41dfca-168] in query: {'startkey': '[2008,7,null]', 'connection_timeout': 60000, 'stale': 'false'}"]}]

Logs:
[couchdb:error,2014-09-23T16:10:22.266,ns_1@10.3.2.133:<0.11152.50>:couch_log:error:44]Set view `default`, replica group `_design/test_view-72da6c5`, doc loader error
error: {dcp_client_died,<0.9950.50>,
                {{badmatch,{error,timeout}},
                 [{couch_dcp_client,store_snapshot_mutation,3,
                      [{file,
                           "c:/Jenkins/workspace/cs_301_win6408/couchbase/couchdb/src/couch_dcp/src/couch_dcp_client.erl"},
                       {line,1419}]},
                  {couch_dcp_client,handle_info,2,
                      [{file,
                           "c:/Jenkins/workspace/cs_301_win6408/couchbase/couchdb/src/couch_dcp/src/couch_dcp_client.erl"},
                       {line,545}]},
                  {gen_server,handle_msg,5,
                      [{file,"gen_server.erl"},{line,604}]},
                  {proc_lib,init_p_do_apply,3,
                      [{file,"proc_lib.erl"},{line,239}]}]}}
stacktrace: [{couch_dcp_client,get_stream_event_get_reply,3,
                 [{file,
                      "c:/Jenkins/workspace/cs_301_win6408/couchbase/couchdb/src/couch_dcp/src/couch_dcp_client.erl"},
                  {line,211}]},
             {couch_dcp_client,get_stream_event,2,
                 [{file,
                      "c:/Jenkins/workspace/cs_301_win6408/couchbase/couchdb/src/couch_dcp/src/couch_dcp_client.erl"},
                  {line,202}]},
             {couch_dcp_client,receive_events,4,
                 [{file,
                      "c:/Jenkins/workspace/cs_301_win6408/couchbase/couchdb/src/couch_dcp/src/couch_dcp_client.erl"},
                  {line,867}]},
             {couch_dcp_client,enum_docs_since,8,
                 [{file,
                      "c:/Jenkins/workspace/cs_301_win6408/couchbase/couchdb/src/couch_dcp/src/couch_dcp_client.erl"},
                  {line,257}]},
             {couch_set_view_updater,'-load_changes/8-fun-2-',12,
                 [{file,
                      "c:/Jenkins/workspace/cs_301_win6408/couchbase/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"},
                  {line,501}]},
             {lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
             {couch_set_view_updater,load_changes,8,
                 [{file,
                      "c:/Jenkins/workspace/cs_301_win6408/couchbase/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"},
                  {line,574}]},
             {couch_set_view_updater,'-update/8-fun-2-',14,
                 [{file,
                      "c:/Jenkins/workspace/cs_301_win6408/couchbase/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"},
                  {line,268}]}]

Uploading Logs

 Comments   
Comment by Meenakshi Goel [ 24/Sep/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12247/11dd43ca/10.3.2.133-9232014-175-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12247/1707cac5/10.3.2.134-9232014-1710-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12247/964762dc/10.3.2.135-9232014-1712-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12247/dfebd10e/10.3.2.139-9232014-1714-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12247/5ca067fa/10.3.2.141-9232014-1716-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12247/4da05243/10.3.2.143-9232014-1717-diag.zip




[MB-12246] querying bucket with non-json values without base64 operator leads to cbq engine crash Created: 24/Sep/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
bucket has 10 base64 items

query:
cbq> select * from default;
{
    "signature": {
        "*": "*"
    },
    "results": [

cbq>


cbq engine log

_time="2014-09-24T04:48:33-07:00" _level="INFO" _msg="New site created with url http://10.1.3.176:8091"
_time="2014-09-24T04:48:33-07:00" _level="INFO" _msg="cbq-engine started" version="0.7.0" datastore="http://10.1.3.176:8091"
_time="2014-09-24T04:48:40-07:00" _level="INFO" _msg="Created New Bucket default"
_time="2014-09-24T04:48:40-07:00" _level="INFO" _msg="Found index on keyspace #primary"
_time="2014-09-24T04:48:40-07:00" _level="INFO" _msg="Number of indexes 1"
_time="2014-09-24T04:48:40-07:00" _level="INFO" _msg="Number of entries fetched from the index 10"
panic: json: error calling MarshalJSON for type *value.annotatedValue: json: error calling MarshalJSON for type *value.ScopeValue: json: error calling MarshalJSON for type value.objectValue: json: error calling MarshalJSON for type *value.annotatedValue: json: error calling MarshalJSON for type *value.parsedValue: Cannot marshal binary value.

goroutine 286 [running]:
github.com/couchbaselabs/query/server/http.(*httpRequest).writeResult(0xc2001b30f0, 0xc20070a6e0, 0xc200759920, 0x2b1ea5e96f00)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_response.go:105 +0x186
github.com/couchbaselabs/query/server/http.(*httpRequest).writeResults(0xc2001b30f0, 0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_response.go:80 +0x139
github.com/couchbaselabs/query/server/http.(*httpRequest).Execute(0xc2001b30f0, 0xc200196b40, 0xc2001954d0, 0xc2006d9580, 0xc2006a3e40, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_response.go:43 +0xed
created by github.com/couchbaselabs/query/server.(*Server).serviceRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:125 +0x56e

goroutine 1 [IO wait]:
net.runtime_pollWait(0x2b1ea5217dc0, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2001ad980, 0xb, 0xc2001af990)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).accept(0xc2001ad900, 0xab0ec0, 0x0, 0xc2001af990, 0xb, ...)
/usr/local/go/src/pkg/net/fd_unix.go:385 +0x2c1
net.(*TCPListener).AcceptTCP(0xc2002131e0, 0x5bb2c6, 0x2b1ea3d04cf0, 0x5bb2c6)
/usr/local/go/src/pkg/net/tcpsock_posix.go:229 +0x45
net.(*TCPListener).Accept(0xc2002131e0, 0xc2001961e0, 0xc2001e8020, 0xc2001ada20, 0x0, ...)
/usr/local/go/src/pkg/net/tcpsock_posix.go:239 +0x25
net/http.(*Server).Serve(0xc200196bb0, 0xc2001e75c0, 0xc2002131e0, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/server.go:1542 +0x85
net/http.(*Server).ListenAndServe(0xc200196bb0, 0xc2001d3b40, 0xc200196ba0)
/usr/local/go/src/pkg/net/http/server.go:1532 +0x9e
github.com/couchbaselabs/query/server/http.(*HttpEndpoint).ListenAndServe(0xc200196ba0, 0x1, 0x9c0160)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_endpoint.go:36 +0x2a
main.main()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/main/main.go:87 +0x5c2

goroutine 2 [syscall]:

goroutine 4 [IO wait]:
net.runtime_pollWait(0x2b1ea5217f00, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2001ad080, 0xb, 0xc2001af990)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).accept(0xc2001ad000, 0xab0ee0, 0x0, 0xc2001af990, 0xb, ...)
/usr/local/go/src/pkg/net/fd_unix.go:385 +0x2c1
net.(*UnixListener).AcceptUnix(0xc200184700, 0x0, 0xc200193898, 0x74ff1a)
/usr/local/go/src/pkg/net/unixsock_posix.go:286 +0x49
net.(*UnixListener).Accept(0xc200184700, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/unixsock_posix.go:297 +0x49
github.com/couchbaselabs/retriever/logger.doHandleConnections(0xc2001ab780, 0x9ed910, 0xa)
/root/tuq/gocode/src/github.com/couchbaselabs/retriever/logger/accept.go:50 +0x32f
github.com/couchbaselabs/retriever/logger.handleConnections(0xc2001ab780, 0x9ed910, 0xa)
/root/tuq/gocode/src/github.com/couchbaselabs/retriever/logger/accept.go:24 +0x39
created by github.com/couchbaselabs/retriever/logger.NewLogger
/root/tuq/gocode/src/github.com/couchbaselabs/retriever/logger/logger.go:136 +0x26f

goroutine 14 [chan receive]:
github.com/couchbaselabs/query/datastore/couchbase.keepPoolFresh(0xc2001959a0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/datastore/couchbase/couchbase.go:214 +0x5f
created by github.com/couchbaselabs/query/datastore/couchbase.loadNamespace
/root/tuq/gocode/src/github.com/couchbaselabs/query/datastore/couchbase/couchbase.go:131 +0x5de

goroutine 273 [runnable]:
github.com/couchbaselabs/query/server/http.(*HttpEndpoint).ServeHTTP(0xc200196ba0, 0xc2001b8080, 0xc2001dfa80, 0xc200186ea0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_endpoint.go:46 +0xf3
net/http.serverHandler.ServeHTTP(0xc200196bb0, 0xc2001b8080, 0xc2001dfa80, 0xc200186ea0)
/usr/local/go/src/pkg/net/http/server.go:1517 +0x16c
net/http.(*conn).serve(0xc2001ada20)
/usr/local/go/src/pkg/net/http/server.go:1096 +0x765
created by net/http.(*Server).Serve
/usr/local/go/src/pkg/net/http/server.go:1564 +0x266

goroutine 274 [IO wait]:
net.runtime_pollWait(0x2b1ea5217e60, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2001ada10, 0xb, 0xc2001af990)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc2001ad990, 0xc200665000, 0x8000, 0x8000, 0x0, ...)
/usr/local/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc2001e8020, 0xc200665000, 0x8000, 0x8000, 0x8000, ...)
/usr/local/go/src/pkg/net/net.go:123 +0xc3
io.Copy(0xc2001a1f90, 0xc2001e80b8, 0xc2001afb10, 0xc2001e8020, 0x0, ...)
/usr/local/go/src/pkg/io/io.go:348 +0x1c6
net/http.func·004()
/usr/local/go/src/pkg/net/http/server.go:162 +0x66
created by net/http.(*conn).closeNotify
/usr/local/go/src/pkg/net/http/server.go:168 +0x1c6

goroutine 17 [runnable]:
github.com/couchbaselabs/query/execution.func·023()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:75 +0x687
sync.(*Once).Do(0xc200162c30, 0xc2006e0320)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200162be0, 0xc2006ec200, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:84 +0xf8
github.com/couchbaselabs/query/server.(*Server).serviceRequest(0xc200196b40, 0xc200162dc0, 0xc2001b30f0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:129 +0x6e4
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:89 +0x9a
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 18 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 19 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 20 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 21 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 22 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 23 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 24 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 25 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 26 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 27 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 28 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 29 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 30 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 31 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 32 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 33 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 34 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 35 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 36 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 37 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 38 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 39 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 40 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 41 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 42 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 43 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 44 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 45 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 46 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 47 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 48 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 49 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 50 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 51 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 52 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 53 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 54 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 55 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 56 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 57 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 58 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 59 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 60 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 61 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 62 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 63 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 64 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 65 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 66 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 67 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 68 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 69 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 70 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 71 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 72 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 73 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 74 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 75 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 76 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 77 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 78 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 79 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 80 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 81 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 82 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 83 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 84 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 85 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 86 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 87 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 88 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 89 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 90 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 91 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 92 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 93 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 94 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 95 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 96 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 97 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 98 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 99 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 100 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 101 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 102 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 103 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 104 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 105 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 106 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 107 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 108 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 109 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 110 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 111 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 112 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 113 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 114 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 115 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 116 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 117 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 118 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 119 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 120 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 121 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 122 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 123 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 124 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 125 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 126 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 127 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 128 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 129 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 130 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 131 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 132 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 133 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 134 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 135 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 136 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 137 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 138 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 139 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 140 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 141 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 142 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 143 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 144 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 145 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 146 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 147 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 148 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 149 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 150 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 151 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 152 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 153 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 154 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 155 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 156 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 157 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 158 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 159 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 160 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 161 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 162 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 163 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 164 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 165 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 166 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 167 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 168 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 169 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 170 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 171 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 172 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 173 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 174 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 175 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 176 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 177 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 178 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 179 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 180 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 181 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 182 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 183 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 184 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 185 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 186 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 187 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 188 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 189 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 190 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 191 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 192 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 193 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 194 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 195 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 196 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 197 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 198 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 199 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 200 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 201 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 202 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 203 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 204 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 205 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 206 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 207 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 208 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 209 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 210 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 211 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 212 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 213 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 214 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 215 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 216 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 217 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 218 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 219 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 220 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 221 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 222 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 223 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 224 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 225 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 226 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 227 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 228 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 229 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 230 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 231 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 232 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 233 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 234 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 235 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 236 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 237 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 238 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 239 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 240 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 241 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 242 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 243 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 244 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 245 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 246 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 247 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 248 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 249 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 250 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 251 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 252 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 253 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 254 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 255 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 256 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 257 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 258 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 259 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 260 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 261 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 262 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 263 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 264 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 265 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 266 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 267 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 268 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 269 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 270 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 271 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 272 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc200196b40)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:88 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:82 +0x58

goroutine 275 [chan receive]:
github.com/couchbaselabs/query/server/http.func·001()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:93 +0x43
created by github.com/couchbaselabs/query/server/http.newHttpRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:95 +0x6e8

goroutine 288 [select]:
github.com/couchbaselabs/query/execution.func·023()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:75 +0x687
sync.(*Once).Do(0xc200162af0, 0xc2006e0420)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200162aa0, 0xc2006ec200, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:84 +0xf8
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:129 +0x1a8

goroutine 289 [runnable]:
github.com/couchbaselabs/query/execution.func·011()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:71 +0x524
sync.(*Once).Do(0xc2001ad7a0, 0xc2006e0440)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Parallel).RunOnce(0xc2001ad750, 0xc2006ec200, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:80 +0xf8
created by github.com/couchbaselabs/query/execution.func·023
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x53b

goroutine 287 [select]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:141 +0x3a6
sync.(*Once).Do(0xc2006ec1d0, 0xc2006e1510)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc2006ec180, 0xc2006d9680, 0xc2006ec180, 0xc2006ec200, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:153 +0x131
github.com/couchbaselabs/query/execution.(*Stream).RunOnce(0xc2006ec180, 0xc2006ec200, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/stream.go:38 +0x85
created by github.com/couchbaselabs/query/execution.func·023
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x53b

goroutine 301 [runnable]:
sync.runtime_Semacquire(0xc2006ec0d4)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc2006ec0d0)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc2006ec0d0, 0xc2006e1750)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc2006ec080, 0xc2006d9740, 0xc2006ec080, 0xc2006ec200, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:153 +0x131
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc2006ec080, 0xc2006ec200, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x85
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:129 +0x1a8

goroutine 307 [IO wait]:
net.runtime_pollWait(0x2b1ea5217d20, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc20074d080, 0xb, 0xc2001af990)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc20074d000, 0xc20074f000, 0x1000, 0x1000, 0x0, ...)
/usr/local/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc2006d3a00, 0xc20074f000, 0x1000, 0x1000, 0x0, ...)
/usr/local/go/src/pkg/net/net.go:123 +0xc3
bufio.(*Reader).fill(0xc20070c8a0)
/usr/local/go/src/pkg/bufio/bufio.go:79 +0x10c
bufio.(*Reader).Peek(0xc20070c8a0, 0x1, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/pkg/bufio/bufio.go:107 +0xc9
net/http.(*persistConn).readLoop(0xc2006ec700)
/usr/local/go/src/pkg/net/http/transport.go:670 +0xc4
created by net/http.(*Transport).dialConn
/usr/local/go/src/pkg/net/http/transport.go:511 +0x574

goroutine 308 [select]:
net/http.(*persistConn).writeLoop(0xc2006ec700)
/usr/local/go/src/pkg/net/http/transport.go:774 +0x26f
created by net/http.(*Transport).dialConn
/usr/local/go/src/pkg/net/http/transport.go:512 +0x58b




[MB-12245] incorrect error msg for querying existing bucket Created: 24/Sep/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
I have default bucket with items(base64 values). didn't create primary index yet

Queries shows error that bucket doesn't exists:
cbq> select * from default;
{
    "signature": {
        "*": "*"
    },
    "results": [
    ],
    "errors": [
        {
            "caller": "view_index:204",
            "code": 5000,
            "key": "Internal Error",
            "message": "bucket default not found"
        }
    ],
    "state": "errors",
    "metrics": {
        "elapsedTime": "472.894ms",
        "executionTime": "472.449ms",
        "resultCount": 0,
        "errorCount": 1
    }
}





[MB-12244] explain query always shows empty result Created: 24/Sep/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
cbq> explain select * from default;
{
    "signature": "json",
    "results": [
    ],
    "state": "success",
    "metrics": {
        "elapsedTime": "1.248ms",
        "executionTime": "1.173ms",
        "resultCount": 0
    }
}




[MB-12243] Files should have their own config instances Created: 24/Sep/14  Updated: 24/Sep/14  Resolved: 24/Sep/14

Status: Resolved
Project: Couchbase Server
Component/s: forestdb
Affects Version/s: bug-backlog
Fix Version/s: bug-backlog
Security Level: Public

Type: Bug Priority: Minor
Reporter: Jung-Sang Ahn Assignee: Jung-Sang Ahn
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
In 'struct filemgr' structure, '*config' variable points to global variable 'global_config', which is initialized by filemgr_init().

If user tries to open a file with different option, there is no problem at that time, but previous options in global_config is restored after compaction is performed. This is because fdb_link_new_file(...) function calls filemgr_open(...) passing 'handle->file->config' as an option, which is pointing to 'global_config' variable, so that current file option is ignored and previous setting is restored.

For example,

1. File 'A' is opened with synchronous write option. -> global_config is set to synchronous write.
2. File 'A' is closed.
3. File 'A' is re-opened with asynchronous write option. -> no problem.
4. Compaction is performed on the file 'A'.
5. fdb_link_new_file(...) is invoked. -> calls filemgr_open(...) with handle->file->config option, which points to global_config. -> the new file is opened with synchronous write option.

To avoid this problem, all files should have their own config instances.


 Comments   
Comment by Jung-Sang Ahn [ 24/Sep/14 ]
http://review.couchbase.org/#/c/41600/
Comment by Chiyoung Seo [ 24/Sep/14 ]
http://review.couchbase.org/#/c/41600/




Generated at Wed Oct 01 03:13:36 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.