[MB-12098] [Query-XDCR] One extra returned from query with stale=false Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Server
Component/s: cross-datacenter-replication, view-engine
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sangharsh Agarwal Assignee: Sangharsh Agarwal
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: CentOS64

Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: [Source]
10.5.2.228 : https://s3.amazonaws.com/bugdb/jira/MB-12098/a8918449/10.5.2.228-8282014-836-diag.zip
10.5.2.228 : https://s3.amazonaws.com/bugdb/jira/MB-12098/c121150a/10.5.2.228-diag.txt.gz
10.5.2.229 : https://s3.amazonaws.com/bugdb/jira/MB-12098/2634a327/10.5.2.229-diag.txt.gz
10.5.2.229 : https://s3.amazonaws.com/bugdb/jira/MB-12098/ed39638b/10.5.2.229-8282014-840-diag.zip
10.5.2.230 : https://s3.amazonaws.com/bugdb/jira/MB-12098/b7045d19/10.5.2.230-diag.txt.gz
10.6.2.192 : https://s3.amazonaws.com/bugdb/jira/MB-12098/4ebadc41/10.6.2.192-8282014-844-diag.zip
10.6.2.192 : https://s3.amazonaws.com/bugdb/jira/MB-12098/eb774511/10.6.2.192-diag.txt.gz

[Destination]
10.5.2.231 : https://s3.amazonaws.com/bugdb/jira/MB-12098/7011abe7/10.5.2.231-8282014-847-diag.zip
10.5.2.231 : https://s3.amazonaws.com/bugdb/jira/MB-12098/cb0e8440/10.5.2.231-diag.txt.gz
10.5.2.232 : https://s3.amazonaws.com/bugdb/jira/MB-12098/6be49b53/10.5.2.232-8282014-852-diag.zip
10.5.2.232 : https://s3.amazonaws.com/bugdb/jira/MB-12098/ba1df511/10.5.2.232-diag.txt.gz
10.5.2.233 : https://s3.amazonaws.com/bugdb/jira/MB-12098/d60532b2/10.5.2.233-diag.txt.gz
10.5.2.233 : https://s3.amazonaws.com/bugdb/jira/MB-12098/e021c3d9/10.5.2.233-8282014-858-diag.zip
Is this a Regression?: Unknown

 Description   
http://qa.hq.northscale.net/job/centos_x64--107_01--rebalanceXDCR-P1/57/consoleFull

[Test Error]

======================================================================
ERROR: swap_rebalance_replication_with_view_queries_and_ops (xdcr.rebalanceXDCR.Rebalance)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/xdcr/rebalanceXDCR.py", line 283, in swap_rebalance_replication_with_view_queries_and_ops
    [task.result(self._poll_timeout) for task in tasks]
  File "lib/tasks/future.py", line 160, in result
    return self.__get_result()
  File "lib/tasks/future.py", line 112, in __get_result
    raise self._exception
QueryViewException: Error occured querying view default0: expected number of rows: '70000' is greater than expected 70001

----------------------------------------------------------------------
Ran 1 test in 2215.179s


[Test Steps]
Source: 10.5.2.228, 10.5.2.229, 10.5.2.230
Destination: 10.5.2.231, 10.5.2.232, 10.5.2.233

1. Setup 3-3 node Source and Destination Cluster.
2. Bucket: default, mode: CAPI mode unidirectional xdcr.
3. Load 1M items on Source.
4. Create DesignDoc: ddoc1 on Source and Destination
5. Created views: default0, default1, default2, default3, default4 on Source and Destination.
6. RUN Parallel
     i. 30K and Delete 30K items on Source
    ii. SWAP Rebalance one node on Source (Swaped 10.5.2.230 with 10.6.2.192 on Source)
   iii. Running view-queries on all views on source and destination until rebalance is finished and mutations are finished with stale=false e.g.http://10.5.2.231:8092/default/_design/ddoc1/_view/default1?full_set=true&stale=false)

7. Wait for re-balance to finished.
8. Again run queries on all views with stale=false to verify expected rows=70000.

Error Expected rows: 70000 and Actual: 70001


 Comments   
Comment by Sangharsh Agarwal [ 29/Aug/14 ]
Bug is not always and easily reproducible, I tried 5 times but couldn't reproduced.
Comment by Sriram Melkote [ 29/Aug/14 ]
Sangharsh, why do you suspect this to be view engine issue?

For example, if you have you verified the database has 70,000 docs but views returned 70,001 results, then it'd be a view engine issue.

I'd appreciate if you can point me to the data in the logs that may suggest this.




[MB-12099] Erlang crashes if xdcr_trace attempts to log invalid utf8 Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Aleksey Kondratenko Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Yes

 Description   
See MB-12097 where attempt to capi-replicate invalid utf8 key causes find_missing to crash which causes some crash within ale.


 Comments   
Comment by Aleksey Kondratenko [ 29/Aug/14 ]
It looks like change of unicode:characters_to_binary to simple iolist_to_binary helps.




[MB-12063] KV+XDCR System test : Between expiration and purging, getMeta() retrieves revID as 1 for deleted key from Source, same deleted key from Destination returns 2. Created: 25/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0.1, 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Aruna Piravi Assignee: Sundar Sridharan
Resolution: Unresolved Votes: 0
Labels: rc2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: CentOS 6.x , build 3.0.0-1174-rel

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
--> Before expiration and after purging, all metadata match between source and destination clusters.
--> However, after expiration, there's some code that causes a deleted key at the source(!!!) to have seqno as 1. The seqno at destination is however 2, as expected.
--> The data below for uni-xdcr (C1 -> C2).
--> Not a recent regression, seen it once before in 3.0.0-9xx. Catching this bug totally depends on when I run the validation script after system test is completed. Expiration is usually set to 1 day and tombstone purge interval is 3 days on both source and destination. Once tombstones are purged, I don't see this mismatch. So I don't have a live cluster.

{'C1_location:': u'172.23.105.44', 'vb': 90, 'C2_node': u'172.23.105.54', 'C1_key_count': 19919, 'C2_key_count': 19919, 'missing_keys': 0}
RevID or CAS mismatch -
  172.23.105.44(C1): key:65ABEE18-153_100061 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902841111553483, 'flags': 0, 'expiration': 1408646731}
  172.23.105.54(C2): key:65ABEE18-153_100061 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902841111553484, 'flags': 0, 'expiration': 1408646731}
 RevID or CAS mismatch -
  172.23.105.44(C1): key:65ABEE18-153_100683 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902841111336520, 'flags': 0, 'expiration': 1408646731}
  172.23.105.54(C2): key:65ABEE18-153_100683 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902841111336521, 'flags': 0, 'expiration': 1408646731}
RevID or CAS mismatch -
  172.23.105.44(C1): key:65ABEE18-153_100713 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902841111837669, 'flags': 0, 'expiration': 1408646731}
  172.23.105.54(C2): key:65ABEE18-153_100713 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902841111837670, 'flags': 0, 'expiration': 1408646731}
 RevID or CAS mismatch -
  172.23.105.44(C1): key:65ABEE18-153_103240 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902843752129235, 'flags': 0, 'expiration': 1408646733}
  172.23.105.54(C2): key:65ABEE18-153_103240 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902843752129236, 'flags': 0, 'expiration': 1408646733}
 RevID or CAS mismatch -
  172.23.105.44(C1): key:65ABEE18-153_105170 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902847773405994, 'flags': 0, 'expiration': 1408646737}
  172.23.105.54(C2): key:65ABEE18-153_105170 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902847773405995, 'flags': 0, 'expiration': 1408646737}

Please let me know what/if you need in particular to diagnose this issue. Thanks!




 Comments   
Comment by Aruna Piravi [ 25/Aug/14 ]
Worth mentioning that if the same key gets recreated at Source with seq <= seqno(same key at destination), the create will/may not get propagated.
Comment by Aruna Piravi [ 25/Aug/14 ]
We have many xdcr functional tests with expiration, after which we compare revIDs, even of deleted items. We did not hit this particular bug. Venu did some unit tests and could not catch it either.

I'd like to perform the same system test on 2.5.1 to determine if this is a regression. Again system test runs itself runs for 12-15 hrs, we keep loading throughout the test, items keep expiring until the next 24 hrs, expiry pager runs 3 days from the start of the test and the validation script runs for half a day so it's all about timing.

Will get you the data files and start the system test.

Comment by Aruna Piravi [ 25/Aug/14 ]
http://172.23.105.44:8091/index.html
http://172.23.105.54:8091/index.html
Comment by Aruna Piravi [ 25/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12063/stdbucket.rtf --> keys that have this mismatch for bucket 'standardbucket'.

For a quick look -
https://s3.amazonaws.com/bugdb/jira/MB-12063/44_couch.tar (source)
https://s3.amazonaws.com/bugdb/jira/MB-12063/54_couch.tar (dest)
Comment by Aruna Piravi [ 25/Aug/14 ]
Also seeing some keys where revID is greater at source than dest. Pls look at vbuckets 15 and 28.

RevID or CAS mismatch -
  172.23.105.44(C1): key:6B67A321-142_4666321 metadata:{'deleted': 1, 'seqno': 4, 'cas': 13175771830561215, 'flags': 0, 'expiration': 1408649286}
  172.23.105.54(C2): key:6B67A321-142_4666321 metadata:{'deleted': 1, 'seqno': 3, 'cas': 13175771830561214, 'flags': 0, 'expiration': 1408649286}
RevID or CAS mismatch -
  172.23.105.44(C1): key:6B67A321-142_4666453 metadata:{'deleted': 1, 'seqno': 4, 'cas': 13175771778347790, 'flags': 0, 'expiration': 1408649286}
  172.23.105.54(C2): key:6B67A321-142_4666453 metadata:{'deleted': 1, 'seqno': 3, 'cas': 13175771778347789, 'flags': 0, 'expiration': 1408649286}
 RevID or CAS mismatch -
  172.23.105.44(C1): key:6B67A321-142_4674099 metadata:{'deleted': 1, 'seqno': 4, 'cas': 13175775983769867, 'flags': 0, 'expiration': 1408649290}
  172.23.105.54(C2): key:6B67A321-142_4674099 metadata:{'deleted': 1, 'seqno': 3, 'cas': 13175775983769866, 'flags': 0, 'expiration': 1408649290}
 RevID or CAS mismatch -
  172.23.105.44(C1): key:6B67A321-142_4674109 metadata:{'deleted': 1, 'seqno': 4, 'cas': 13175775977754745, 'flags': 0, 'expiration': 1408649290}
  172.23.105.54(C2): key:6B67A321-142_4674109 metadata:{'deleted': 1, 'seqno': 3, 'cas': 13175775977754744, 'flags': 0, 'expiration': 1408649290}
RevID or CAS mismatch -
  172.23.105.44(C1): key:6B67A321-142_4677328 metadata:{'deleted': 1, 'seqno': 4, 'cas': 13175778211679600, 'flags': 0, 'expiration': 1408649293}
  172.23.105.54(C2): key:6B67A321-142_4677328 metadata:{'deleted': 1, 'seqno': 3, 'cas': 13175778211679599, 'flags': 0, 'expiration': 1408649293}
Comment by Aruna Piravi [ 25/Aug/14 ]
cbcollect :

https://s3.amazonaws.com/bugdb/jira/MB-12063/source.tar
https://s3.amazonaws.com/bugdb/jira/MB-12063/dest.tar

Note: .44 is down now. Was up until this morning. .44 ran out of diskspace. The above script ran 2 days back.

I can start system test on 2.5.1 tomorrow morning if required. Pls let me know, thanks.
Comment by Wayne Siu [ 26/Aug/14 ]
Reviewed with Cihan. Potentially a RC2 candidate.
If the fix is contained and is ready by this week.
Need Dev's risk assessment.
Comment by Mike Wiederhold [ 26/Aug/14 ]
The logs have rolled over so I can't see exactly what happened. I think the reason that the cluster got into this state is because the expiry pager was run on one side of the cluster and not the other. This can easily explain the why destination would have a different sequence number and the source wouldn't. This happens because expiring a key will increment the rev sequence number and unfortunately I don't think there is anything we can do about this.

I am assuming that the same issue is happening when the source clusters seqno is incremented, but the destination seqno is not. In this case we should see the delete replicated to the other side and I cannot determine whether or not this happened because the logs are rolled over. I'm also not sure if you just checked those keys while keys were being propagated to the destination node. All of the traffic could have stopped, but the expiry pager might have kicked in during the verification phase.

I'm seeing a log message that I need to investigate further so I will leave this assigned to me.
Comment by Aruna Piravi [ 26/Aug/14 ]
 > I think the reason that the cluster got into this state is because the expiry pager was run on one side of the cluster and not the other.

That's possible but what can explain seeing 'deleted': 1 with 'seqno': 1? My understanding is - any key with deleted flag = 1 needs to have revid as atleast 2. RevID for any doc can be 1 only at the time of creation. If expired or deleted, revID should be incremented, right?
Comment by Aruna Piravi [ 27/Aug/14 ]
Also, this is C1 -> C2 (uni-xdcr). So seeing something like

172.23.105.44(C1): key:65ABEE18-153_100061 metadata:{'deleted': 1, 'seqno': 1, 'cas': 1902841111553483, 'flags': 0, 'expiration': 1408646731}
172.23.105.54(C2): key:65ABEE18-153_100061 metadata:{'deleted': 1, 'seqno': 2, 'cas': 1902841111553484, 'flags': 0, 'expiration': 1408646731}

would mean - expiry pager ran at C1('deleted':1) but did not increment revid which is clearly a bug.
Comment by Cihan Biyikoglu [ 27/Aug/14 ]
is there an ETA on the resolution? if this won't resolve in the next day, we need to push this one out.
thanks
Comment by Abhinav Dangeti [ 27/Aug/14 ]
Mike's already submitted the fixes:
http://review.couchbase.org/#/c/40996/
http://review.couchbase.org/#/c/40997/
Comment by Mike Wiederhold [ 28/Aug/14 ]
The above changes do not resolve this issue. When looking through the logs I saw two separate issues which needed to be fixed.
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
need a fix by EOD today so we can consider this for RC2 . otherwise we'll need to delay the RC
Comment by Sundar Sridharan [ 28/Aug/14 ]
Aruna, if possible, could you please list the operations that the test is doing since we need clues to reproduce and verify. thanks
Comment by Aruna Piravi [ 28/Aug/14 ]
Hi Sundar, as we discussed, here are the steps

Clusters
-----------
C1 : http://172.23.105.44:8091/
C2 : http://172.23.105.54:8091/

Steps
--------
1. Setup uni-xdcr on "standardbucket1" , load till active_resident_ratio = ~70 on standardbucket1
2. Access phase with 50% gets, 50%deletes for 3 hrs
3. Rebalance-out 1 node at cluster1
4. Rebalance-in 1 node at cluster1
5. Failover and remove node at cluster1
6. Failover and add-back node at cluster1
7. Rebalance-out 1 node at cluster2
8. Rebalance-in 1 node at cluster2
9. Failover and remove node at cluster2
10. Failover and add-back node at cluster2
11. Soft restart all nodes in cluster1 one by one

Run verification script after 15 hrs(after all items are expired).
Comment by Sundar Sridharan [ 28/Aug/14 ]
Aruna the toy build is couchbase-server-community_cent58-3.0.0-toy-sundar-x86_64_3.0.0-710-toy.rpm
Comment by Aruna Piravi [ 28/Aug/14 ]
System test started on the above toy build.

You can check
C1 : http://172.23.105.44:8091/
C2 : http://172.23.105.54:8091/
if you would like to watch the test.

It should be finished by tomorrow afternoon. If there are crashes, it would stop sooner. I will check back late night and tomorrow morning.
Comment by Aruna Piravi [ 28/Aug/14 ]
Sundar, there seems to a problem with the toy build. The buckets got created but did not get successfully loaded on all nodes in the cluster. Pls take a look at any of the clusters. All nodes in both clusters are in pending state.
Comment by Raju Suravarjjala [ 29/Aug/14 ]
Thanks Sundar, we need to make a call on this quickly
Comment by Sundar Sridharan [ 29/Aug/14 ]
Raju, I had a discussion with Aruna and Chiyoung and will try to get around to this issue as soon as possible.
Aruna, the new toy build is ready at..
couchbase-server-community_cent58-3.0.0-toy-sundar-x86_64_3.0.0-711-toy.rpm
thanks
Comment by Chiyoung Seo [ 29/Aug/14 ]
Aruna,

I looked at getMeta implementation and confirmed that we return "deleted" flag to the caller if an item is expired. Therefore, this mismatch issue can happen if the expiry pager was executed only in source or destination cluster, but not both. As we discussed, please run the expiry pager by force in both clusters before running the verification step.
Comment by Aruna Piravi [ 29/Aug/14 ]
Started a small scale system test with latest which I expect to complete in 2 hrs. Set expiration is 10 mins. Will run expiry pager on both sides before verification.

If we do get "deleted" flag for "expired but undeleted" items, it may not be a bug. Will confirm.




[MB-10291] cbmcd connections cannot be efficiently used since operations are never interleaved Created: 24/Feb/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.0, 3.0
Fix Version/s: bug-backlog
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Dave Finlay
Resolution: Unresolved Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged

 Description   
This long-known architectural issue is that currently every operation is sent back in the order in which it had been received. Thus for a set of requests A, B, C, D, any background fetch latency on item B will also affect service times with C and D.

Existing memcached protocol does not state that operations must be handled in order, but owing to the one implementation doing so and all clients having been built/tested against that implementation, it's possible that there are erroneous dependencies on current behavior.

 Comments   
Comment by James Gurney [ 26/Feb/14 ]
The nature of our application (Realtime bidder) demands a response to incoming requests of typically around 100ms. Because we can't control external network latency, we prefer to target <50ms per request and we actually tend to average about 2ms per request, so the time we can spend waiting on Couchbase is limited. As such, we do an async get with a 25ms timeout. In the event that the Future doesn't return in time, we log it and move on. Under non bg_fetch operation situations, this almost never happens.

When we ran into this problem, what we observed was that, although our Couchbase disk fetch rate was still low (0.5%), it had a disproportionate impact on the rate of requests which took more than 25ms to return. I no longer have access to the stats, but as I recall, we saw our lookup "failure" (as classified as taking >25ms) rate go up from almost zero to about 30%.
Comment by cihangirb [ 06/Mar/14 ]
Chatted with Steve and a few other folks. this is a large change and we cannot take this for 3.0 for sure.
Comment by Steve Yen [ 14/Mar/14 ]
Matt had a separate email on this laying out options, and one was to consider a new protocol. I think a new protocol would need to have explicit support for responses that come out-of-order w.r.t. the original sequence of requests. That way a slow operation doesn't block up fast operations.

Also, I'd propose that a "FENCE" or barrier option would be useful, so that a client can know that all its in-flight requests up to a certain point all have responses completed and on-the-wire.

Started prototyping a few ideas here, but definitely post-3.0.
Comment by Matt Ingenthron [ 17/Mar/14 ]
I seem to remember you, Steve, proposing that there may be a protocol change here at some point in time. There are a few other things that should go into that bucket too, like consolidation of metadata into something extensible (common problem) and timeout advisories (Alk's suggestion).
Comment by Matt Ingenthron [ 28/Aug/14 ]
Dave: assigning this to you as it should be considered for post 3.0. Also, you're a good assignee because you haven't enough bugs. :)
Comment by Dave Finlay [ 28/Aug/14 ]
Appreciate the thought, Matt. You're very considerate. :-)
Comment by Don Pinto [ 29/Aug/14 ]
Matt, does this issue happen for 100% write workloads as well? (Because for use-cases like Data aggregation, which are write-heavy, “High data ingestion rate” was mentioned as a requirement). Is this not the case?




[MB-12005] vbucket-seqno stats getting timed out during Views DGM test Created: 19/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket, view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Meenakshi Goel Assignee: Sriram Melkote
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1166-rel

Triage: Triaged
Operating System: Ubuntu 64-bit
Is this a Regression?: Yes

 Description   
Test to Reproduce:
./testrunner -i yourfile.ini -t view.createdeleteview.CreateDeleteViewTests.pending_removal_with_ddoc_ops,ddoc_ops=update,test_with_view=True,num_ddocs=3,num_views_per_ddoc=3,items=200000,nodes_out=1,active
_resident_threshold=10,dgm_run=True,eviction_policy=fullEviction,skip_cleanup=true

Steps to Reproduce:
1. Setup a 5-node cluster
2. Rebalance in all nodes
3. Load bucket to achieve dgm 10%
4. Failover 1 node
5. Create Views and perform ddoc update operations
6. Test exits with error during ddoc validation

014-08-18 04:00:37 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://10.3.5.90:8092/default/_design/ddoc_test1/_view/views0?stale=false&connection_timeout=60000&full_set=true
2014-08-18 04:15:37 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.3.5.90:8092/default/_design/ddoc_test1/_view/views0?stale=false&connection_timeout=60000&full_set=true error timed out
2014-08-18 04:15:37 | ERROR | MainProcess | Cluster_Thread | [task.execute] Unexpected Exception Caught
ERROR
[('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 56, 'step', 'self.execute(task_manager)'), ('lib/tasks/task.py', 1525, 'execute', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
Mon Aug 18 04:15:37 2014
[('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/case.py', 391, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/case.py', 327, 'run', 'testMethod()'), ('pytests/view/createdeleteview.py', 626, 'pending_removal_with_ddoc_ops', 'self._verify_ddoc_data_all_buckets()'), ('pytests/view/createdeleteview.py', 274, '_verify_ddoc_data_all_buckets', 'result = self.cluster.query_view(self.master, ddoc_name, view.name, query, num_items, bucket)'), ('lib/couchbase/cluster.py', 464, 'query_view', 'return _task.result(timeout)'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')]
2014-08-18 04:15:37 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED

======================================================================
ERROR: pending_removal_with_ddoc_ops (view.createdeleteview.CreateDeleteViewTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/view/createdeleteview.py", line 626, in pending_removal_with_ddoc_ops
    self._verify_ddoc_data_all_buckets()
  File "pytests/view/createdeleteview.py", line 274, in _verify_ddoc_data_all_buckets
    result = self.cluster.query_view(self.master, ddoc_name, view.name, query, num_items, bucket)
  File "lib/couchbase/cluster.py", line 464, in query_view
    return _task.result(timeout)
  File "lib/tasks/future.py", line 160, in result
    return self.__get_result()
  File "lib/tasks/future.py", line 112, in __get_result
    raise self._exception
ServerUnavailableException: unable to reach the host @ 10.3.5.90

Logs:
[couchdb:error,2014-08-18T4:39:41.963,ns_1@10.3.5.90:<0.12717.6>:couch_log:error:44]Set view `default`, replica group `_design/ddoc_test0`, doc loader error
error: {timeout,{gen_server,call,
                                 [<0.12658.6>,
                                  {add_stream,706,0,0,5637,6},
                                  60000]}}
stacktrace: [{gen_server,call,3,[{file,"gen_server.erl"},{line,188}]},
             {couch_dcp_client,enum_docs_since,8,
                 [{file,
                      "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_dcp/src/couch_dcp_client.erl"},
                  {line,246}]},
             {couch_set_view_updater,'-load_changes/8-fun-2-',12,
                 [{file,
                      "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"},
                  {line,516}]},
             {lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
             {couch_set_view_updater,load_changes,8,
                 [{file,
                      "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"},
                  {line,589}]},
             {couch_set_view_updater,'-update/8-fun-3-',14,
                 [{file,
                      "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"},
                  {line,281}]}]

[couchdb:error,2014-08-18T6:46:34.997,ns_1@10.3.5.90:<0.12648.6>:couch_log:error:44]dcp client (<0.12660.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...
[couchdb:error,2014-08-18T6:46:39.608,ns_1@10.3.5.90:<0.21856.6>:couch_log:error:44]dcp client (<0.21861.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...
[couchdb:error,2014-08-18T6:46:42.611,ns_1@10.3.5.90:<0.21856.6>:couch_log:error:44]dcp client (<0.21861.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...

*Observed stacktraces and crashes in logs. Uploading logs.

Live Cluster:
1:10.3.5.90
2:10.3.5.91
3:10.3.5.92
4:10.3.5.93
5:10.3.4.75

 Comments   
Comment by Meenakshi Goel [ 19/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12005/f806d72b/10.3.5.90-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12005/8f3bae64/10.3.5.91-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12005/4fc16d8a/10.3.5.92-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12005/16116a7e/10.3.4.75-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12005/a3e503f6/10.3.5.93-diag.zip
Comment by Sarath Lakshman [ 19/Aug/14 ]
I believe we tried a toy build with separate connection for stats before in MB-11706. So I think I don't have much information about this problem
Comment by Nimish Gupta [ 22/Aug/14 ]
I have a toy build with change for separate connection (http://latestbuilds.hq.couchbase.com/couchbase-server-community_ubunt12-3.0.0-toy-nimish-x86_64_3.0.0-704-toy.deb). Meenakshi, please run the test with this build.
Comment by Meenakshi Goel [ 22/Aug/14 ]
Started test with the toy build.
Comment by Nimish Gupta [ 25/Aug/14 ]
I don't see any crash in couchdb after using the separate connection. The crashes are in ns_server logs, looks due to rebalance and failover which were in earlier logs also. The separate connection has reduced the number of stats timeout messages. Previously there were 714 stats timeout message which now has reduced to 113.
Comment by Sriram Melkote [ 25/Aug/14 ]
Wayne, please help us obtain a toy build with mctimings compiled.
Comment by Cihan Biyikoglu [ 26/Aug/14 ]
Hi team, could you explain why this is critical? trying to shut the gates for 3.0 and want to make sure we keep only things that have high impact on the release.
Comment by Ketaki Gangal [ 27/Aug/14 ]
This causes a number of view-test timeouts. While these can be worked around by adjusting test timeouts, imo it is not the right way for the code/query timings to work.
As mentioned earlier, there are a large number of stat calls being made here.

This should manifest in performance tests already, but with functional tests, there is a large increase in query-runtime due to the above.


Comment by Meenakshi Goel [ 27/Aug/14 ]
Started test with toy build http://qa.sc.couchbase.com/job/ubuntu_x64--65_02--view_query_extended-P1/171/console
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
not a blocker for RC2 - continue research and we can consider the fix for GA if it is low impact.
Comment by Meenakshi Goel [ 29/Aug/14 ]
I am unable to reproduce this issue with 3.0.0-1199-rel and with RC1(3.0.0-1174-rel) as test doesn't progresses after starting the View Query.
It has been observed that slave from which test is launched gets out of Memory as running python process consumes all the available memory due to which it exits when there is no memory left.
Test has been tried on multiple slaves and multiple clusters, Also tried with less number of items upto 1.5M but seeing same behaviour.

http://qa.sc.couchbase.com/job/ubuntu_x64--65_03--view_dgm_tests-P1/119/consoleFull




[MB-12096] collect_server_info.py does not work on a dev tree on windows.. Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Server
Component/s: test-execution
Affects Version/s: techdebt-backlog
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Trond Norbye Assignee: Tommie McAfee
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
cbcollectinfo.py tries to use ssh to collect the files even if the target machine is the same machine as the test is running on, and that doesn't seem to work on my windows development box. Since all of the files should be local it could might as well use normal copy.




[MB-12095] Change default values of the spatial view UI Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Server
Component/s: UI, view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Volker Mische Assignee: Volker Mische
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Spatial views don't need a bounding box anymore. If none is defined the full data set is returned. Additionally spatial views support skip and limit to page through the results.




[MB-4593] Windows Installer hangs on "Computing Space Requirements" Created: 27/Dec/11  Updated: 29/Aug/14

Status: Reopened
Project: Couchbase Server
Component/s: installer
Affects Version/s: 2.0-developer-preview-3, 2.0-developer-preview-4
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Bin Cui Assignee: Bin Cui
Resolution: Unresolved Votes: 3
Labels: windows, windows_pm_triaged
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Windows 7 Ultimate 64. Sony Vaio, i3 with 4GB RAM and 200 GB of 500 GB free. Also on a Sony Vaio, Windows 7 Ultimate 64, i7, 6 GB RAM and a 750GB drive with about 600 GB free.

Attachments: PNG File couchbase-installer.png     PNG File image001.png     PNG File ss 2014-08-28 at 4.16.09 PM.png    
Triage: Triaged

 Description   
When installing the Community Server 2.0 DP3 on Windows, the installer hangs on the "Computing space requirements screen." There is no additional feedback from the installer. After 90-120 minutes or so, it does move forward and complete. The same issue was reported on Google Groups a few months back - http://groups.google.com/group/couchbase/browse_thread/thread/37dbba592a9c150b/f5e6d80880f7afc8?lnk=gst&q=msi.

Executable: couchbase-server-community_x86_64_2.0.0-dev-preview-3.setup.exe

WORKAROUND IN 3.0 - Create a registry key HKLM\SOFTWARE\Couchbase, name=SkipVcRuntime, type=DWORD, value=1 to skip installing VC redistributable installation which is causing this issue. If VC redistributable is necessary, it must be installed manually if the registry key is set to skip automatic install of it.


 Comments   
Comment by Filip Stas [ 23/Feb/12 ]
Is there any solution for this? I'm experiencing the same problem. Running the unpacked msi does not seem to work because the Installshield setup has been configured to require to install through the exe.

Comment by Farshid Ghods (Inactive) [ 22/Mar/12 ]
from Bin:

Looks like it is related to installshield engine. Maybe installshield tries to access system registry and it is locked by other process. The suggestion is to shut down other running programs and try again if such problem pops up.
Comment by Farshid Ghods (Inactive) [ 22/Mar/12 ]
we were unable to reproduce this on windows 2008 64-bit

the bug mentions this happened on windows 7 64-bit which is not a supported platform but that should not make any difference
Comment by Farshid Ghods (Inactive) [ 23/Mar/12 ]
From Bin:

Windows 7 is my dev environment. And I have no problem to install and test it. From your description, I cannot tell whether it is failed during the installation or after installation finishes but couchcbase server cannot start.
 
If it is due to installshield failure, you can generate the log file for debugging as:
setup.exe /debuglog"C:\PathToLog\setupexe.log"
 
If Couchbase server fails to start, the most possible reason is due to missing or incompatible Microsoft runtime library. You can manually service_start.bat under bin directory and check what is going on. And you can run cbbrowse_log.bat to generate log file for further debugging.
Comment by John Zablocki (Inactive) [ 23/Mar/12 ]
This is an installation only problem. There's not much more to it other than the installer hangs on the screen (see attachment).

However, after a failed install, I did get it to work by:

a) deleting C:\Program Files\Couchbase\*

b) deleting all registry keys with Couchbase Server left over from the failed install

c) rebooting

Next time I see this problem, I'll run it again with the /debuglog

I think the problem might be that a previous install of DP3 or DP4 (nightly build) failed and left some bits in place somewhere.
Comment by Steve Yen [ 05/Apr/12 ]
from Perry...
Comment by Thuan Nguyen [ 05/Apr/12 ]
I can not repo this bug. I test on Windows 7 Professional 64 bit and Windows Server 2008 64 bit.
Here are steps:
- Install couchbase server 2.0.0r-388 (dp3)
- Open web browser and go to initial setup in web console.
- Uninstall couchbase server 2.0.0r-388
- Install couchbase server 2.0.0dp4r-722
- Open web browser and go to initial setup in web console.
Install and uninstall couchbase server go smoothly without any problem.
Comment by Bin Cui [ 25/Apr/12 ]
Maybe we need to get the installer verbose log file to get some clues.

setup.exe /verbose"c:\temp\logfile.txt"
Comment by John Zablocki (Inactive) [ 06/Jul/12 ]
Not sure if this is useful or not, but without fail, every time I encounter this problem, simply shutting down apps (usually Chrome for some reason) causes the hanging to stop. Right after closing Chrome, the C++ redistributable dialog pops open and installation completes.
Comment by Matt Ingenthron [ 10/Jul/12 ]
Workarounds/troubleshooting for this issue:


On installshield's website, there are similar problems reported for installshield. There are several possible reasons behind it:

1. The installation of the Microsoft C++ redistributable is blocked by some other running program, sometimes Chrome.
2. There are some remote network drives that are mapped to local system. Installshield may not have enough network privileges to access them.
3. Couchbase server was installed on the machine before and it was not totally uninstalled and/or removed. Installshield tried to recover from those old images.

To determine where to go next, run setup with debugging mode enabled:
setup.exe /debuglog"C:\temp\setupexe.log"

The contents of the log will tell you where it's getting stuck.
Comment by Bin Cui [ 30/Jul/12 ]
Matt's explanation should be included in document and Q&A website. I reproduced the hanging problem during installation if Chrome browser is running.
Comment by Farshid Ghods (Inactive) [ 30/Jul/12 ]
so does that mean the installer should wait until chrome and other browsers are terminated before proceeding ?

i see this as a very common use case with many installers that they ask the user to stop those applications and if user does not follow the instructions the set up process does not continue until these conditions are met.
Comment by Dipti Borkar [ 31/Jul/12 ]
Is there no way to fix this? At the least we need to provide an error or guidance that chrome needs to be quit before continuing. Is chrome the only one we have seen causing this problem?
Comment by Steve Yen [ 13/Sep/12 ]
http://review.couchbase.org/#/c/20552/
Comment by Steve Yen [ 13/Sep/12 ]
See CBD-593
Comment by Øyvind Størkersen [ 17/Dec/12 ]
Same bug when installing 2.0.0 (build-1976) on Windows 7. Stopping Chrome did not help, but killing the process "Logitech ScrollApp" (KhalScroll.exe) did..
Comment by Joseph Lam [ 13/Sep/13 ]
It's happening to me when installing 2.1.1 on Windows 7. What is this step for and it is really necessary? I see that it happens after the files have been copied to the installation folder. No entirely sure what it's computing space requirements for.
Comment by MikeOliverAZ [ 16/Nov/13 ]
Same problem on 2.2.0x86_64. I have tried everything, closing down chrome and torch from Task Manager to ensure no other apps are competing. Tried removing registry entries but so many, my time please. As is noted above this doesn't seem to be preventing writing the files under Program Files so what's it doing? So I cannot install, it now complains it cannot upgrade and run the installer again.

BS....giving up and going to MongoDB....it installs no sueat.

Comment by Sriram Melkote [ 18/Nov/13 ]
Reopening. Testing on VMs is a problem because they are all clones. We miss many problems like these.
Comment by Sriram Melkote [ 18/Nov/13 ]
Please don't close this bug until we have clear understanding of:

(a) What is the Runtime Library that we're trying to install that conflicts with all these other apps
(b) Why we need it
(c) A prioritized task to someone to remove that dependency on 3.0 release requirements

Until we have these, please do not close the bug.

We should not do any fixes on the lines of checking for known apps that conflict etc, as that is treating the symptom and not fixing the cause.
Comment by Bin Cui [ 18/Nov/13 ]
We install window runtime library because erlang runtime libraries depend on it. Not any runtime library, but the one that comes with erlang distribution package. Without it or with incompatible versions, erl.exe won't run.

In stead of checking any particular applications, the current solution is:
Run a erlang test script. If it runs correctly, no runtime library installed. Otherwise, installer has to install the runtime library.

Please see CBD-593.

Comment by Sriram Melkote [ 18/Nov/13 ]
My suggestion is that let us not attempt to install MSVCRT ourselves.

Let us check the library we need is present or not prior to starting the install (via appropriate registry keys).

If it is absent, let us direct the user to download and install it and exit.
Comment by Bin Cui [ 18/Nov/13 ]
The approach is not totally right. Even if the msvcrt exists, we still need to install it. Here the key is the absolute same msvrt package that comes with erlang distribution. We had problems before that with the same version, but different build of msvcrt installed, erlang won't run.

One possible solution is to ask user to download the msvcrt library from our website and make it a prerequisite for installing couchbase server.
Comment by Sriram Melkote [ 18/Nov/13 ]
OK. It looks like MS distributes some versions of VC runtime with the OS itself. I doubt that Erlang needs anything newer.

So let us rebuild Erlang and have it link to the OS supplied version of MSVCRT (i.e., msvcr70.dll) in Couchbase 3.0 onwards

In the meanwhile, let us point the user to the vcredist we ship in Couchbase 2.x versions and ask them to install it from there.
Comment by Steve Yen [ 23/Dec/13 ]
Saw this in the email inboxes...

From: Tal V
Date: December 22, 2013 at 1:19:36 AM PST
Subject: Installing Couchbase on Windows 7

Hi CouchBase support,
I would like to get your assist on an issue I’m having. I have a windows 7 machine on which I tried to install Couchbase, the installation is stuck on the “Computing space requirements”.
I tried several things without success:

1. 1. I tried to download a new installation package.

2. 2. I deleted all records of the software from the Registry.

3. 3. I deleted the folder that was created under C:\Program Files\Couchbase

4. 4. I restart the computer.

5. 5. Opened only the installation package.

6. 6. Re-install it again.
And again it was stuck on the same step.
What is the solution for it?

Thank you very much,


--
Tal V
Comment by Steve Yen [ 23/Dec/13 ]
Hi Bin,
Not knowing much about installshield here, but one idea - are there ways of forcibly, perhaps optionally, skipping the computing space requirements step? Some environment variable flag, perhaps?
Thanks,
Steve

Comment by Bin Cui [ 23/Dec/13 ]
This "Computing space requirements" is quite misleading. It happens at the post install step while GUI still shows that message. Within the step, we run the erlang test script and fails and the installer runs "vcredist.exe" for microsoft runtime library which gets stuck.

For the time being, the most reliable way is not to run this vcredist.exe from installer. Instead, we should provide a link in our download web site.

1. During installation, if we fails to run the erlang test script, we can pop up a warning dialog and ask customers to download and run it after installation.
 
Comment by Bin Cui [ 23/Dec/13 ]
To work around the problem, we can instruct the customer to download the vcredist.exe and run it manually before set up couchbase server. If running environment is set up correctly, installer will bypass that step.
Comment by Bin Cui [ 30/Dec/13 ]
Use windows registry key to install/skip the vcredist.exe step:

On 32bit windows, Installer will check HKEY_LOCAL_MACHINE\SOFTWARE\Couchbase\SkipVcRuntime
On 64bit windows, Installer will check HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Couchbase\SkipVcRuntime,
where SkipVcRuntime is a DWORD (32-bit) value.

When SkipVcRuntime is set to 1, installer will skip the step to install vcredist.exe. Otherwise, installer will follow the same logic as before.
vcredist_x86.exe can be found in the root directory of couchbase server. It can be run as:
c:\<couchbase_root>\vcredist_x86.exe

http://review.couchbase.org/#/c/31501/
Comment by Bin Cui [ 02/Jan/14 ]
Check into branch 2.5 http://review.couchbase.org/#/c/31558/
Comment by Iryna Mironava [ 22/Jan/14 ]
tested with Win 7 and Win Server 2008
I am unable to reproduce this issue(build 2.0.0-1976, dp3 is no longer available)
Installed/uninstalled couchbase several times
Comment by Sriram Melkote [ 22/Jan/14 ]
Unfortunately, for this problem, if it did not reproduce, we can't say it is fixed. We have to find a machine where it reproduces and then verify a fix.

Anyway, no change made actually addresses the underlying problem (the registry key just gives a way to workaround it when it happens), so reopening the bug and targeting for 3.0
Comment by Sriram Melkote [ 23/Jan/14 ]
Bin - I just noticed that the Erlang installer itself (when downloaded from their website) installs VC redistributable in non-silent mode. The Microsoft runtime installer dialog pop us up, indicates it will install VC redistributable and then complete. Why do we run it in silent mode (and hence assume liability of it running properly)? Why do we not run the MSI in interactive mode like ESL Erlang installer itself does?
Comment by Wayne Siu [ 05/Feb/14 ]
If we could get the information on the exact software version, it could be helpful.
From registry, Computer\HKLM\Software\Microsoft\WindowsNT\CurrentVersion
Comment by Wayne Siu [ 12/Feb/14 ]
Bin, looks like the erl.ini was locked when this issue happened.
Comment by Pavel Paulau [ 19/Feb/14 ]
Just happened to me in 2.2.0-837.
Comment by Anil Kumar [ 18/Mar/14 ]
Triaged by Don and Anil as per Windows Developer plan.
Comment by Bin Cui [ 08/Apr/14 ]
http://review.couchbase.org/#/c/35463/
Comment by Chris Hillery [ 13/May/14 ]
I'm new here, but it seems to me that vcredist_x64.exe does exactly the same thing as the corresponding MS-provided merge module for MSVC2013. If that's true, we should be able to just include that merge module in our project, and not need to fork out to install things. In fact, as of a few weeks ago, the 3.0 server installers are doing just that.

http://msdn.microsoft.com/en-us/library/dn501987.aspx

Is my understanding incomplete in some way?
Comment by Chris Hillery [ 14/May/14 ]
I can confirm that the most recent installers do install msvcr120.dll and msvcp120.dll in apparently the correct places, and the server can start with them. I *believe* this means that we no longer need to fork out vcredist_x64.exe, or have any of the InstallShield tricks to detect whether it is needed and/or skip installing it, etc. I'm leaving this bug open to both verify that the current merge module-based solution works, and to track removal of the unwanted code.
Comment by Sriram Melkote [ 16/May/14 ]
I've also verified that 3.0 build installed VCRT (msvcp100) is sufficient for Erlang R16.




[MB-12090] add stale=false semantic changes to dev guide Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0, 3.0-Beta
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Matt Ingenthron Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: No

 Description   
Need to change the dev guide to explain the semantics change with the stale parameter.

 Comments   
Comment by Matt Ingenthron [ 28/Aug/14 ]
I could not find the 3.0 dev guide to write up something. I've generated a diff based on the 2.5 dev guide. Note that much of that dev guide refers to the 3.0 admin guide section on views. I could not find that in the "dita" directory so I could contribute a change to the XML. I think based on this and what I put in MB-12052 should help.


diff --git a/content/couchbase-devguide-2.5/finding-data-with-views.markdown b/content/couchbase-devguide-2.5/finding-data-with-views.markdown
index 77735b9..811dff0 100644
--- a/content/couchbase-devguide-2.5/finding-data-with-views.markdown
+++ b/content/couchbase-devguide-2.5/finding-data-with-views.markdown
@@ -1,6 +1,6 @@
 # Finding Data with Views
 
-In Couchbase 2.1.0 you can index and query JSON documents using *views*. Views
+In Couchbase you can index and query JSON documents using *views*. Views
 are functions written in JavaScript that can serve several purposes in your
 application. You can use them to:
 
@@ -323,16 +323,25 @@ Forinformation about the sort order of indexes, see the
 [Couchbase Server Manual](http://docs.couchbase.com/couchbase-manual-2.5/cb-admin/).
 
 The real-time nature of Couchbase Server means that an index can become outdated
-fairly quickly when new entries and updates occur. Couchbase Server generates
-the index when it is queried, but in the meantime more data can be added to the
-server and this information will not yet be part of the index. To resolve this,
-Couchbase SDKs and the REST API provide a `stale` parameter you use when you
-query a view. With this parameter you can indicate you will accept the most
-current index as it is, you want to trigger a refresh of the index and retrieve
-these results, or you want to retrieve the existing index as is but also trigger
-a refresh of the index. For instance, to query a view with the stale parameter
-using the Ruby SDK:
+fairly quickly when new entries and updates occur. Couchbase Server updates
+the index at the time the query is received if you supply the argument
+`false` to the `stale` parameter.
+
+<div class="notebox">
+<p>Note</p>
+<p>Starting with the 3.0 release, the "stale" view query argument
+"false" has been enhanced so it will consider all document changes
+which have been received at the time the query has been received. This
+means that use of the `durability requirements` or `observe` feature
+to block for persistence in application code before issuing the
+`false` stale query is no longer needed. It is recommended that you
+remove all such application level checks after completing the upgrade
+to the 3.0 release.
+</p>
+</div>
 
+For instance, to query a view with the stale parameter
+using the Ruby SDK:
 
 ```
 doc.recent_posts(:body => {:stale => :ok})
@@ -905,13 +914,14 @@ for(ViewRow row : result) {
 }
 ```
 
-Before we create a Couchbase client instance and connect to the server, we set a
-system property 'viewmode' to 'development' to put the view into production
-mode. Then we query our view and limit the number of documents returned to 20
-items. Finally when we query our view we set the `stale` parameter to FALSE to
-indicate we want to reindex and include any new or updated beers in Couchbase.
-For more information about the `stale` parameter and index updates, see Index
-Updates and the Stale Parameter in the
+Before we create a Couchbase client instance and connect to the
+server, we set a system property 'viewmode' to 'development' to put
+the view into production mode. Then we query our view and limit the
+number of documents returned to 20 items. Finally when we query our
+view we set the `stale` parameter to FALSE to indicate we want to
+consider any recent changes to documents. For more information about
+the `stale` parameter and index updates, see Index Updates and the
+Stale Parameter in the
 [Couchbase Server Manual](http://docs.couchbase.com/couchbase-manual-2.5/cb-admin/#couchbase-views-writing-stale).
 
 The last part of this code sample is a loop we use to iterate through each item




[MB-12052] add stale=false semantic changes to release notes Created: 22/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0, 3.0-Beta
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Matt Ingenthron Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Need to release note the stale=false semantic changes.
This doesn't seem to be in current release notes, though MB-11589 seems loosely related.

Please use/adapt from the following text:
Starting with the 3.0 release, the "stale" view query argument "false" has been enhanced so it will consider all document changes which have been received at the time the query has been received. This means that use of the `durability requirements` or `observe` feature to block for persistence in application code before issuing the `false` stale query is no longer needed. It is recommended that you remove all such application level checks after completing the upgrade to the 3.0 release.

- - -

Ruth: assigning this to you to work out the right way to work the text into the release notes. This probably goes with a change in a different MB.




[MB-12089] Support in packages for installing as a different user from 'couchbase' Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: installer
Affects Version/s: 2.5.1, 3.0.1, 3.0, 3.0-Beta
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Major
Reporter: Brent Woodruff Assignee: Bin Cui
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We package Couchbase on Linux using RPM and DEB. During installation, a 'couchbase' user and group is created if it does not exist. It would be useful and practical to support the installation/upgrade of the package using a username and group name different than 'couchbase'.

I suggest we support the exporting of a name in the environment prior to installation that would affect the packaging scripts, and then maintain the username in a configuration file for running and upgrades.

i.e.

export COUCHBASE_USER=cbuser
export COUCHBASE_GROUP=cbgroup
rpm -ivh couchbase-server*.rpm

Results in using cbuser:cbgroup and records this in a configuration file somewhere that is sourced by the init.d script and the installer in the future for upgrades.

Based on Couchbase's ability to run in a non-root install and also on Mac as any Mac user, I don't believe there would be any problem with supporting this, as the user does not appear to be a hard, baked-in assumption anywhere.

 Comments   
Comment by Bin Cui [ 28/Aug/14 ]
Please assign it back to me if this request can be met in couchbase-server.sh script.
Comment by Bin Cui [ 28/Aug/14 ]
http://review.couchbase.org/#/c/41070/




[MB-10741] Paths on windows displayed in the logs looks like a mess Created: 03/Apr/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Minor
Reporter: Trond Norbye Assignee: Chris Hillery
Resolution: Unresolved Votes: 0
Labels: Windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Windows

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
When starting the server on windows paths is printed like:

c:/compile/couchbase/install\\lib\\foo\\bar

I don't know for sure if this is part of the reason why things are failing or not, but it just looks weird. http://review.couchbase.org/#/c/35131/ replace all of '\\' with '/' making them at least look the same

 Comments   
Comment by Aleksey Kondratenko [ 19/Jun/14 ]
I don't think it's 3.0 stuff, but in any case, I need either specific evidence from Trond who originally opened bug or dev environment that's set up in a way that I can run things myself and see.

If there's VM image so that I can run on my own hardware rather than stealing resources from everybody else, that would be best.
Comment by Anil Kumar [ 19/Jun/14 ]
Triage - June 19 2014 Alk, Parag, Anil
Comment by Anil Kumar [ 17/Jul/14 ]
Triage - Alk, Wayne, Anil, Tony .. July 17th




[MB-11245] Voltron build instructions for Windows are outdated Created: 29/May/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Dave Rigby Assignee: Chris Hillery
Resolution: Unresolved Votes: 0
Labels: windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
The voltron instructions from the voltron README.md still mention using package-win to build for windows, however Chris said in MB-11202 that it is no longer used:

> Windows doesn't use voltron's Makefile at all anymore. So far the way we get updated deps is to aask Trond to recompile and repackage them. So don't worry about it for this bug; as the description says, it's for linux and Mac.

The Readme therefore needs updating to reflect this.







[MB-9584] icu-config is shipped in our bin directory Created: 18/Nov/13  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: build
Affects Version/s: 2.1.1, 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Marty Schoch Assignee: Chris Hillery
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: MacOSX 64-bit

 Description   
Today I ran:

$ icu-config --ldconfig
### icu-config: Can't find /opt/couchbase/lib/libicuuc.dylib - ICU prefix is wrong.
### Try the --prefix= option
### or --detect-prefix
### (If you want to disable this check, use the --noverify option)
### icu-config: Exitting.

I was surprised that icu-config was in the couchbase bin dir, and therefore in my path.

I asked in IRC and no one seemed to think it was actually useful for anything, and it doesn't even appear to output valid paths anyway. Recommend we remove it to avoid any user confusion.

 Comments   
Comment by Aleksey Kondratenko [ 31/Jul/14 ]
It is actually useful on non-osx unix boxes. OSX is unique in a way due to this prefix changing/rewriting/whatever. On GNU/Linux icu we ship is actually usable to compile and link to.
Comment by Chris Hillery [ 31/Jul/14 ]
So, I could either figure out how to make icu-config work on a MacOS build, or eliminate it on MacOS. The latter is certainly quicker so I'll probably do that.




[MB-11567] [Windows] failed to load beer sample in initial setup Created: 26/Jun/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Thuan Nguyen Assignee: Chris Hillery
Resolution: Unresolved Votes: 0
Labels: Windows
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: windows 2008 R2 64-bit

Attachments: Zip Archive 10.17.1.166-6262014-1641-diag.zip    
Triage: Untriaged
Operating System: Windows 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: Link to binary http://factory.hq.couchbase.com:8080/job/cs_300_win6408/292/artifact/voltron/couchbase_server/3.0.0/881/couchbase_server-enterprise-windows-amd64-3.0.0-881.exe
Is this a Regression?: Yes

 Description   
Install couchbase server 3.0.0-881 on one node
Go through initial setup with beer-sample and gamesim-sample.
Couchbase server failed to load keys to beer-sample and gamesim-sample
Error:

Loading sample bucket gamesim-sample failed: eacces samples_loader_tasks000 ns_1@127.0.0.1 16:31:14 - Thu Jun 26, 2014
Loading sample bucket beer-sample failed: eacces samples_loader_tasks000 ns_1@127.0.0.1 16:31:14 - Thu Jun 26, 2014
Bucket "gamesim-sample" loaded on node 'ns_1@127.0.0.1' in 0 seconds. ns_memcached000 ns_1@127.0.0.1 16:31:13 - Thu Jun 26, 2014
Bucket "beer-sample" loaded on node 'ns_1@127.0.0.1' in 0 seconds. ns_memcached000 ns_1@127.0.0.1 16:31:13 - Thu Jun 26, 2014
Bucket "default" loaded on node 'ns_1@127.0.0.1' in 0 seconds. ns_memcached000 ns_1@127.0.0.1 16:31:12 - Thu Jun 26, 2014




 Comments   
Comment by Aleksey Kondratenko [ 26/Jun/14 ]
eaccess is something with permissions. Which points to some issue in windows installer.
Comment by Thuan Nguyen [ 21/Jul/14 ]
Tested on build 3.0.0-991, I still repro this bug with error eacces

Loading sample bucket gamesim-sample failed: eacces samples_loader_tasks000 ns_1@127.0.0.1 17:11:12 - Mon Jul 21, 2014
Loading sample bucket beer-sample failed: eacces samples_loader_tasks000 ns_1@127.0.0.1 17:11:12 - Mon Jul 21, 2014
Bucket "beer-sample" loaded on node 'ns_1@127.0.0.1' in 0 seconds. ns_memcached000 ns_1@127.0.0.1 17:11:11 - Mon Jul 21, 2014
Bucket "gamesim-sample" loaded on node 'ns_1@127.0.0.1' in 0 seconds. ns_memcached000 ns_1@127.0.0.1 17:11:10 - Mon Jul 21, 2014
Created bucket "gamesim-sample" of type: membase
[{num_replicas,1},
{replica_index,false},
{ram_quota,104857600},
{auth_type,sasl},
{flush_enabled,false},
{num_threads,3},
{eviction_policy,value_only}] menelaus_web012 ns_1@127.0.0.1 17:11:10 - Mon Jul 21, 2014
Created bucket "beer-sample" of type: membase
[{num_replicas,1},
{replica_index,false},
{ram_quota,104857600},
{auth_type,sasl},
{flush_enabled,false},
{num_threads,3},
{eviction_policy,value_only}] menelaus_web012 ns_1@127.0.0.1 17:11:10 - Mon Jul 21, 2014
Comment by Aleksey Kondratenko [ 22/Jul/14 ]
This is very sad. I cannot download this unfortunate installer.

Also keep in mind that although I might be able to figure it out I'm by no means a person that's suitable for this job. Something is messed up during install phase. It must be easy for installer folks to simply verify that samples loader works.
Comment by Thuan Nguyen [ 22/Jul/14 ]
windows builds are not in factory.hq.couchbase.com any more. They moved them to latestbuilds page as in the link below

http://builds.hq.northscale.net/latestbuilds/couchbase_server-enterprise-windows-amd64-3.0.0-999.exe
Comment by Aleksey Kondratenko [ 22/Jul/14 ]
$PREFIX/bin/tools/cbdocloader program is not converted python->exe and therefore doesn't work in windows.




[MB-10214] Mac version update check is incorrectly identifying newest version Created: 14/Feb/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: build
Affects Version/s: 2.0.1, 2.2.0, 2.1.1
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: David Haikney Assignee: Chris Hillery
Resolution: Unresolved Votes: 0
Labels: None
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified
Environment: Mac OS X

Attachments: PNG File upgrade_check.png    
Sub-Tasks:
Key
Summary
Type
Status
Assignee
MB-12051 Update the Release_Server job on Jenk... Technical task Open Chris Hillery  
Is this a Regression?: Yes

 Description   
Running 2.1.1 version of couchbase on a Mac, "check for latest version" reports the latest version is already running (e.g. see attached screenshot)


 Comments   
Comment by Aleksey Kondratenko [ 14/Feb/14 ]
Definitely not ui bug. It's using phone home to find out about upgrades. And I have no idea who owns that now.
Comment by Steve Yen [ 12/Jun/14 ]
got an email from ravi to look into this
Comment by Steve Yen [ 12/Jun/14 ]
Not sure if this is correct analysis, but I did a quick scan of what I think is the mac installer, which I think is...

  https://github.com/couchbase/couchdbx-app

It gets its version string by running a "git describe", in the Makefile here...

  https://github.com/couchbase/couchdbx-app/blob/master/Makefile#L1

Currently, a "git describe" on master branch returns...

  $ git describe
  2.1.1r-35-gf6646fa

...which is *kinda* close to the reported version string in the screenshot ("2.1.1-764-rel").

So, I'm thinking one fix needed would be a tagging (e.g., "git tag -a FOO -m FOO") of the couchdbx-app repository.

So, reassigning to Phil to do that appropriately.

Also, it looks like the our mac installer is using an open-source packaging / installer / runtime library called "sparkle" (which might be a little under-maintained -- not sure).

  https://github.com/andymatuschak/Sparkle/wiki

The sparkle library seems to check for version updates by looking at the URL here...

  https://github.com/couchbase/couchdbx-app/blob/master/cb.plist.tmpl#L42

Which seems to either be...

  http://appcast.couchbase.com/membasex.xml

Or, perhaps...

  http://appcast.couchbase.com/couchbasex.xml

The appcast.couchbase.com appears to be actually an S3 bucket, off of our production couchbase AWS account. So those *.xml files need to be updated, as they currently have content that has older versions. For example, http://appcast.couchbase.com/couchbase.xml looks currently like...

    <rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:sparkle="http://www.andymatuschak.org/xml-namespaces/sparkle" version="2.0">
    <channel>
    <title>Updates for Couchbase Server</title>
    <link>http://appcast.couchbase.com/couchbase.xml&lt;/link>
    <description>Recent changes to Couchbase Server.</description>
    <language>en</language>
    <item>
    <title>Version 1.8.0</title>
    <sparkle:releaseNotesLink>
    http://www.couchbase.org/wiki/display/membase/Couchbase+Server+1.8.0
    </sparkle:releaseNotesLink>
    <!-- date -u +"%a, %d %b %Y %H:%M:%S GMT" -->
    <pubDate>Fri, 06 Jan 2012 16:11:17 GMT</pubDate>
    <enclosure url="http://packages.couchbase.com/1.8.0/Couchbase-Server-Community.dmg" sparkle:version="1.8.0" sparkle:dsaSignature="MCwCFAK8uknVT3WOjPw/3LkQpLBadi2EAhQxivxe2yj6EU6hBlg9YK/5WfPa5Q==" length="33085691" type="application/octet-stream"/>
    </item>
    </channel>
    </rss>

Not updating the xml files, though, probably causes no harm. Just that our osx users won't be pushed news on updates.
Comment by Phil Labee [ 12/Jun/14 ]
This has nothing to do with "git describe". There should be no place in the product that "git describe" should be used to determine version info. See:

    http://hub.internal.couchbase.com/confluence/display/CR/Branching+and+Tagging

so there's definitely a bug in the Makefile.

The version update check seems to be out of date. The phone-home file is generated during:

    http://factory.hq.couchbase.com:8080/job/Product_Staging_Server/

but the process of uploading it is not automated.
Comment by Steve Yen [ 12/Jun/14 ]
Thanks for the links.

> This has nothing to do with "git describe".

My read of the Makefile makes me think, instead, that "git describe" is the default behavior unless it's overridden by the invoker of the make.

> There should be no place in the product that "git describe" should be used to determine version info. See:
> http://hub.internal.couchbase.com/confluence/display/CR/Branching+and+Tagging

It appears all this couchdbx-app / sparkle stuff predates that wiki page by a few years, so I guess it's inherited legacy.

Perhaps voltron / buildbot are not setting the PRODUCT_VERSION correctly before invoking the the couchdbx-app make, which makes the Makefile default to 'git describe'?

    commit 85710d16b1c52497d9f12e424a22f3efaeed61e4
    Date: Mon Jun 4 14:38:58 2012 -0700

    Apply correct product version number
    
    Get version number from $PRODUCT_VERSION if it's set.
    (Buildbot and/or voltron will set this.)
    If not set, default to `git describe` as before.
    
> The version update check seems to be out of date.

Yes, that's right. The appcast files are out of date.

> The phone-home file is generated during:
> http://factory.hq.couchbase.com:8080/job/Product_Staging_Server/

I think appcast files for OSX / sparkle are a _different_ mechanism than the phone-home file, and an appcast XML file does not appear to be generated/updated by the Product_Staging_Server job.

But, I'm not an expert or really qualified on the details here -- this is just my opinions from a quick code scan, not from actually doing/knowing.

Comment by Wayne Siu [ 01/Aug/14 ]
Per PM (Anil), we should get this fixed by 3.0 RC1.
Raising the priority to Critical.
Comment by Wayne Siu [ 07/Aug/14 ]
Phil,
Please provide update.
Comment by Anil Kumar [ 12/Aug/14 ]
Triage - Upgrading to 3.0 Blocker

Comment by Wayne Siu [ 20/Aug/14 ]
Looks like we may have a short term "fix" for this ticket which Ceej and I have tested.
@Ceej, can you put in the details here?
Comment by Chris Hillery [ 20/Aug/14 ]
The file is hosted in S3, and we proved tonight that overwriting that file (membasex.xml) with a version containing updated version information and download URLs works as expected. We updated it to point to 2.2 for now, since that is the latest version with a freely-available download URL.

We can update the Release_Server job on Jenkins to create an updated version of this XML file from a template, and upload it to S3.

Assigning back to Wayne for a quick question: Do we support Enterprise edition for MacOS? If we do, then this solution won't be sufficient without more effort, because the two editions will need different Sparkle configurations for updates. Also, Enterprise edition won't be able to directly download the newer release, unless we provide a "hidden" URL for that (the download link on the website goes to a form).




[MB-12085] query with key=value requires inclusive_end=true Created: 27/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Tommie McAfee Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Yes

 Description   
In 3.0 the view editor creates urls with inclusive_end=false. For some reason this causes key=value queries to return empty results. In 2.5 the param was left unset.

1. load beer sample

2. created view with map:

function (doc, meta) {
   emit(doc.state, doc.name);
}

3. In filter result drop down:
key = "Alaska"

Generates:
http://172.23.106.53:8092/beer-sample/_design/dev_ddoc1/_view/view1?stale=false&inclusive_end=false&key=%22Alaska%22&connection_timeout=60000&limit=10&skip=0

need to manually check inclusive_end=false.

2.5 Generates:
http://172.23.106.53:8092/beer-sample/_design/dev_ddoc1/_view/view1?stale=false&key=%22Alaska%22&connection_timeout=60000&limit=10&skip=0
 

If I didn't look at the url generated in 2.5 I'm not sure if I would've got this to work.

 Comments   
Comment by Volker Mische [ 28/Aug/14 ]
Assigning to "UI" as this is not a view engine issue but a UI one. Also assigning it to Alk.
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
I need somebody to decide if it's 3.0.0 or 3.0.1.
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
at this point if it isn't related to data loss, we won't take it.
could this be causing apps to break? potentially turn into a hotfix if we don't fix it now?
-cihan
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
3.0.1 then




[MB-12079] Cannot edit documents with textual or numeric data. Created: 27/Aug/14  Updated: 28/Aug/14

Status: Reopened
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0-Beta
Fix Version/s: None
Security Level: Public

Type: Task Priority: Minor
Reporter: Brett Lawson Assignee: Sriram Melkote
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When attempting to create or modify a document in the Web UI, and that value is a string or a number, errors occur preventing you from saving the document.

 Comments   
Comment by Matt Ingenthron [ 27/Aug/14 ]
Related is MB-12078. Looks like the validator we're using in the console is wrong.
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
It is not "validator is wrong". It's deliberate choice to refuse editing such values.
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
See MB-9208
Comment by Matt Ingenthron [ 27/Aug/14 ]
It looks like things have changed for the better and the console is a bit out of sync. Now views do work with numbers such as "1" which may be incr/decr'd too. So, what is said in MB-9208 isn't valid any more. Brett verified this.

At this stage you can insert "1234", incr it, but you can't insert it through the console because it says "JSON should represent an object": http://puu.sh/b9H8i.png

Do you see a downside to syncing up the console to what views/view-engine actually do these days?

Comment by Aleksey Kondratenko [ 27/Aug/14 ]
Have you tried arrays ?
Comment by Matt Ingenthron [ 28/Aug/14 ]
I have not, but you certainly could. The use case we were looking to was the number with incr/decr. Strings should also be handled. From code inspection, it looks like the current JSON parser will handle these things correctly.
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
Parsers ? Or parser?
Comment by Matt Ingenthron [ 28/Aug/14 ]
To my knowledge, as of 3.0 the memcached process does the "isJson()" identification and then in the view engine, that'll control whether meta.type is document or base64. So, I was referring to that parser in 3.0. Sorry for the lack of clarity. There could be other issues at the view-engine level or at the console level, but at least for numbers it seems to do what users want it to do at the view-engine, memcached, SDK level. The console doesn't allow insert/edit even though 1234 is technically JSON.
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
3.0's ep-engine to my surprise indeed does full json detection (unlike 2.5 which AFAIK only allowed objects).

But AFAIK 3.0.0's view engine _does not_ use datatype inferred by memcached. Same is true for some obscure CAPI XDCR paths which need json-ness. For xdcr _I know_ that detection happens via couchdb routines. I'm sure there's no other way for views to do json detection.

BTW this is one of areas where some higher level entity could finally decide something. And let me say that I'm not eager to let my team work on this until there's clear resolution on json detection (from tag or from ep-engine folks or from anybody else).

Comment by Matt Ingenthron [ 28/Aug/14 ]
Siri, can you clarify? From one of our previous conversations I was under the impression that over DCP, view-engine is trusting the JSON detection done in the memcached process. Is that accurate that as of 3.0,
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
Haven't we made decision to disable data type in 3.0?
Comment by Matt Ingenthron [ 28/Aug/14 ]
Alk: agreed this should be a higher level thing. We ended up having to disable datatype at memcached HELLO because if it had gone in and a mixed version cluster had failed over, logical corruption would have occurred. That's one of the areas that probably should have had better review.

That said, I'm just in the "what do we do given the current state" kind of mode. The other changes are in and good-- I don't see any reason we'd avoid making the Web UI changes in a future release.




[MB-12091] [Windows].compact files not cleaned up after compaction Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Venu Uppalapati Assignee: Sriram Ganesan
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive cbinfo.zip    
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Steps to reproduce:
1)Load 1M items on default so that compaction runs several times by end of the loading.
2)It will be seen that all the temporary .compact files are kept even after compaction is done.
3)for ex:
*.couch.1.compact.btree-tmp-1
*.couch.1.compact.btree-tmp-2
*.couch.1.compact.btree-tmp-3
*.couch.1.compact.btree-tmp-4
when the current rev number of the vbucket file is 5
4)This is windows specific. Pretty soon disk space is claimed by all these temp files.

 Comments   
Comment by Chiyoung Seo [ 28/Aug/14 ]
Sriram,

Sundar is busy with working on the RC2-related issue. Can you please take a look at this window compaction issue?
Comment by Sriram Ganesan [ 28/Aug/14 ]
If there are any logs available from the test, please do upload them.
Comment by Venu Uppalapati [ 28/Aug/14 ]
Sundar mentioned that he used unlink function which is deprecated on Windows version of the compiler, http://msdn.microsoft.com/en-us/library/ms235350.aspx will upload the logs shortly.
Comment by Venu Uppalapati [ 28/Aug/14 ]
cbcollectinfo attached




[MB-12094] Integrate ForestDB into EP-Engine Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: feature-backlog
Fix Version/s: feature-backlog
Security Level: Public

Type: Bug Priority: Critical
Reporter: Chiyoung Seo Assignee: Sundar Sridharan
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
We plan to replace Couchstore with ForestDB as the underlying storage engine for the EP-Engine. This integration task mainly requires the implementation of the abstract KV APIs defined in the EP-Engine.
 




[MB-11633]  Support append / prepend APIs Created: 03/Jul/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: forestdb
Affects Version/s: feature-backlog
Fix Version/s: feature-backlog
Security Level: Public

Type: Task Priority: Major
Reporter: Patrick Varley Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Heavy append workload.

Issue Links:
Dependency
blocks MB-11591 Optimism pre/append operations throug... Open

 Description   
Heavy append workload can put pressure on the disks

Instead of writing the whole doc, we should be able to write just the append chunky and then read across all the chunky to get the document, of course the compactor would have to do the same.




[MB-7250] Mac OS X App should be signed by a valid developer key Created: 22/Nov/12  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: build, installer
Affects Version/s: 2.0-beta-2, 2.1.0, 2.2.0, 2.5.0, 2.5.1
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: J Chris Anderson Assignee: Phil Labee
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Build_2.5.0-950.png     PNG File Screen Shot 2013-02-17 at 9.17.16 PM.png     PNG File Screen Shot 2013-04-04 at 3.57.41 PM.png     PNG File Screen Shot 2013-08-22 at 6.12.00 PM.png     PNG File ss_2013-04-03_at_1.06.39 PM.png    
Issue Links:
Dependency
depends on MB-9437 macosx installer package fails during... Closed
Relates to
relates to CBLT-104 Enable Mac developer signing on Mac b... Open
Is this a Regression?: No

 Description   
Currently launching the Mac OS X version tells you it's from an unidentified developer. You have to right click to launch the app. We can fix this.

 Comments   
Comment by Farshid Ghods (Inactive) [ 22/Nov/12 ]
Chris,

do you know what needs to change on the build machine to embed our developer key ?
Comment by J Chris Anderson [ 22/Nov/12 ]
I have no idea. I could start researching how to get a key from Apple but maybe after the weekend. :)
Comment by Farshid Ghods (Inactive) [ 22/Nov/12 ]
we can discuss this next week : ) . Thanks for reporting the issue Chris.
Comment by Steve Yen [ 26/Nov/12 ]
we'll want separate, related bugs (tasks) for other platforms, too (windows, linux)
Comment by Jens Alfke [ 30/Nov/12 ]
We need to get a developer ID from Apple; this will give us some kind of cert, and a local private key for signing.
Then we need to figure out how to get that key and cert onto the build machine, in the Keychain of the account that runs the buildbot.
Comment by Farshid Ghods (Inactive) [ 02/Jan/13 ]
the instructions to build is available here :
https://github.com/couchbase/couchdbx-app
we need to add codesign as a build step there
Comment by Farshid Ghods (Inactive) [ 22/Jan/13 ]
Phil,

do you have any update on this ticket. ?
Comment by Phil Labee [ 22/Jan/13 ]
I have signing cert installed on 10.17.21.150 (MacBuild).

Change to Makefile: http://review.couchbase.org/#/c/24149/
Comment by Phil Labee [ 23/Jan/13 ]
need to change master.cfg and pass env.var. to package-mac
Comment by Phil Labee [ 29/Jan/13 ]
disregard previous. Have added signing to Xcode projects.

see http://review.couchbase.org/#/c/24273/
Comment by Phil Labee [ 31/Jan/13 ]
To test this go to System Preferences / Security & Privacy, and on the General tab set "Allow applications downloaded from" to "Mac App Store and Identified Developers". Set this before running Couchbase Server.app the first time. Once an app has been allowed to run this setting is no longer checked for that app, and there doesn't seem to be a way to reset that.

What is odd is that on my system, I allowed one unsigned build to run before restricting the app run setting, and then no other unsigned builds would be checked (and would all be allowed to run). Either there is a flaw in my testing methodology, or a serious weakness in this security setting: Just because one app called Couchbase Server was allowed to run should confer this privilege to other apps with the same name. A common malware tactic is to modify a trusted app and distribute it as update, and if the security setting keys off the app name it will do nothing to prevent that.

I'm approving this change without having satisfactorily tested it.
Comment by Jens Alfke [ 31/Jan/13 ]
Strictly speaking it's not the app name but its bundle ID, i.e. "com.couchbase.CouchbaseServer" or whatever we use.

> I allowed one unsigned build to run before restricting the app run setting, and then no other unsigned builds would be checked

By OK'ing an unsigned app you're basically agreeing to toss security out the window, at least for that app. This feature is really just a workaround for older apps. By OK'ing the app you're not really saying "yes, I trust this build of this app" so much as "yes, I agree to run this app even though I don't trust it".

> A common malware tactic is to modify a trusted app and distribute it as update

If it's a trusted app it's hopefully been signed, so the user wouldn't have had to waive signature checking for it.
Comment by Jens Alfke [ 31/Jan/13 ]
Further thought: It might be a good idea to change the bundle ID in the new signed version of the app, because users of 2.0 with strict security settings have presumably already bypassed security on the unsigned version.
Comment by Jin Lim [ 04/Feb/13 ]
Per bug scrubs, keep this a blocker since customers ran into this issues (and originally reported it).
Comment by Phil Labee [ 06/Feb/13 ]
revert the change so that builds can complete. App is currently not being signed.
Comment by Farshid Ghods (Inactive) [ 11/Feb/13 ]
i suggest for 2.0.1 release we do this build manually.
Comment by Jin Lim [ 11/Feb/13 ]
As one-off fix, add the signature manually and automate the required steps later in 2.0.2 or beyond.
Comment by Jin Lim [ 13/Feb/13 ]
Please move this bug to 2.0.2 after populating the required signature manually. I am lowing the severity to critical for it isn't no longer a blocking issue.
Comment by Farshid Ghods (Inactive) [ 15/Feb/13 ]
Phil to upload the binary to latestbuilds , ( 2.0.1-101-rel.zip )
Comment by Phil Labee [ 15/Feb/13 ]
Please verify:

http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-160-rel-signed.zip
Comment by Phil Labee [ 15/Feb/13 ]
uploaded:

http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-160-rel-signed.zip

I can rename it when uploading for release.
Comment by Farshid Ghods (Inactive) [ 17/Feb/13 ]
i still do get the error that it is from an identified developer.

Comment by Phil Labee [ 18/Feb/13 ]
operator error.

I rebuilt the app, this time verifying that the codesign step occurred.

Uploaded now file to same location:

http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-160-rel-signed.zip
Comment by Phil Labee [ 26/Feb/13 ]
still need to perform manual workaround
Comment by Phil Labee [ 04/Mar/13 ]
release candidate has been uploaded to:

http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-172-signed.zip
Comment by Wayne Siu [ 03/Apr/13 ]
Phil, looks like version 172/185 is still getting the error. My Mac version is 10.8.2
Comment by Thuan Nguyen [ 03/Apr/13 ]
Install couchbase server (build 2.0.1-172 community version) in my mac osx 10.7.4 , I only see the warning message
Comment by Wayne Siu [ 03/Apr/13 ]
Latest version (04.03.13) : http://builds.hq.northscale.net/latestbuilds/couchbase-server-community_x86_64_2.0.1-185-rel.zip
Comment by Maria McDuff (Inactive) [ 03/Apr/13 ]
works in 10.7 but not in 10.8.
if we can get the fix for 10.8 by tomorrow, end of day, QE is willing to test for release on tuesday, april 9.
Comment by Phil Labee [ 04/Apr/13 ]
The mac builds are not being automatically signed, so build 185 is not signed. The original 172 is also not signed.

Did you try

    http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-172-signed.zip

to see if that was signed correctly?

Comment by Wayne Siu [ 04/Apr/13 ]
Phil,
Yes, we did try the 172-signed version. It works on 10.7 but not 10.8. Can you take a look?
Comment by Phil Labee [ 04/Apr/13 ]
I rebuilt 2.0.1-185 and uploaded a signed app to:

    http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-185-rel.SIGNED.zip

Test on a machine that has never had Couchbase Server installed, and has the security setting to only allow Appstore or signed apps.

If you get the "Couchbase Server.app was downloaded from the internet" warning and you can click OK and install it, then this bug is fixed. The quarantining of files downloaded by a browser is part of the operating system and is not controlled by signing.
Comment by Wayne Siu [ 04/Apr/13 ]
Tried the 185-signed version (see attached screen shot). Same error message.
Comment by Phil Labee [ 04/Apr/13 ]
This is not an error message related to this bug.

Comment by Maria McDuff (Inactive) [ 14/May/13 ]
per bug triage, we need to have mac 10.8 osx working since it is a supported platform (published in the website).
Comment by Wayne Siu [ 29/May/13 ]
Work Around:
Step One
Hold down the Control key and click the application icon. From the contextual menu choose Open.

Step Two
A popup will appear asking you to confirm this action. Click the Open button.
Comment by Anil Kumar [ 31/May/13 ]
we need to address signed key for both Windows and Mac deferring this to next release.
Comment by Dipti Borkar [ 08/Aug/13 ]
Please let's make sure this is fixed in 2.2.
Comment by Phil Labee [ 16/Aug/13 ]
New keys will be created using new account.
Comment by Phil Labee [ 20/Aug/13 ]
iOS Apps
--------------
Certificates:
  Production:
    "Couchbase, Inc." type=iOS Distribution expires Aug 12, 2014

    ~buildbot/Desktop/appledeveloper.couchbase.com/certs/ios/ios_distribution_appledeveloper.couchbase.com.cer

Identifiers:
  App IDS:
    "Couchbase Server" id=com.couchbase.*

Provisining Profiles:
  Distribution:
    "appledeveloper.couchbase.com" type=Distribution

  ~buildbot/Desktop/appledeveloper.couchbase.com/profiles/ios/appledevelopercouchbasecom.mobileprovision
Comment by Phil Labee [ 20/Aug/13 ]
Mac Apps
--------------
Certificates:
  Production:
    "Couchbase, Inc." type=Mac App Distribution (Aug,15,2014)
    "Couchbase, Inc." type=Developer ID installer (Aug,16,2014)
    "Couchbase, Inc." type=Developer ID Application (Aug,16,2014)
    "Couchbase, Inc." type=Mac App Distribution (Aug,15,2014)

     ~buildbot/Desktop/appledeveloper.couchbase.com/certs/mac_app/mac_app_distribution.cer
     ~buildbot/Desktop/appledeveloper.couchbase.com/certs/mac_app/developerID_installer.cer
     ~buildbot/Desktop/appledeveloper.couchbase.com/certs/mac_app/developererID_application.cer
     ~buildbot/Desktop/appledeveloper.couchbase.com/certs/mac_app/mac_app_distribution-2.cer

Identifiers:
  App IDs:
    "Couchbase Server" id=couchbase.com.* Prefix=N2Q372V7W2
    "Coucbase Server adhoc" id=couchbase.com.* Prefix=N2Q372V7W2
    .

Provisioning Profiles:
  Distribution:
    "appstore.couchbase.com" type=Distribution
    "Couchbase Server adhoc" type=Distribution

     ~buildbot/Desktop/appledeveloper.couchbase.com/profiles/appstorecouchbasecom.privisioningprofile
     ~buildbot/Desktop/appledeveloper.couchbase.com/profiles/Couchbase_Server_adhoc.privisioningprofile

Comment by Phil Labee [ 21/Aug/13 ]

As of build 2.2.0-806 the app is signed by a new provisioning profile
Comment by Phil Labee [ 22/Aug/13 ]
 Install version 2.2.0-806 on a macosx 10.8 machine that has never had Couchbase Server installed, which has the security setting to require applications to be signed with a developer ID.
Comment by Phil Labee [ 22/Aug/13 ]
please assign to tester
Comment by Maria McDuff (Inactive) [ 22/Aug/13 ]
just tried this against newest build 809:
still getting restriction message. see attached.
Comment by Maria McDuff (Inactive) [ 22/Aug/13 ]
restriction still exists.
Comment by Maria McDuff (Inactive) [ 28/Aug/13 ]
verified in rc1 (build 817). still not fixed. getting same msg:
“Couchbase Server” can’t be opened because it is from an unidentified developer.
Your security preferences allow installation of only apps from the Mac App Store and identified developers.

Work Around:
Step One
Hold down the Control key and click the application icon. From the contextual menu choose Open.

Step Two
A popup will appear asking you to confirm this action. Click the Open button.
Comment by Phil Labee [ 03/Sep/13 ]
Need to create new certificates to replace these that were revoked:

Certificate: Mac Development
Team Name: Couchbase, Inc.

Certificate: Mac Installer Distribution
Team Name: Couchbase, Inc.

Certificate: iOS Development
Team Name: Couchbase, Inc.

Certificate: iOS Distribution
Team Name: Couchbase, Inc.
Comment by Maria McDuff (Inactive) [ 18/Sep/13 ]
candidate for 2.2.1 bug fix release.
Comment by Dipti Borkar [ 28/Oct/13 ]
This is going to make it into 2.5? We seemed to keep differing it?
Comment by Phil Labee [ 29/Oct/13 ]
cannot test changes with installer that fails
Comment by Phil Labee [ 11/Nov/13 ]
Installed certs as buildbot and signed app with "(recommended) 3rd Party Mac Developer Application", producing

    http://factory.hq.couchbase.com//couchbase_server_2.5.0_MB-7250-001.zip

Signed with "(Oct 30) 3rd Party Mac Developer Application: Couchbase, Inc. (N2Q372V7W2)", producing

    http://factory.hq.couchbase.com//couchbase_server_2.5.0_MB-7250-002.zip

These zip files were made on the command line, not a result of the make command. They are 2.5G in size, so they obviously include mote than the zip files produced by the make command.

Both versions of the app appear to be signed correctly!

Note: cannot run make command from ssh session. Must Remote Desktop in and use terminal shell natively.
Comment by Phil Labee [ 11/Nov/13 ]
Finally, some progress: If the zip file is made using the --symlinks argument it appears to be un-signed. If the symlinked files are included, the app appears to be signed correctly.

The zip file with symlinks is 60M, while the zip file with copies of the files is 2.5G, more than 40X the size.
Comment by Phil Labee [ 25/Nov/13 ]
Fixed in 2.5.0-950
Comment by Dipti Borkar [ 25/Nov/13 ]
Maria, can QE please verify this?
Comment by Wayne Siu [ 28/Nov/13 ]
Tested with build 2.5.0-950. Still see the warning box (attached).
Comment by Wayne Siu [ 19/Dec/13 ]
Phil,
Can you give an update on this?
Comment by Ashvinder Singh [ 14/Jan/14 ]
I tested the code signature with apple utility "spctl -a -v /Applications/Couchbase\ Server.app/" and got the output :
>>> /Applications/Couchbase Server.app/: a sealed resource is missing or invalid

also tried running the command:
 
bash: codesign -dvvvv /Applications/Couchbase\ Server.app
>>>
Executable=/Applications/Couchbase Server.app/Contents/MacOS/Couchbase Server
Identifier=com.couchbase.couchbase-server
Format=bundle with Mach-O thin (x86_64)
CodeDirectory v=20100 size=639 flags=0x0(none) hashes=23+5 location=embedded
Hash type=sha1 size=20
CDHash=868e4659f4511facdf175b44a950b487fa790dc4
Signature size=4355
Authority=3rd Party Mac Developer Application: Couchbase, Inc. (N2Q372V7W2)
Authority=Apple Worldwide Developer Relations Certification Authority
Authority=Apple Root CA
Signed Time=Jan 8, 2014, 10:59:16 AM
Info.plist entries=31
Sealed Resources version=1 rules=4 files=5723
Internal requirements count=1 size=216

It looks like the code signature is present but got invalid as the new file were added/modified to the project. I suggest for the build team to rebuild and add the code signature again.
Comment by Phil Labee [ 17/Apr/14 ]
need VM to clone for developer experimentation
Comment by Anil Kumar [ 18/Jul/14 ]
Any update on this? We need this for 3.0.0 GA.

Please update the ticket.

Triage - July 18th
Comment by Wayne Siu [ 02/Aug/14 ]
Siri is helping to figure out what the next step is.
Comment by Anil Kumar [ 13/Aug/14 ]
Jens - Assigning as per Ravi's request.
Comment by Chris Hillery [ 13/Aug/14 ]
Jens requested assistance in setting up a MacOS development environment for building Couchbase. Phil (or maybe Siri?), can you help him with that?
Comment by Phil Labee [ 13/Aug/14 ]
The production macosx builder has been cloned:

    10.6.2.159 macosx-x64-server-builder-01-clone

if you want to use your own host, see:

    http://hub.internal.couchbase.com/confluence/display/CR/How+to+Setup+a+MacOSX+Server+Build+Node
Comment by Jens Alfke [ 15/Aug/14 ]
Here are the Apple docs on building apps signed with a Developer ID: https://developer.apple.com/library/mac/documentation/IDEs/Conceptual/AppDistributionGuide/DistributingApplicationsOutside/DistributingApplicationsOutside.html#//apple_ref/doc/uid/TP40012582-CH12-SW2

I've got everything configured, but the build process fails at the final step, after I press the Distribute button in the Organizer window. I get a very uninformative error alert, "Code signing operation failed / Check that the identity you selected is valid."

I've asked for help on the xcode-users mailing list. Blocked until I hear something back.
Comment by Anil Kumar [ 18/Aug/14 ]
Triage - Not blocking 3.0 RC1
Comment by Phil Labee [ 25/Aug/14 ]
from Apple Developer mail list:

Dear Developer,

With the release of OS X Mavericks 10.9.5, the way that OS X recognizes signed apps will change. Signatures created with OS X Mountain Lion 10.8.5 or earlier (v1 signatures) will be obsoleted and Gatekeeper will no longer recognize them. Users may receive a Gatekeeper warning and will need to exempt your app to continue using it. To ensure your apps will run without warning on updated versions of OS X, they must be signed on OS X Mavericks 10.9 or later (v2 signatures).

If you build code with an older version of OS X, use OS X Mavericks 10.9 or later to sign your app and create v2 signatures using the codesign tool. Structure your bundle according to the signature evaluation requirements for OS X Mavericks 10.9 or later. Considerations include:

 * Signed code should only be placed in directories where the system expects to find signed code.

 * Resources should not be located in directories where the system expects to find signed code.

 * The --resource-rules flag and ResourceRules.plist are not supported.

Make sure your current and upcoming releases work properly with Gatekeeper by testing on OS X Mavericks 10.9.5 and OS X Yosemite 10.10 Developer Preview 5 or later. Apps signed with v2 signatures will work on older versions of OS X.

For more details, read “Code Signing changes in OS X Mavericks” and “Changes in 
OS X 10.9.5 and Yosemite Developer Preview 5” in OS X Code Signing In Depth":

    http://c.apple.com/r?v=2&la=en&lc=us&a=EEjRsqZNfcheZauIAhlqmxVG35c6HJuf50mGu47LWEktoAjykEJp8UYqbgca3uWG&ct=AJ0T0e3y2W

Best regards,
Apple Developer Technical Support
Comment by Phil Labee [ 28/Aug/14 ]
change to buildbot-internal to unlock keychain before running make and lock after:

    http://review.couchbase.org/#/c/41028/

change to couchdbx-app to sign app, on dev branch "plabee/MB-7250":

    http://review.couchbase.org/#/c/41025/

change to manifest to use this dev branch for 3.0.1 builds:

    http://review.couchbase.org/#/c/41026/




[MB-12092] Spatial views UI doesn't display results Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: UI, view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Volker Mische Assignee: Volker Mische
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
When you have a spatial view and click on "Show Results" no results are shown.

 Comments   
Comment by Volker Mische [ 28/Aug/14 ]
http://review.couchbase.org/41024




[MB-12093] Spatial views return value is double encoded Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: None
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Volker Mische Assignee: Volker Mische
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Yes

 Description   
When you query a spatial view then the `value` in the response is double JSON encoded as string. This means that if you return a number it becomes a string. And a string becomes a string with escaped quotes. For example:

The emit -> current value in the response -> expected value in the response
emit(key, 5) -> {"value: "5"} -> {"value": 5}
emit(key, "some string"} -> {"value": "\"some string\""} -> {"value": "some string"}
emit(key, null) -> {"value: "null"} -> {"value": null}


 Comments   
Comment by Volker Mische [ 28/Aug/14 ]
http://review.couchbase.org/41027




[MB-10790] Transaction log support for individual mutations Created: 07/Apr/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: feature-backlog
Fix Version/s: feature-backlog
Security Level: Public

Type: Bug Priority: Critical
Reporter: Chiyoung Seo Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
There is always a time window that we lose a given mutation from an application, because we do both persistence and replication in an async manner. To address this limitation, we need to consider supporting a transaction (commit) log for individual mutations from applications, and later extend it to support a full transaction on multi docs across different nodes.


 Comments   
Comment by Matt Ingenthron [ 07/Apr/14 ]
+1

There were some earlier thoughts on how to accomplish this that I can share if it'd be useful.
Comment by Chiyoung Seo [ 07/Apr/14 ]
Thanks Matt. Please feel free to share them with me. We can schedule a separate meeting if necessary.
Comment by Matt Ingenthron [ 07/Apr/14 ]
Sure, thus week is bad, but want to grab 30 mins next week?
Comment by Chiyoung Seo [ 07/Apr/14 ]
Sure, I will then schedule a meeting sometime next week. Thanks!




[MB-10789] Bloom Filter based optimization to reduce the I/O overhead Created: 07/Apr/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: feature-backlog
Fix Version/s: feature-backlog
Security Level: Public

Type: Bug Priority: Critical
Reporter: Chiyoung Seo Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
A bloom filter can be considered as an optimization to reduce the disk IO overhead. Basically, we maintain a separate bloom filter per vbucket database file, and rebuild the bloom filter (e.g., increasing the filter size to reduce a false positive error rate) as part of vbucket database compaction.

As we know the number of items in a vBucket database file, we can determine the number of hash functions and the size of the bloom filter to achieve the desired false positive error rate. Note that Murmur hash has been widely used in Hadoop and Cassandra because it is much faster than MD5 and Jenkins. It has been widely known that fewer than 10 bits per element are required for a 1% false positive probability, independent of the number of elements in the set.

We expect that having a bloom filter will enhance both XDCR and full-ejection cache management performance at the expense of the filter's memory overhead.






[MB-10788] Support synchronous replication through DCP for mutations (SET, DEL, etc.) from applications Created: 07/Apr/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: feature-backlog
Fix Version/s: feature-backlog
Security Level: Public

Type: Bug Priority: Critical
Reporter: Chiyoung Seo Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
There are more and more demands on synchronous replication support for mutation events from applications. As a starting point, we need to investigate how this can be supported for the replication between master and slave nodes in the same cluster (e.g., quorum based).


 Comments   
Comment by Cihan Biyikoglu [ 08/Apr/14 ]
There are a number of reasons why this comes up regularly with customers.
- they see replication as a better way to create durability over local node disk persistence.
- this does allow replica reads without compromise on consistency




[MB-10206] Replication from 3.0 node to 2.x node does not uncompress documents (with compressed datatype) results in lost datatype information during failover. Created: 13/Feb/14  Updated: 28/Aug/14

Status: Reopened
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: bug-backlog
Security Level: Public

Type: Bug Priority: Major
Reporter: Venu Uppalapati Assignee: Abhinav Dangeti
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Operating System: Ubuntu 64-bit
Is this a Regression?: Yes

 Description   
Replication from 3.0 node to 2.x node does not uncompress documents with compressed datatype. This bug manifests in two user visible discrepancies.

Step to reproduce:
1) Setup 2.x node and add a 3.0 node to make a two node cluster.
2) Issue a SET for a compressed document(datatype 2) against the cluster such that the active copy goes to the 3.0 node i.e, doc key hashes to 3.0 node.
3) Issue a GET for this doc from a 2.x client. You will get the uncompressed document back.
4) Now failover the 3.0 node. Replica on 2.x will be promoted as active.
5) Issue a GET for the same doc from a 2.x client. You will get a compressed copy back. This is the first discrepancy.
6) Add back the 3.0 node and rebalance. Now 3.0 has the compressed doc but with datatype marked as raw binary(datatype 0).
7)Issue a GET for the same doc from a 2.x client. You will get a compressed copy back. This is the second discrepancy.
8)basically, during a failover/recover scenario in a mixed cluster we lose all datatype information of a document.

 Comments   
Comment by Cihan Biyikoglu [ 17/Mar/14 ]
seems like we are using a 3.0 feature during upgrade. we don't support that.
thanks
-cihan
Comment by Matt Ingenthron [ 28/May/14 ]
Agreed with Cihan's comment, but we need to define the contract here. Options:
1) Cluster does not respond to HELLO until all nodes are upgraded (&& downgrades are not supported anyway)
2) Client needs to HELLO to all nodes of the cluster and not use datatype JSON & Compression features unless all nodes report supporting it (&& downgrades are not supported anyway)
Comment by Cihan Biyikoglu [ 29/May/14 ]
sending this back to Abhinav after the discussion we had. If option #1 is cheap to implement we should enable that.
Comment by Matt Ingenthron [ 16/Jun/14 ]
Since it would be the client's responsibility to ensure that it either HELLOs or does not HELLO at all nodes, it would be the more expensive path you refer to.

Scenario: three nodes, one supports HELLO/datatype and two do not.
Expected behavior: Client issues a HELLO to all three. Finding that only one of the three supports HELLO, the client must drop the connections and re-establish connections not performing a HELLO operation.

The 'overhead' is small, but non-trivial and would need to be regularly done.

One other option is that all clients have this, but it's off by default and the user has to turn it on. I dislike this a lot since the best UI element is the one you don't have to use. It should just work out of the box in my opinion.
Comment by Chiyoung Seo [ 24/Jun/14 ]
Moving this to post 3.0 as the datatype is not supported in 3.0




[MB-11373] "Error: internal (memcached_error)" seen when raw document is sent with compressed datatype. Created: 10/Jun/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: bug-backlog
Security Level: Public

Type: Bug Priority: Critical
Reporter: Venu Uppalapati Assignee: Abhinav Dangeti
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: No

 Description   
Steps to reproduce:
1)Using a 3.x client send a binary raw document with the datatype set as "raw compressed" to the server.
2)From UI, go to Documents page /index.html#sec=documents&viewsBucket=default&documentsPageNumber=0
3)The error, "Error: internal (memcached_error)" is seen.
4)From a 2.x client issue a GET for this document, the error
2014-06-10 11:14:56.785 INFO com.couchbase.client.CouchbaseConnection: Reconnection due to exception handling a memcached operation on {QA sa=, #Rops=1, #Wops=0, #iq=0, topRop=Cmd: 0 Opaque: 1 Key: fooaaa, topWop=null, toWrite=0, interested=1}. This may be due to an authentication failure.
OperationException: SERVER: Internal error
at net.spy.memcached.protocol.BaseOperationImpl.handleError(BaseOperationImpl.java:166)

is seen.
5)This is due to server trying to uncompress a raw document.

 Comments   
Comment by Abhinav Dangeti [ 17/Jun/14 ]
Since this was before Datatype support was disabled in 3.0, just confirming: the client did do the HELLO exchange in your scenario right?
Comment by Abhinav Dangeti [ 17/Jun/14 ]
Point (5) is correct, uncompression of an uncompressed document is failing.
On the documents page, the internal_memcached_error that shows up is very likely a result of the absence of the HELLO exchange again (this time by the view client), as it tries to uncompress and non-compressed document.
Comment by Abhinav Dangeti [ 18/Jun/14 ]
On another note,if the client isn't datatype compliant, memcached will try to uncompress documents through snappy. So how about if snappy_uncompress fails we send the document as is?
Comment by Trond Norbye [ 18/Jun/14 ]
I may be reading this wrong but it sounds to me that you're sending just raw data, but you're telling the system that it is compressed... I wouldn't expect that to work..
Comment by Abhinav Dangeti [ 18/Jun/14 ]
I guess that is the scenario that Venu is attempting with a HELLO compliant client though (as a negative test),
Comment by Trond Norbye [ 18/Jun/14 ]
For the test to be "valid" he would have to have a new compatible client and:

1) send hello and have the server accept the use of datatype
2) compress the data with snappy and store it as compressed with the bit set
3) connect with another client, don't do hello
4) request the data.

It's like copying a pdf-version of /etc/passwd in as /etc/passwd on your server and expect it to work..
Comment by Chiyoung Seo [ 24/Jun/14 ]
Moving this to post 3.0 as the datatype support is not in 3.0




[MB-11428] JSON versions and encodings supported by Couchbase Server need to be defined Created: 16/Jun/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1, 3.0
Fix Version/s: bug-backlog
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Abhinav Dangeti
Resolution: Unresolved Votes: 0
Labels: documentation
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
While JSON is a standard, there are multiple unicode encodings and the definition of how to interact with this encoding has changed over the course of time. Also, our dependencies (mochiweb, view engine's JSON) may not actually conform to these standards.

Couchbase Server needs to define and document what it supports with respect to JSON.

See:
http://tools.ietf.org/html/draft-ietf-json-rfc4627bis-10 and
http://tools.ietf.org/html/rfc4627


 Comments   
Comment by Cihan Biyikoglu [ 16/Jun/14 ]
making this a documentation item - we should make this public.
Comment by Chiyoung Seo [ 24/Jun/14 ]
Moving this to post 3.0 as the datatype support is not supported in 3.0




[MB-10711] If in case of legacy client, check if document is of JSON type before setting datatype Created: 01/Apr/14  Updated: 28/Aug/14

Status: Reopened
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: bug-backlog
Security Level: Public

Type: Task Priority: Critical
Reporter: Abhinav Dangeti Assignee: Abhinav Dangeti
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Abhinav Dangeti [ 01/Apr/14 ]
memcached: http://review.couchbase.org/#/c/35169/
ep-engine: http://review.couchbase.org/#/c/35170/
Comment by Abhinav Dangeti [ 09/Apr/14 ]
Merged.
Comment by Matt Ingenthron [ 16/Jun/14 ]
I had reason to look at this earlier and I noticed that while the C code imported supports UTF-16 and UTF-32, we've used a function that just checks UTF-8. Is this an issue?
Comment by Volker Mische [ 16/Jun/14 ]
I think it depends on whether we officially support JSON that isn't UTF-8 encoded. I strongly lean towards that we only support UTF-8 at the backend (I think currently the view engine also has some UTF-8 only checks IIRC). The newest JSON RFC (RFC 7159 [1]), which added a lot of information about interoperability also states that UTF-8 has the broadest support when it comes to JSON parsers.

[1]: http://rfc7159.net/
Comment by Chiyoung Seo [ 24/Jun/14 ]
Moving this to post 3.0 as the datatype is not supported in 3.0
Comment by Volker Mische [ 24/Jun/14 ]
I'm not quite following. This is already implemented and used by the view engine. I thought that only the compression is out, but the datatype in general is in. If it is not in I'd like to know why :)
Comment by Abhinav Dangeti [ 24/Jun/14 ]
Datatype will still be stored and propagated as per original plan, but memcached's HELLO command has been disabled, therefore, a user/client will not be able to set any datatype (other than 0x00) as pre-3.0.
This was the last I heard from management. We will however still continue to do a JSON check on the documents before setting the datatype in 3.0.




[MB-12086] View Query timing out when rebalance out running in parallel Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server, view-engine
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Meenakshi Goel Assignee: Sarath Lakshman
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1199-rel, Debian 7.0

Attachments: File ns_logs.tar     File query.pcap     Text File Run.log    
Triage: Triaged
Operating System: Centos 64-bit
Is this a Regression?: Yes

 Description   
Jenkins Ref Link:
http://qa.hq.northscale.net/job/centos_x64--44_04--observe-P0/137/console
http://qa.hq.northscale.net/job/debian_x64--90_04--observe_tests-P0/5/console

Test to Reproduce:
./testrunner -i test.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False -t observe.observetest.ObserveTests.test_observe_basic_data_load_delete,items=100,nodes_in=1,rebalance=out
./testrunner -i tests.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False -t observe.observetest.ObserveTests.test_observe_basic_data_load_delete,items=100,nodes_in=1,mutate_by=multi_set,rebalance=out,GROUP=P1

Steps to Reproduce:
1. Create bucket
2. Rebalance-in 2 nodes
3. Load data 100 items
4. Wait for item to get persist on disk
5. Run re-balance out and Observe in parallel
6. Create View and Run query on view with stale=false

Uploading Logs



 Comments   
Comment by Meenakshi Goel [ 28/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12086/b2519932/10.1.3.74-8282014-131-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12086/748f9bed/10.1.3.75-8282014-132-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12086/66df8790/10.1.3.80-8282014-133-diag.zip

These logs are with RC1 build, I am uploading logs with latest build 3.0.0-1199-rel too
Comment by Meenakshi Goel [ 28/Aug/14 ]
With 3.0.0-1199-rel
http://qa.hq.northscale.net/job/centos_x64--44_04--observe-P0/142/console
https://s3.amazonaws.com/bugdb/jira/MB-12086/8d54b309/10.3.5.163-8282014-316-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12086/95a40c18/10.5.2.94-8282014-317-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-12086/2382e371/10.5.3.20-8282014-318-diag.zip
Comment by Sriram Melkote [ 28/Aug/14 ]
I've attached a pcap. At packet #9716, there's a stale=false query made.

But that returns only after 1m4s with node_vbuckets_dict_failed / not_present error. Apparently, testrunner doesn't expect it to take that long.

Given this worked in the past, I think the question open is why it's taking much longer to report this.
Comment by Sarath Lakshman [ 28/Aug/14 ]
When a query hits a rebalanced out node, we are getting this error "{"error":"node_vbuckets_dict_failed","reason":"not_present"}", which is fine. The client can retry and hit a node which is part of the cluster.

Is it normal for ns_server to take 1m for vbucket_map_mirror lookup and return node_vbuckets_dict_failed error ?
Comment by Aleksey Kondratenko [ 28/Aug/14 ]
It is not expected for vbucket_map_mirror to take 1 minute. I cannot rule it out completely but it's very unlikely.

What likely happens is that we retry request multiple times internally.
Comment by Cihan Biyikoglu [ 28/Aug/14 ]
let research - we'll consider for RC2 once we know the resolution & impact.




[MB-12088] Memcached should return an uninitiated error code Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Server
Component/s: memcached
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Major
Reporter: Patrick Varley Assignee: Trond Norbye
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
This is a blow out from MB-11875 and CBSE-1370.

This is no way that CCCP can tell if a node is uninitiated versus auth error.

Currently LCB does not retry on auth error.

This can be painful when a user is doing maintenance as it mean they have to keep updating the bootstrap list when they have removed a node from the cluster which they plan to add back in.

If memcached returned an uninitiated error code LCB would know to try the other nodes in the bootstrap list.

 Comments   
Comment by Patrick Varley [ 28/Aug/14 ]
What I mean by "uninitiated" is the node has been removed from the cluster and is at the setup wizard.




[MB-10012] cbrecovery hangs in the case of multi-instance case Created: 24/Jan/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: test-execution
Affects Version/s: 2.5.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Venu Uppalapati Assignee: Ashvinder Singh
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive cbrecovery1.zip     Zip Archive cbrecovery2.zip     Zip Archive cbrecovery3.zip     Zip Archive cbrecovery4.zip     Zip Archive cbrecovery_source1.zip     Zip Archive cbrecovery_source2.zip     Zip Archive cbrecovery_source3.zip     Zip Archive cbrecovery_source4.zip     PNG File recovery.png    
Issue Links:
Relates to
Triage: Triaged
Operating System: Centos 64-bit

 Description   
2.5.0-1055

during verification MB-9967 I performed the same steps:
source cluster: 3 modes, 4 buckets
destination cluster: 3 nodes, 1 bucket
failover 2 nodes on destination cluster(without rebalance)

cbrecovery hangs on

[root@centos-64-x64 ~]# /opt/couchbase/bin/cbrecovery http://172.23.105.158:8091 http://172.23.105.159:8091 -u Administrator -U Administrator -p password -P password -b RevAB -B RevAB -v
Missing vbuckets to be recovered:[{"node": "ns_1@172.23.105.159", "vbuckets": [513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023]}]
2014-01-22 01:27:59,304: mt cbrecovery...
2014-01-22 01:27:59,304: mt source : http://172.23.105.158:8091
2014-01-22 01:27:59,305: mt sink : http://172.23.105.159:8091
2014-01-22 01:27:59,305: mt opts : {'username': '<xxx>', 'username_destination': 'Administrator', 'verbose': 1, 'dry_run': False, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'data_only': 1.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, 'cbb_max_mb': 100000.0, 'try_xwm': 1.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'batch_max_size': 1000.0, 'report': 5.0, 'design_doc_only': 0.0, 'recv_min_bytes': 4096.0}, 'bucket_destination': 'RevAB', 'vbucket_list': '{"172.23.105.159": [513]}', 'threads': 4, 'password_destination': 'password', 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': 'RevAB'}
2014-01-22 01:27:59,491: mt bucket: RevAB
2014-01-22 01:27:59,558: w0 source : http://172.23.105.158:8091(RevAB@172.23.105.156:8091)
2014-01-22 01:27:59,559: w0 sink : http://172.23.105.159:8091(RevAB@172.23.105.156:8091)
2014-01-22 01:27:59,559: w0 : total | last | per sec
2014-01-22 01:27:59,559: w0 batch : 1 | 1 | 15.7
2014-01-22 01:27:59,559: w0 byte : 0 | 0 | 0.0
2014-01-22 01:27:59,559: w0 msg : 0 | 0 | 0.0
2014-01-22 01:27:59,697: s1 warning: received NOT_MY_VBUCKET; perhaps the cluster is/was rebalancing; vbucket_id: 513, key: RAB_111001636418, spec: http://172.23.105.159:8091, host:port: 172.23.105.159:11210
2014-01-22 01:27:59,719: s1 warning: received NOT_MY_VBUCKET; perhaps the cluster is/was rebalancing; vbucket_id: 513, key: RAB_111001636418, spec: http://172.23.105.159:8091, host:port: 172.23.105.159:11210
2014-01-22 01:27:59,724: w2 source : http://172.23.105.158:8091(RevAB@172.23.105.158:8091)
2014-01-22 01:27:59,724: w2 sink : http://172.23.105.159:8091(RevAB@172.23.105.158:8091)
2014-01-22 01:27:59,727: w2 : total | last | per sec
2014-01-22 01:27:59,728: w2 batch : 1 | 1 | 64.0
2014-01-22 01:27:59,728: w2 byte : 0 | 0 | 0.0
2014-01-22 01:27:59,728: w2 msg : 0 | 0 | 0.0
2014-01-22 01:27:59,738: s1 warning: received NOT_MY_VBUCKET; perhaps the cluster is/was rebalancing; vbucket_id: 513, key: RAB_111001636418, spec: http://172.23.105.159:8091, host:port: 172.23.105.159:11210
2014-01-22 01:27:59,757: s1 warning: received NOT_MY_VBUCKET; perhaps the cluster is/was rebalancing; vbucket_id: 513, key: RAB_111001636418, spec: http://172.23.105.159:8091, host:port: 172.23.105.159:11210



 Comments   
Comment by Anil Kumar [ 04/Jun/14 ]
Triage - June 04 2014 Bin, Ashivinder, Venu, Tony
Comment by Cihan Biyikoglu [ 27/Aug/14 ]
does this need to be considered for 3.0 or is this a test issue only?




[MB-9632] diag / master events captured in log file Created: 22/Nov/13  Updated: 27/Aug/14

Status: Reopened
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 2.2.0, 2.5.0
Fix Version/s: techdebt-backlog
Security Level: Public

Type: Task Priority: Blocker
Reporter: Steve Yen Assignee: Ravi Mayuram
Resolution: Unresolved Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The information available in the diag / master events REST stream should be captured in a log (ALE?) file and hence available to cbcollect-info's and later analysis tools.

 Comments   
Comment by Aleksey Kondratenko [ 22/Nov/13 ]
It is already available in collectinfo
Comment by Dustin Sallings (Inactive) [ 26/Nov/13 ]
If it's only available in collectinfo, then it's not available at all. We lose most of the useful information if we don't run an http client to capture it continually throughout the entire course of a test.
Comment by Aleksey Kondratenko [ 26/Nov/13 ]
Feel free to submit a patch with exact behavior you need
Comment by Cihan Biyikoglu [ 27/Aug/14 ]
is this still relevant?




[MB-12010] XDCR@next release - Parts : XMEM Nozzle Created: 19/Aug/14  Updated: 27/Aug/14

Status: In Progress
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: techdebt-backlog
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Xiaomei Zhang Assignee: Xiaomei Zhang
Resolution: Unresolved Votes: 0
Labels: sprint1_xdcr
Remaining Estimate: 40h
Time Spent: Not Specified
Original Estimate: 40h

Epic Link: XDCR next release




[MB-7761] Move operations stats out of memcached and into the engines Created: 15/Feb/13  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 1.8.1, 2.0, 2.5.1, 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Improvement Priority: Major
Reporter: Mike Wiederhold Assignee: Mike Wiederhold
Resolution: Unresolved Votes: 0
Labels: supportability
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by MB-11986 Stats for every operations. (prepend ... Resolved
Relates to
relates to MB-8793 Prepare spec on stats updates Open
Sub-Tasks:
Key
Summary
Type
Status
Assignee
MB-5011 gat (get and touch) operation not rep... Technical task Open Mike Wiederhold  
MB-6121 More operation stats please Technical task Open Mike Wiederhold  
MB-7419 Disk reads for append/prepend/incr/de... Technical task Open Mike Wiederhold  
MB-7711 UI: Getandlock doesn't show up in any... Technical task Closed Mike Wiederhold  
MB-7807 aggregate all kinds of ops in ops/sec... Technical task Open Anil Kumar  
MB-8183 getAndTouch (and touch) operations ar... Technical task Resolved Aleksey Kondratenko  
MB-10377 getl and cas not reported in the GUI ... Technical task Open Aleksey Kondratenko  
MB-11655 Stats: Getandlock doesn't show up in ... Technical task Open Mike Wiederhold  

 Description   
Stats have increasingly been an issue to deal with since they are half done in memcached and half done in ep-engine. Memcached should simply handle connections and not really care or track anything operation related. This stuff should happen in the engines and memcached should just ask for it when it needs the info.

 Comments   
Comment by Tug Grall (Inactive) [ 01/May/13 ]
Just to be sure they are linked. I let the engr team chose how to deal with this JIRA
Comment by Perry Krug [ 07/Jul/14 ]
Raising awareness to this broad supportability issue which sometimes makes it hard for the field and customers to accurately understand their Couchbase traffic




[MB-12001] Stats Issue - Number of documents is misleading during rebalance with delta recovery Created: 18/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Pavel Paulau Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Build 3.0.0-1169

Platform = Physical
OS = CentOS 6.5
CPU = Intel Xeon E5-2630 (24 vCPU)
Memory = 64 GB
Disk = RAID 10 HDD

Attachments: PNG File curr_items.png    
Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: http://ci.sc.couchbase.com/job/ares-dev/45/artifact/
Is this a Regression?: No

 Description   
1 of 4 nodes is being re-added after failover.
500M x 2KB items, 10K mixed ops/sec.

Steps:
1. Failover one of nodes.
2. Add it back.
3. Enabled delta recovery.
4. Sleep 20 minutes.
5. Rebalance cluster.

When rebalance starts reported curr_items is significantly greater than actual number of documents.

 Comments   
Comment by Anil Kumar [ 19/Aug/14 ]
Stats Issue for new 3.0 feature 'delta node recovery'




[MB-11984] Intra-cluster replication slows down during intensive ejection Created: 18/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Pavel Paulau Assignee: Sundar Sridharan
Resolution: Unresolved Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Build 3.0.0-1166

Platform = Physical
OS = CentOS 6.5
CPU = Intel Xeon E5-2680 v2 (40 vCPU)
Memory = 256 GB
Disk = RAID 10 SSD

Attachments: PNG File ejections_and_replication_queue.png    
Issue Links:
Relates to
relates to MB-11642 Intra-replication falling far behind ... Reopened
Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/bugdb/jira/MB-11984/172.23.100.17.zip
https://s3.amazonaws.com/bugdb/jira/MB-11984/172.23.100.18.zip
https://s3.amazonaws.com/bugdb/jira/MB-11984/172.23.100.19.zip
https://s3.amazonaws.com/bugdb/jira/MB-11984/172.23.100.20.zip
https://s3.amazonaws.com/bugdb/jira/MB-11984/172.23.100.21.zip
https://s3.amazonaws.com/bugdb/jira/MB-11984/172.23.100.22.zip
https://s3.amazonaws.com/bugdb/jira/MB-11984/172.23.100.23.zip
https://s3.amazonaws.com/bugdb/jira/MB-11984/172.23.100.24.zip
https://s3.amazonaws.com/bugdb/jira/MB-11984/172.23.100.25.zip
Is this a Regression?: Yes

 Description   
Following up MB-11642.

I noticed that higher replication queue (10-100K items) is caused by intensive ejection (by intensive I mean 1-2M items ejected per second).

My setup:
9 nodes, 1 bucket, 2 replicas
50K inserts/sec (~1KB docs)

 Comments   
Comment by Raju Suravarjjala [ 18/Aug/14 ]
Triage: Moving to 3.0.1 as this is due to millions of documents
Comment by Sundar Sridharan [ 19/Aug/14 ]
Still trying to reproduce, does not occur with lower set ops/sec




[MB-11999] Resident ratio of active items drops from 3% to 0.06% during rebalance with delta recovery Created: 18/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Pavel Paulau Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: performance, releasenote
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Build 3.0.0-1169

Platform = Physical
OS = CentOS 6.5
CPU = Intel Xeon E5-2630 (24 vCPU)
Memory = 64 GB
Disk = RAID 10 HDD

Attachments: PNG File vb_active_resident_items_ratio.png     PNG File vb_replica_resident_items_ratio.png    
Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: http://ci.sc.couchbase.com/job/ares-dev/45/artifact/
Is this a Regression?: No

 Description   
1 of 4 nodes is being re-added after failover.
500M x 2KB items, 10K mixed ops/sec.

Steps:
1. Failover one of nodes.
2. Add it back.
3. Enabled delta recovery.
4. Sleep 20 minutes.
5. Rebalance cluster.

Most importantly it happens due to excessive memory usage.




[MB-11998] Working set is screwed up during rebalance with delta recovery (>95% cache miss rate) Created: 18/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Pavel Paulau Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Build 3.0.0-1169

Platform = Physical
OS = CentOS 6.5
CPU = Intel Xeon E5-2630 (24 vCPU)
Memory = 64 GB
Disk = RAID 10 HDD

Attachments: PNG File cache_miss_rate.png    
Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: http://ci.sc.couchbase.com/job/ares-dev/45/artifact/
Is this a Regression?: No

 Description   
1 of 4 nodes is being re-added after failover.
500M x 2KB items, 10K mixed ops/sec.

Steps:
1. Failover one of nodes.
2. Add it back.
3. Enabled delta recovery.
4. Sleep 20 minutes.
5. Rebalance cluster.




[MB-12038] build go programs in ns_server instead of shipping pre-built binaries (was: [windows] F-Secure flagging binary) Created: 21/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 2.5.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Sriram Melkote Assignee: Wayne Siu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
A popular virus scanner, F-Secure is deleting a file installed by us, generate_cert.exe.

We need to analyze why this is happening.

See: https://groups.google.com/forum/#!topic/couchbase/E3QvNolCknQ


 Comments   
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
What you want me to do? Those crappy antiviruses are known to have false positives occasionally.
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
And btw our build machines _are_ clean.
Comment by Sriram Melkote [ 27/Aug/14 ]
You could compile it latest go on Windows, and the resulting binary does not trip any virus scanners
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
I don't compile anything on windows. This binaries are built on GNU/Linux machine using go cross compiler. I can rebuild using more recent go but it's unclear if it's really going to help.
Comment by Aleksey Kondratenko [ 27/Aug/14 ]
And surely if build folks are ready to build our go stuff in ns_server I'm ok if some fan of cmake can make it happen.
Comment by Sriram Melkote [ 27/Aug/14 ]
Yes, let's request build team to add ability to compile Go code. We'll need it soon enough, independent of this bug anyway.




[MB-12084] Create 3.0.0 chef-based rightscale template for EE and CE Created: 27/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: cloud
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Major
Reporter: Anil Kumar Assignee: Wei-Li Liu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Need this before 3.0 GA




[MB-12082] Marketplace AMI - Enterprise Edition and Community Edition - provide AMI id to PM Created: 27/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: cloud
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Major
Reporter: Anil Kumar Assignee: Wei-Li Liu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Need AMI's before 3.0.0 GA




[MB-12083] Create 3.0.0 legacy rightscale templates for Enterprise and Community Edition (non-chef) Created: 27/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: cloud
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Major
Reporter: Anil Kumar Assignee: Wei-Li Liu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We need this before 3.0 GA




[MB-11589] Sliding endseqno during initial index build or upr reading from disk snapshot results in longer stale=false query latency and index startup time Created: 28/Jun/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0-Beta
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Sarath Lakshman Assignee: Sriram Melkote
Resolution: Unresolved Votes: 0
Labels: performance, releasenote
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
blocks MB-11920 DCP based rebalance with views doesn'... Closed
Relates to
relates to MB-11919 3-5x increase in index size during re... Open
relates to MB-12081 Remove counting mutations introduced ... Resolved
relates to MB-11918 Latency of stale=update_after queries... Closed
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
We have to fix this depending on the development cycles we have left for 3.0

 Comments   
Comment by Anil Kumar [ 17/Jul/14 ]
Triage - July 17

Currently investigating we will decide depending on the scope of changes needed.
Comment by Anil Kumar [ 30/Jul/14 ]
Triage : Anil, Wayne .. July 29th

Raising this issue to "Critical" this needs to be fixed by RC.
Comment by Sriram Melkote [ 31/Jul/14 ]
The issue is that we'll have to change the view dcp client to stream all 1024 vbuckets in parallel, or we'll need an enhancement in ep-engine to stop streaming at the point requested. Neither is a simple change - the reason it's in 3.0 is because Dipti had requested we try to optimize query performance. I'll leave it at Major as I don't want to commit to fixing this in RC and also, the product works with reasonable performance without this fix and so it's not a must have for RC.
Comment by Sriram Melkote [ 31/Jul/14 ]
Mike noted that even streaming all vbuckets in parallel (which was perhaps possible to do in 3.0) won't directly solve the issue as the backfills are scheduled one at a time. ep-engine could hold onto smaller snapshots but that's not something we can consider in 3.0 - so net effect is that we'll have to revisit this in 3.0.1 to design a proper solution.
Comment by Sriram Melkote [ 12/Aug/14 ]
Bringing back to 3.0 as this is the root cause of MB-11920 and MB-11918
Comment by Anil Kumar [ 13/Aug/14 ]
Deferring this to 3.0.1 since making this out of scope for 3.0.




[MB-11470] fdb_seek can't seek backwards Created: 18/Jun/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: forestdb
Affects Version/s: 2.5.1
Fix Version/s: bug-backlog
Security Level: Public

Type: Improvement Priority: Minor
Reporter: Jens Alfke Assignee: Sundar Sridharan
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
If fdb_seek is called with a key that collates before its next key, it appears to have no effect. In other words, it can't seek backwards.

Couchbase Lite does need to do random-access seeks when a view query is given an array of keys. The keys can be given in any order, and the results need to be returned in the same order.

Currently I'm working around this by detecting when the seek would be backwards, and instead closing the iterator and opening a new one. I don't know how much more expensive this is; if it would be a lot faster to allow the iterator to seek backwards, then this might be worth implementing.




[MB-11071] Support multiple KV instances in a single database file Created: 08/May/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: forestdb
Affects Version/s: feature-backlog
Fix Version/s: feature-backlog
Security Level: Public

Type: Improvement Priority: Major
Reporter: Chiyoung Seo Assignee: Jung-Sang Ahn
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
In some use cases, applications need to create lots of KV instances and read/write items from/to those instances. If we create a separate database file for each KV instance, there will be lots of files to be opened, which causes too many file descriptors to be used, or we should open / close each file repeatedly, which incurs lots of additional overhead (e.g.,open / close system call, reading a database header, reading btree nodes again, etc.).

To address this issue, we need to support multiple KV instances in a single database file. This will allow us to manage our own buffer cache more efficiently and increase a write batch size significantly, which improve the write throughput a lot.

In addition, this will help us improve the ep-engine flusher write throughput when we switch from Couchstore to ForestDB, because we can put mulitple VBuckets in a single ForestDB file (i.e., a single shard file).





[MB-11620] Database encryption Created: 02/Jul/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: forestdb
Affects Version/s: 2.5.1
Fix Version/s: feature-backlog
Security Level: Public

Type: Improvement Priority: Major
Reporter: Jens Alfke Assignee: Sundar Sridharan
Resolution: Unresolved Votes: 0
Labels: security
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
A number of current/potential Couchbase Lite developers want to be able to encrypt their databases. (This is a requirement in some areas like healthcare, and strongly desired by financial institutions.) We'd like to have a way to do this with ForestDB.

The desired behavior is that a symmetric key would be provided when opening the database, and all file I/O would be encrypted using that key (transparently, without affecting access via the public API.)




[MB-11187] V8 crashes on memory allocation errors, closes erlang on some indexing loads Created: 22/May/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.2.0, 2.5.1
Fix Version/s: 2.5.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Brent Woodruff Assignee: Wayne Siu
Resolution: Unresolved Votes: 0
Labels: hotfix
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
On some indexing workloads, V8 can experience issues allocating memory. This, in turn, will cause Erlang to close resulting in the node becoming pending while the babysitter restarts the main Erlang VM.

This can occur even when there is sufficient memory available on the node. The node does not need to experience out-of-memory for this to happen.

To diagnose if this is occurring, it's possible to check logs for a few messages.

babysitter log:

====
[ns_server:info,2014-05-16T20:46:44.417,babysitter_of_ns_1@127.0.0.1:<0.71.0>:ns_port_server:log:168]ns_server<0.71.0>:
ns_server<0.71.0>: #
ns_server<0.71.0>: # Fatal error in CALL_AND_RETRY_2
ns_server<0.71.0>: # Allocation failed - process out of memory
ns_server<0.71.0>: #
ns_server<0.71.0>:

[ns_server:info,2014-05-16T20:46:44.744,babysitter_of_ns_1@127.0.0.1:<0.71.0>:ns_port_server:log:168]ns_server<0.71.0>: /opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/memsup: Erlang has closed.Erlang has closed
ns_server<0.71.0>:

[ns_server:info,2014-05-16T20:46:44.745,babysitter_of_ns_1@127.0.0.1:<0.70.0>:supervisor_cushion:handle_info:58]Cushion managed supervisor for ns_server failed: {abnormal,134}
[error_logger:error,2014-05-16T20:46:44.745,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generic server <0.71.0> terminating
** Last message in was {#Port<0.2943>,{exit_status,134}}
** When Server state == {state,#Port<0.2943>,ns_server,
                               {[[],
                                 "/opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/memsup: Erlang has closed.Erlang has closed ",
                                 [],"#",
                                 "# Allocation failed - process out of memory",
                                 "# Fatal error in CALL_AND_RETRY_2","#",[],
                                 "working as port","working as port",
                                 "Apache CouchDB has started. Time to relax.",
                                 "Apache CouchDB 1.2.0a-01dda76-git (LogLevel=info) is starting.",
                                 empty],
                                [empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty]},
                               {ok,{1400273204926,#Ref<0.0.0.18660>}},
                               [[],
                                "/opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/memsup: Erlang has closed.Erlang has closed "],
                               0,true}
** Reason for termination ==
** {abnormal,134}
====

One can check for the latest occurrence of this error (or for all occurrences by removing the tail -1) with this command:

$ /opt/couchbase/bin/cbbrowse_logs babysitter | awk '/\]ns_server<.*>: $/,/# Allocation failed - process out of memory/' | grep -E -o '[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]*:[0-9]{2}:[0-9]{2}'
2014-05-16T20:46:44


 Comments   
Comment by Sarath Lakshman [ 23/May/14 ]
Backported changes for v8 version upgrade.

Following changes are under review:
http://review.couchbase.org/#/c/37506/
http://review.couchbase.org/#/c/37507/
Comment by Sarath Lakshman [ 23/May/14 ]
Merged back ported changes
Comment by Sarath Lakshman [ 23/May/14 ]
I have sent a mail to build team for providing the build with the following changes:

couchdb commit (https://github.com/couchbase/couchdb/commit/505c2278b34eb7a47843d5017d101e98aa856d6a)

v8 version change (http://review.couchbase.org/#/c/32781/2/override-3.0.0.xml)
Comment by Sriram Melkote [ 29/May/14 ]
Wayne - it's good to have some QE coverage for this as it's shipping to a production customer. Can Ketaki or Meenakshi please validate and also do some additional sanity testing on this build? Thanks!
Comment by Sarath Lakshman [ 29/May/14 ]
Brent, Could you help us verify this patch using the reproducible setup you have ?
Comment by Sarath Lakshman [ 29/May/14 ]
No. We should wait for QE team to certify this fix before rollout to the customer.
Comment by Wayne Siu [ 29/May/14 ]
Brent,
We still need to run the regression tests on the hotfix. I'll update the ticket with an ETA on Monday.
In a mean time, if the customer could help verify the hotfix, it will be ok with the expectation that regression tests are still in progress.
Comment by Wayne Siu [ 09/Jun/14 ]
Brent,
We can provide the ubuntu 10.4 package. We'll run a quick sanity on the binary, and update the ticket here later. Will shoot for later today.
Comment by Wayne Siu [ 12/Jun/14 ]
The 10.4 package passed the sanity tests.
Comment by Wayne Siu [ 20/Jun/14 ]
Brent,
Please let us know if we could close this ticket.
Comment by Brent Woodruff [ 20/Jun/14 ]
I believe it would be ok to close this MB. The backporting work has been completed, the builds required have been made and tested, and the updated files were provided.
Comment by Wayne Siu [ 31/Jul/14 ]
Brent,
Let us know if there is any open item at this time.
Comment by Brent Woodruff [ 01/Aug/14 ]
I don't have any outstanding items regarding this ticket, thanks.
Comment by Volker Mische [ 27/Aug/14 ]
Wayne, I think this one is ready to be closed.




Allow for dynamic change of the number of connections (MB-11066)

[MB-11924] Add 'max_connections' setting to UI / REST. Created: 11/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Technical task Priority: Major
Reporter: Dave Rigby Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We want to be able to change the maximum number of connections in memcached dynamically. We have MB-11066 (parent of this) to add the support in memcached to allow this, but we also need an interface for the user to do this.

I'll leave it up to ns_server to determine the best way to show this in the UI (but one suggestion is to add it to the "internalSettings" window).

For the ns_server -> memcached interface, we have added a new pair of binary protocol messages - IOCTL_GET / IOCTL_SET (see http://review.couchbase.org/#/c/39608/ ).

For the maximum connections setting, the proposal for IOCTL_GET / IOCTL_SET is:

key: max_conns_on_port_XXXX

    XXXX is the port number to change, e.g. "max_conns_on_port_11210"

IOCTL_GET: returns a nil-terminated string (in the response body) specifying the current connection limit.

IOCT_SET: value (body) is a nul-terminated string specify the new connection limit.

    responses:

- PROTOCOL_BINARY_RESPONSE_SUCCESS - New value accepted, limit has been changed.
- PROTOCOL_BINARY_RESPONSE_KEY_ENOENT - Invalid key
- PROTOCOL_BINARY_RESPONSE_EACCESS - User is not authenticated as admin.
- PROTOCOL_BINARY_RESPONSE_EINVAL - Value was not in the correct format (string, convertible to non-zero, non-negative).


 Comments   
Comment by Aleksey Kondratenko [ 11/Aug/14 ]
There's very generic ticket somewhere about "settings for everything UI" we can make it part of that work. In which case expect it done in next decade :)

But we can also implement it in less pretty way. E.g. diag-eval-lable config entry or internal settings.
Comment by Dave Rigby [ 12/Aug/14 ]
I think internal settings (and I assume implicitly a REST endpoint) should be fine.

Ideally, upon changing the maximum ns_server would both send the IOCTL_SET message to all memcached instances, *and* update the memcached.json config file so the change persists across restarts.

Comment by Dave Rigby [ 27/Aug/14 ]
@Alk: The memcached changes have been merged to the 3.0.1 branch - see http://review.couchbase.org/#/c/40751/ Please let me know if you need any more info to be able to implement this.




[MB-11066] Allow for dynamic change of the number of connections Created: 07/May/14  Updated: 27/Aug/14

Status: In Progress
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.0, 2.5.1, 3.0
Fix Version/s: techdebt-backlog, 3.0.1
Security Level: Public

Type: Improvement Priority: Major
Reporter: Trond Norbye Assignee: Dave Rigby
Resolution: Unresolved Votes: 0
Labels: #memcached
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relates to
relates to MB-11064 Increase default memcached connection... Open
Sub-Tasks:
Key
Summary
Type
Status
Assignee
MB-11924 Add 'max_connections' setting to UI /... Technical task Open Aleksey Kondratenko  
MB-11949 Document which memcached.json setting... Technical task Resolved Trond Norbye  

 Description   
The number of connection should not be a preconfigured value, but something we can change "on the fly". There is no good reason why it is currently kept in a fixed array today (except the possibility to easy dump all connection objects in a debugger)

 Comments   
Comment by Dave Rigby [ 02/Jul/14 ]
http://review.couchbase.org/#/c/39018/
Comment by Wayne Siu [ 23/Jul/14 ]
Fix merged to 3.0.1.
Comment by Dave Rigby [ 24/Jul/14 ]
This isn't complete - the patch merged to 3.0.1 (ab7d44a) is only the groundwork to support this, there isn't yet code committed to actually allow the user to change the connection count.
Comment by Matt Ingenthron [ 11/Aug/14 ]
Related to this, should we better define the behavior when number of connections are exhausted? Right now IIRC, it's that the connection will be established and immediately dropped, which is hard for a client library or another part of the system to distinguish between network problems and a server that's exhausted it's connections.
Comment by Dave Rigby [ 11/Aug/14 ]
@Matt: That's a good point, but essentially orthogonal to this issue - can you create a separate MB to track that request?
Comment by Matt Ingenthron [ 11/Aug/14 ]
It's not fully orthogonal and I've filed it before, but since I can't find it I'll file it now.
Comment by Trond Norbye [ 11/Aug/14 ]
Its been a while since I checked, but I believe that we at one point stopped accepting the clients when we hit the limit and then started accepting again when we fell below a watermark (but it could be that we reverted that; I'd have to check).

BUT this is another thing that we should have in our system: the UI should give more warnings when bad things are about to happen (or happened)..
Comment by Trond Norbye [ 11/Aug/14 ]
I just looked at the file and the logic I described only happens for EMFILE...
Comment by Matt Ingenthron [ 11/Aug/14 ]
I opened MB-11926 to cover the "reliable, understandable behavior when hit max connections" side of this.
Comment by Dave Rigby [ 12/Aug/14 ]
memcached changes: http://review.couchbase.org/#/c/40531/
Comment by Dave Rigby [ 13/Aug/14 ]
As discussed with Alk and agreed with Trond, going with an alternative mechanism - will add two new binary protocol commands - CONFIG_VERIFY and CONFIG_RELOAD.

* CONFIG_VERIFY passes a proposed new config to memcached in the body, which memcached will verify and return either SUCCESS or EINVAL if the config is not valid (for example attempting to change settings which cannot be changed dynamically).

* CONFIG_RELOAD instructs memcached to reload the memcached.json file from disk. Any modified parameters which are permitted to be changed dynamically will be updated.


Comment by Dave Rigby [ 20/Aug/14 ]
memcached changes (new approach):

http://review.couchbase.org/#/c/40749 - MB-11066: Refactor config parsing code to allow parse at arbitrary time
http://review.couchbase.org/#/c/40750 - MB-11066: Unit tests for config_parse
http://review.couchbase.org/#/c/40751 - MB-11066: Add support for dynamically reloading memcached JSON config
Comment by Dave Rigby [ 27/Aug/14 ]
memcached changes merged.




[MB-12077] Add 'Update/Fetch' button to UI docs query window. Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Venu Uppalapati Assignee: Anil Kumar
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2014-08-26 at 3.56.27 PM.png    
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
refer to attached screenshot.
Steps to reproduce:
1)On Bucket->Documents screen click the Documents filter to open the range query window
2)There is a reset button to clear user input and a close button that closes the range query and fetches the docs.
3)This is un-intuitive to close the window to fetch the docs. If the user wants try a new range, the window has to be closed and opened every time.
4)There should be an Update/Fetch button that allows user to get results without closing window.




[MB-12076] Internal moxi misconfiguration Created: 22/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: moxi
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: kay Assignee: Sergey Avseyev
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos 6.5

Attachments: Text File normal.log     Text File problem.log    
Triage: Untriaged
Operating System: Centos 64-bit
Is this a Regression?: Unknown

 Description   
I have 4 servers cluster. Four buckets. One of them is default couchbase bucket with replica=1.

On one server moxi behavior is very strange. Third server's moxi lives its own live.
I've telneted to 11211 moxi's port and tried to set test keys. This key apperared only on that server, not on whole cluster. Also couchbase monitoring tool doesn't show any activity on cluster.

I've noticed that problem moxi process listens only three TCP port's:
{code}
netstat -nlpt | grep 30070
tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 30070/moxi
tcp 0 0 :::11211 :::* LISTEN 30070/moxi
tcp 0 0 :::6696 :::* LISTEN 30070/moxi
{code}

Other servers' moxies have four listen ports:
{code}
netstat -nltp | grep 2577
tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 2577/moxi
tcp 0 0 0.0.0.0:60593 0.0.0.0:* LISTEN 2577/moxi
tcp 0 0 :::11211 :::* LISTEN 2577/moxi
tcp 0 0 :::18347 :::* LISTEN 2577/moxi

netstat -nlpt | grep 23001
tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 23001/moxi
tcp 0 0 0.0.0.0:11339 0.0.0.0:* LISTEN 23001/moxi
tcp 0 0 :::11211 :::* LISTEN 23001/moxi
tcp 0 0 :::5191 :::* LISTEN 23001/moxi

netstat -nlpt | grep 31535
tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 31535/moxi
tcp 0 0 0.0.0.0:33578 0.0.0.0:* LISTEN 31535/moxi
tcp 0 0 :::11211 :::* LISTEN 31535/moxi
tcp 0 0 :::53475 :::* LISTEN 31535/moxi
{code}

So it seems that moxi on problem server was not able to listen one TCP port.

I've attached debug logs for two servers: problem and normal server.

The problem process is still running. Please let me know which logs do you need for further problem investigation.

 Comments   
Comment by kay [ 22/Aug/14 ]
I use couchbase-server-2.5.1-1083.x86_64
Comment by kay [ 22/Aug/14 ]
please change subproject to moxi for this issue




[MB-11894] Documentation does not feel smooth between pages with a scroll bar and pages without. Created: 06/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: doc-system
Affects Version/s: 3.0-Beta
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Patrick Varley Assignee: Amy Kurtzman
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Operating System: MacOSX 64-bit
Is this a Regression?: Unknown

 Description   
This might be me being nitpicking. When you flick between these two links it does not feel smooth:

http://docs.couchbase.com/prebuilt/couchbase-manual-3.0/Features/features.html
http://docs.couchbase.com/prebuilt/couchbase-manual-3.0/Features/bcp.html

Which are sequence sections on the left hand bar; "Couchbase 3.0 Features" to "Database Change Protocol"

 Comments   
Comment by Amy Kurtzman [ 26/Aug/14 ]
Yes, it might be nitpicking. :-)

Can you clarify what the problem is? I'm not really sure what you mean. Are you commenting on the physical function of the page or the textual content of the page/

The last sentence is unclear:
"Which are sequence sections on the left hand bar; "Couchbase 3.0 Features" to "Database Change Protocol"
Is that a question?

Also please keep in mind that this section will be revised for to the GA release.
Comment by Patrick Varley [ 26/Aug/14 ]
"""
The last sentence is unclear:
"Which are sequence sections on the left hand bar; "Couchbase 3.0 Features" to "Database Change Protocol"
Is that a question?
"""
It was more a indication that it might be noticed easily as the chapters are beside each other.

 It only happens on my bigger display it looks like because "features.html" page is longer and as a result Chrome puts a scroll bar in. Which then causes everything to shift over to the left. I will record it when I'm in the office tomorrow.




[MB-12054] [windows] [2.5.1] cluster hang when flush beer-sample bucket Created: 22/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Thuan Nguyen Assignee: Raju Suravarjjala
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: windows server 2008 R2

Attachments: Zip Archive 172.23.107.124-8222014-1546-diag.zip     Zip Archive 172.23.107.125-8222014-1547-diag.zip     Zip Archive 172.23.107.126-8222014-1548-diag.zip     Zip Archive 172.23.107.127-8222014-1549-diag.zip    
Triage: Triaged
Operating System: Windows 64-bit
Is this a Regression?: Unknown

 Description   
Install couchbase server 2.5.1 on 4 nodes windows server 2008 R2 64-bit
Create a cluster of 4 nodes
Create beer-sample bucket
Enable flush in bucket setting.
Flush beer-sample bucket. Cluster became hang.




[MB-12056] double-click on "Create new Data bucket" produces "Unexpected server error, request logged." after first bucket creation Created: 24/Aug/14  Updated: 26/Aug/14

Status: Reopened
Project: Couchbase Server
Component/s: ns_server, UI
Affects Version/s: 3.0
Fix Version/s: 3.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Andrei Baranouski Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1174

Attachments: PNG File create_bucket.png    
Triage: Triaged
Is this a Regression?: Unknown

 Description   
[ns_server:error,2014-08-24T10:47:53.281,ns_1@127.0.0.1:<0.437.0>:menelaus_web:loop:170]Server error during processing: ["web request failed",
                                 {path,"/pools/default/buckets"},
                                 {type,error},
                                 {what,badarith},
                                 {trace,
                                  [{menelaus_web_buckets,interpret_ram_quota,
                                    4,
                                    [{file,"src/menelaus_web_buckets.erl"},
                                     {line,979}]},
                                   {menelaus_web_buckets,
                                    parse_bucket_params_without_warnings,3,
                                    [{file,"src/menelaus_web_buckets.erl"},
                                     {line,696}]},
                                   {menelaus_web_buckets,parse_bucket_params,
                                    2,
                                    [{file,"src/menelaus_web_buckets.erl"},
                                     {line,665}]},
                                   {menelaus_web_buckets,do_bucket_create,3,
                                    [{file,"src/menelaus_web_buckets.erl"},
                                     {line,533}]},
                                   {menelaus_web_buckets,
                                    handle_bucket_create,2,
                                    [{file,"src/menelaus_web_buckets.erl"},
                                     {line,564}]},
                                   {request_throttler,do_request,3,
                                    [{file,"src/request_throttler.erl"},
                                     {line,59}]},
                                   {menelaus_web,loop,2,
                                    [{file,"src/menelaus_web.erl"},
                                     {line,149}]},
                                   {mochiweb_http,headers,5,
                                    [{file,
                                      "/buildbot/build_slave/centos-5-x64-300-builder/build/build/couchdb/src/mochiweb/mochiweb_http.erl"},
                                     {line,94}]}]}]


 Comments   
Comment by Andrei Baranouski [ 24/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12056/logs.zip
Comment by Aleksey Kondratenko [ 25/Aug/14 ]
Need cbcollectinfos and not just logs
Comment by Andrei Baranouski [ 25/Aug/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-12056/b37d4250/10.3.4.145-8252014-133-diag.zip
Comment by Aleksey Kondratenko [ 25/Aug/14 ]
http://review.couchbase.org/40889
Comment by Wayne Siu [ 26/Aug/14 ]
PM(Cihan) reviewed and agreed that the fix goes to 3.0.1.




[MB-12068] XDCR - only minority of items get replicated Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: 2.2.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Marek Obuchowicz Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Cluster "US" - 3x AWS m3.medium instances
Cluster "EU" - 1x AWS m3.medium instance
All running amzn-ami-pv-2014.03.1.x86_64-ebs
Cochbase server (latest community) installed from couchbase-server-2.2.0-837.x86_64.rpm

Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/couchbase_debug/eu.zip
https://s3.amazonaws.com/couchbase_debug/us.zip
Is this a Regression?: Unknown

 Description   
I have two clusters:
 - "US" with three nodes - there is one bucket, profiles (with 2 replicas)
 - "EU" with one node - there is one bucket, profiles_debug (without replicase)

I have setup XDCR from "US" cluster to "EU" cluster. Number of documents in "US" cluster is 2.35m (+ 4.7m replicas), but number of replicated documents to "EU" cluster is only 780k - exactly 1/3 of all documents. When I create new buckets in destination cluster and setup XDCR again, the situation doesn't change - new bucket after a while gets 1/3 of documents and no more.

In web console -> XDCR Replication on US cluster, status shows "Replicating - Last 10 errors". Last 10 errors state:
{code}
2014-08-26 09:39:14 [Vb Rep] Error replicating vbucket 765. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 649. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 593. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 567. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 552. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 521. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 502. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 454. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 439. Please see logs for details.
2014-08-26 09:39:13 [Vb Rep] Error replicating vbucket 417. Please see logs for details.
{code}

I have been using XDCR (bi-directional) in the past, when both US and EU clusters had only one node (no replicas within clusters). At that time it was working fine. Problems started, when I've added two nodes to US cluster, setup new bucket with 2 replicas and tried to use XDCR on this bucket.

List of steps we did, in chronological order:
 - create cluster US, 1 node
 - create cluster EU, 1 node
 - setup bucket A, no replicas, on both clusters
 - setup bi-directional XDCR A(US) <-> A(EU), default settings
 - add two nodes to US cluster
 - setup bucket B, 2 replicas, on cluster US
 - terminate XDCR A(US) <-> A(EU)
 - setup XDCR of bucket A(US) to bucket B(US), terminate it when all documents are copied.
 - delete bucket A(EU)
 - create bucket B(EU), 0 replicas
 - setup bi-directional XDCR B(US) <-> B(EU)
 - only 1/3 of documents from B(US) got replicated to B(EU)

Attached cbcollectInfo output from both clusters. As suggested by pfehre, created Jira issue. Please get in touch by email to get direct access to servers if it helps with investigation.

 Comments   
Comment by Aleksey Kondratenko [ 26/Aug/14 ]
I'll need logs from all nodes.




[MB-10823] Log failed/successful login with source IP to detect brute force attacks Created: 10/Apr/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: feature-backlog
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Cihan Biyikoglu Assignee: Don Pinto
Resolution: Unresolved Votes: 0
Labels: security
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Cihan Biyikoglu [ 18/Jun/14 ]
http://www.couchbase.com/issues/browse/MB-11463 for covering ports 11209 or 11211.




[MB-11346] Audit logs for User/App actions Created: 06/Jun/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 2.2.0
Fix Version/s: feature-backlog
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Anil Kumar Assignee: Don Pinto
Resolution: Unresolved Votes: 0
Labels: security, supportability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Couchbase Server should be able to get an audit logs for all User/App actions such-as login/logout events, mutations and other bucket and security changes.






[MB-9635] Audit logs for Admin actions Created: 22/Nov/13  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 2.2.0
Fix Version/s: feature-backlog
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Anil Kumar Assignee: Don Pinto
Resolution: Unresolved Votes: 0
Labels: security
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   
Couchbase Server should be able to get an audit logs for all Admin actions such-as login/logout events, significant events (rebalance, failover, etc) etc.



 Comments   
Comment by Matt Ingenthron [ 13/Mar/14 ]
Note there isn't exactly a "login/logout" event. This is mostly by design. A feature like this could be added, but there may be better ways to achieve the underlying requirement. One suggestion would be to log initial activities instead of every activity and have a 'cache' for having seen that user agent within a particular window. That would probably meet most auditing requirements and is, I think, relatively straightforward to implement.
Comment by Aleksey Kondratenko [ 06/Jun/14 ]
We have access.log implemented now. But it's not exactly same as full-blown audit. Particularly we do log that certain POST was handled in access.log, but we do not log any parameters of that action. So it doesn't count as fullly-featured audit log I think.
Comment by Aleksey Kondratenko [ 06/Jun/14 ]
access.log for log and ep-engine's access.log do not conflict due to being in necessarily different directories.
Comment by Perry Krug [ 06/Jun/14 ]
They may not conflict in terms of unique names in the same directory, but to our customers it may be a little bit too close to remember which access.log does what...
Comment by Aleksey Kondratenko [ 06/Jun/14 ]
Ok. Any specific proposals ?
Comment by Perry Krug [ 06/Jun/14 ]
Yes, as mentioned above, login.log would be one proposal but I'm not tied to it.
Comment by Aleksey Kondratenko [ 06/Jun/14 ]
access.log has very little to do with logins. It's full blown equivalent of apache's access.log.
Comment by Perry Krug [ 06/Jun/14 ]
Oh sorry, I misread this specific section.

How about audit.log? I know it's not fully "audit" but I'm just trying to avoid the name clash in our customer's minds...
Comment by Anil Kumar [ 09/Jun/14 ]
Agreed we should rename this file to audit.log to avoid any confusion. Updating the MB-10020 to make that change.
Comment by Larry Liu [ 10/Jun/14 ]
Hi, Anil

Does this feature satisfy PCI compliance?

Larry
Comment by Cihan Biyikoglu [ 11/Jun/14 ]
Hi Larry, PCI is a comprehensive set of requirements that go beyond database features. This does help with some part of PCI but talking about compliance with PCI involve many additional controls and most can be done at the operational levels or at the app level.
thanks




[MB-11758] Change "Redhat" logo to "Tux Logo" for Linux binaries on website Created: 17/Jul/14  Updated: 26/Aug/14  Due: 21/Jul/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP3
Fix Version/s: cbq-DP3
Security Level: Public

Type: Task Priority: Minor
Reporter: Mark Nunberg Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The webpage here: http://www.couchbase.com/communities/n1ql mentions the linux tarball binary with a redhat logo. This is rather confusing and would make many users assume this only contained a .rpm without actually clicking on the link. The "Tux" logo should be used instead, as that's the common convention for denoting "Generic Linux"

 Comments   
Comment by Gerald Sangudi [ 17/Jul/14 ]
Don, please work with marketing to fix this.

Thanks.
Comment by Don Pinto [ 26/Aug/14 ]
Gerald, is the a tux logo file that you can share? or is this standard linux tux?

Thanks,




[MB-8250] Doc: Need description and best practices on how to add a field/column to all documents within a bucket Created: 13/May/13  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: clients, documentation
Affects Version/s: 2.0.1
Fix Version/s: bug-backlog
Security Level: Public

Type: Bug Priority: Major
Reporter: Perry Krug Assignee: Perry Krug
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Is this a Regression?: Yes

 Description   
The two strategies are:
-Update the application to "work" with both types of schemas and add the field to documents as they are accessed
-Create a view to find documents that don't have the new field, and then iterrate over every document to do it.

Want a description on what's being accomplished, why, how, etc including examples in different client libraries.

 Comments   
Comment by kzeller [ 26/Jul/13 ]
As we discussed in our TechPubs Prio meeting, here's a candidate for blog content.
Comment by Don Pinto [ 26/Aug/14 ]
Perry,

Is it fair to assign this to documentation or should this be closed? What are the tasks here?

Thanks,




[MB-12073] cbq-engine crashes when use negative values in ARRAY_REPEAT Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
query:
cbq> select ARRAY_REPEAT(2, -2);
{
    "results": [cbq>

10:22:51.785959 cbq-engine started...
10:22:51.786116 version: 0.7.0
10:22:51.786127 datastore: dir:/tmp/data
panic: runtime error: makeslice: len out of range

goroutine 277 [running]:
github.com/couchbaselabs/query/expression.(*ArrayRepeat).eval(0xc200627d60, 0xc20016b420, 0x4000000000000000, 0xc20016b420, 0xc000000000000000, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/func_array.go:1087 +0x169
github.com/couchbaselabs/query/expression.(*binaryBase).evaluate(0xc200627d60, 0xc20017fa10, 0xc200627d60, 0xc20016bd10, 0xc200695040, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/binary.go:42 +0x175
github.com/couchbaselabs/query/expression.(*ArrayRepeat).Evaluate(0xc200627d60, 0xc20016bd10, 0xc200695040, 0xc20062c7b0, 0xc200654380, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/func_array.go:1051 +0x78
github.com/couchbaselabs/query/execution.(*InitialProject).processTerms(0xc200654200, 0xc20016a700, 0xc200695040, 0xc200654380, 0xc2006a30d8, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:100 +0x397
github.com/couchbaselabs/query/execution.(*InitialProject).processItem(0xc200654200, 0xc20016a700, 0xc200695040, 0xc200654380, 0x801, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:84 +0x4b1
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:142 +0x25a
sync.(*Once).Do(0xc200654250, 0x2b7ba0a77f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654200, 0xc20062d680, 0xc200654200, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200654200, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 1 [IO wait]:
net.runtime_pollWait(0x2b7b9ff73f00, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc200180230, 0xb, 0xc200136f90)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).accept(0xc2001801b0, 0x9bc7c8, 0x0, 0xc200136f90, 0xb, ...)
/usr/local/go/src/pkg/net/fd_unix.go:385 +0x2c1
net.(*TCPListener).AcceptTCP(0xc2000002d0, 0x562186, 0x2b7b9f301d00, 0x562186)
/usr/local/go/src/pkg/net/tcpsock_posix.go:229 +0x45
net.(*TCPListener).Accept(0xc2000002d0, 0xc20016c300, 0xc200000040, 0xc2001802d0, 0x0, ...)
/usr/local/go/src/pkg/net/tcpsock_posix.go:239 +0x25
net/http.(*Server).Serve(0xc20016c2b0, 0xc20015be40, 0xc2000002d0, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/server.go:1542 +0x85
net/http.(*Server).ListenAndServe(0xc20016c2b0, 0x0, 0xc200136360)
/usr/local/go/src/pkg/net/http/server.go:1532 +0x9e
github.com/couchbaselabs/query/server/http.(*HttpEndpoint).ListenAndServe(0xc20016c2a0, 0x80b6c0, 0xcfebd8)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_endpoint.go:36 +0x2a
main.main()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/main/main.go:61 +0x547

goroutine 2 [syscall]:

goroutine 261 [chan receive]:
github.com/couchbaselabs/query/server/http.(*HttpEndpoint).ServeHTTP(0xc20016c2a0, 0xc20015b480, 0xc20017f700, 0xc20015c750)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_endpoint.go:46 +0xb3
net/http.serverHandler.ServeHTTP(0xc20016c2b0, 0xc20015b480, 0xc20017f700, 0xc20015c750)
/usr/local/go/src/pkg/net/http/server.go:1517 +0x16c
net/http.(*conn).serve(0xc2001802d0)
/usr/local/go/src/pkg/net/http/server.go:1096 +0x765
created by net/http.(*Server).Serve
/usr/local/go/src/pkg/net/http/server.go:1564 +0x266

goroutine 5 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc20013baf0, 0x2b7ba0975ea0)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc20013baa0, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
github.com/couchbaselabs/query/server.(*Server).serviceRequest(0xc20016c1e0, 0xc20013bbe0, 0xc2005b3b00)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:127 +0x412
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:80 +0x6c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 6 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 7 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 8 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 9 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 10 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 11 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 12 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 13 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 14 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 15 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 16 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 17 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 18 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 19 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 20 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 21 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 22 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 23 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 24 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 25 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 26 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 27 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 28 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 29 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 30 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 31 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 32 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 33 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 34 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 35 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 36 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 37 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 38 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 39 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 40 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 41 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 42 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 43 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 44 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 45 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 46 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 47 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 48 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 49 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 50 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 51 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 52 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 53 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 54 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 55 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 56 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 57 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 58 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 59 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 60 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 61 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 62 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 63 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 64 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 65 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 66 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 67 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 68 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 69 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 70 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 71 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 72 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 73 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 74 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 75 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 76 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 77 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 78 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 79 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 80 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 81 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 82 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 83 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 84 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 85 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 86 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 87 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 88 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 89 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 90 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 91 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 92 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 93 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 94 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 95 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 96 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 97 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 98 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 99 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 100 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 101 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 102 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 103 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 104 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 105 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 106 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 107 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 108 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 109 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 110 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 111 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 112 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 113 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 114 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 115 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 116 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 117 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 118 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 119 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 120 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 121 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 122 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 123 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 124 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 125 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 126 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 127 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 128 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 129 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 130 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 131 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 132 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 133 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 134 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 135 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 136 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 137 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 138 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 139 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 140 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 141 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 142 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 143 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 144 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 145 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 146 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 147 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 148 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 149 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 150 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 151 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 152 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 153 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 154 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 155 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 156 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 157 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 158 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 159 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 160 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 161 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 162 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 163 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 164 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 165 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 166 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 167 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 168 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 169 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 170 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 171 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 172 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 173 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 174 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 175 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 176 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 177 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 178 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 179 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 180 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 181 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 182 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 183 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 184 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 185 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 186 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 187 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 188 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 189 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 190 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 191 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 192 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 193 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 194 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 195 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 196 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 197 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 198 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 199 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 200 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 201 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 202 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 203 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 204 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 205 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 206 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 207 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 208 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 209 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 210 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 211 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 212 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 213 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 214 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 215 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 216 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 217 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 218 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 219 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 220 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 221 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 222 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 223 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 224 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 225 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 226 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 227 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 228 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 229 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 230 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 231 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 232 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 233 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 234 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 235 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 236 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 237 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 238 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 239 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 240 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 241 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 242 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 243 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 244 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 245 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 246 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 247 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 248 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 249 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 250 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 251 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 252 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 253 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 254 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 255 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 256 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 257 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 258 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 259 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 260 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 262 [IO wait]:
net.runtime_pollWait(0x2b7b9ff73e60, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2001802c0, 0xb, 0xc200136f90)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc200180240, 0xc200605000, 0x8000, 0x8000, 0x0, ...)
/usr/local/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc200000040, 0xc200605000, 0x8000, 0x8000, 0x8000, ...)
/usr/local/go/src/pkg/net/net.go:123 +0xc3
io.Copy(0xc2005b5390, 0xc200000318, 0xc2005b5090, 0xc200000040, 0x0, ...)
/usr/local/go/src/pkg/io/io.go:348 +0x1c6
net/http.func·004()
/usr/local/go/src/pkg/net/http/server.go:162 +0x66
created by net/http.(*conn).closeNotify
/usr/local/go/src/pkg/net/http/server.go:168 +0x1c6

goroutine 263 [chan receive]:
github.com/couchbaselabs/query/server/http.func·001()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:93 +0x39
created by github.com/couchbaselabs/query/server/http.newHttpRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:95 +0x587

goroutine 265 [select]:
github.com/couchbaselabs/query/server/http.(*httpRequest).writeResults(0xc2005b3b00, 0x1)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_response.go:68 +0x155
github.com/couchbaselabs/query/server/http.(*httpRequest).Execute(0xc2005b3b00, 0xc20016cde0, 0xc200180501)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_response.go:43 +0xa7
created by github.com/couchbaselabs/query/server.(*Server).serviceRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:123 +0x337

goroutine 266 [select]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc200654350, 0x2b7ba0a82f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654300, 0xc20062d5c0, 0xc200654300, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*Stream).RunOnce(0xc200654300, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/stream.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 267 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc20013bf50, 0x2b7ba0a81f70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc20013bf00, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 268 [select]:
github.com/couchbaselabs/query/execution.func·010()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:78 +0x3fc
sync.(*Once).Do(0xc2001805f0, 0x2b7ba0a80f70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Parallel).RunOnce(0xc2001805a0, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:87 +0x6c
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 269 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc20013beb0, 0x2b7ba0a7ff70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc20013be60, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 270 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc20013bb90, 0x2b7ba0a7ef70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc20013bb40, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 271 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc200694050, 0x2b7ba0a7df70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200694000, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 272 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc2006940f0, 0x2b7ba0a7cf70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc2006940a0, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 273 [runnable]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2006542d0, 0x2b7ba0a7bf30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654280, 0xc20062d640, 0xc200654280, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200654280, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 274 [runnable]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2006544d0, 0x2b7ba0a7af30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654480, 0xc20062d640, 0xc200654480, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200654480, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 275 [runnable]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2006545d0, 0x2b7ba0a79f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654580, 0xc20062d640, 0xc200654580, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200654580, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 276 [runnable]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2006546d0, 0x2b7ba0a78f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654680, 0xc20062d640, 0xc200654680, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200654680, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 278 [runnable]:
sync.runtime_Semacquire(0xc200654254)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc200654250)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc200654250, 0x2b7ba0a76f30)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654200, 0xc20062d680, 0xc200654200, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200654200, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 279 [semacquire]:
sync.runtime_Semacquire(0xc200654254)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc200654250)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc200654250, 0x2b7ba0a85f30)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654200, 0xc20062d680, 0xc200654200, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200654200, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 280 [semacquire]:
sync.runtime_Semacquire(0xc200654254)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc200654250)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc200654250, 0x2b7ba0a94f30)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200654200, 0xc20062d680, 0xc200654200, 0xc200654380, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200654200, 0xc200654380, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc




[MB-12072] error message for array_remove can be more informative Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
when I pass only one argument instead of 2 to array_remove I see a runtime error, perhaps it has to give me an error like 'parse error' or something like that?

cbq> select array_remove([1,2,3]) ;
PANIC: runtime error: index out of range.cbq>




Investigate other possible memory allocators that provide the better fragmentation management (MB-10496)

[MB-12067] Investigate explicit defragmentation of ep_engine buckets Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1, 3.0
Fix Version/s: techdebt-backlog
Security Level: Public

Type: Technical task Priority: Major
Reporter: Dave Rigby Assignee: Dave Rigby
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2014-08-26 at 17.59.09.png    

 Description   
As discussed in the parent (MB-10496) task, and in https://docs.google.com/document/d/1sgE9LFfT5ZD4FbSZqCuUtzOoLu5BFi1Kf63R9kYuAyY/edit# it has been demonstrated that there is significant memory usage overhead (over mem_used) due to the chunk-based memory allocation used by malloc - see attached graph.

We should investigate reducing this overhead by performing explicit defragmentation of ep_engine's in-memory hash table.




[MB-11859] Restore "connections" stat in a more suitable format. Created: 31/Jul/14  Updated: 26/Aug/14

Status: In Progress
Project: Couchbase Server
Component/s: storage-engine
Affects Version/s: 3.0.1
Fix Version/s: 3.0.1
Security Level: Public

Type: Task Priority: Major
Reporter: Dave Rigby Assignee: Dave Rigby
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Just after 3.0.1 was forked the "unused" connections stats were removed - see http://review.couchbase.org/38737.

After some discussion with Trond (and a couple of recent instances where they would have been useful for debugging) we think it's worth restoring them. However, with changes in 3.0.1 in how connections are created (e.g. MB-11066) this will need a some reworking, as there is no longer a contiguous array of all connections.



 Comments   
Comment by Dave Rigby [ 26/Aug/14 ]
http://review.couchbase.org/#/c/40924/




[MB-12071] cbq-engine crashes when use negative numbers in array_range Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Server
Component/s: query
Affects Version/s: cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
run the command, it hangs:
cbq> select array_range(-20, 1, 2);
{
    "results": [

and then cbq-engine crashes

09:28:42.547108 cbq-engine started...
09:28:42.547249 version: 0.7.0
09:28:42.547261 datastore: dir:/tmp/data
fatal error: runtime: out of memory

goroutine 315 [running]:
[fp=0x2b1d69e84818] runtime.throw(0xcf8937)
/usr/local/go/src/pkg/runtime/panic.c:473 +0x67
[fp=0x2b1d69e84848] runtime.SysMap(0xc37eda0000, 0x85920000)
/usr/local/go/src/pkg/runtime/mem_linux.c:121 +0xec
[fp=0x2b1d69e84888] runtime.MHeap_SysAlloc(0x2b1d586b4000, 0x85920000)
/usr/local/go/src/pkg/runtime/zmalloc_linux_amd64.c:360 +0x142
[fp=0x2b1d69e848c8] MHeap_Grow(0x2b1d586b4000, 0x85920)
/usr/local/go/src/pkg/runtime/mheap.c:224 +0x55
[fp=0x2b1d69e84908] MHeap_AllocLocked(0x2b1d586b4000, 0x85914, 0x0)
/usr/local/go/src/pkg/runtime/mheap.c:108 +0x2fc
[fp=0x2b1d69e84930] runtime.MHeap_Alloc(0x2b1d586b4000, 0x85914, 0x100000000, 0x1)
/usr/local/go/src/pkg/runtime/mheap.c:77 +0x5e
[fp=0x2b1d69e84988] runtime.mallocgc(0x85913ae0, 0x100000000, 0xc200000001)
/usr/local/go/src/pkg/runtime/zmalloc_linux_amd64.c:58 +0x2fa
[fp=0x2b1d69e849c8] cnew(0x7455a0, 0x85913ae, 0x1)
/usr/local/go/src/pkg/runtime/zmalloc_linux_amd64.c:655 +0xee
[fp=0x2b1d69e849e8] runtime.cnewarray(0x7455a0, 0x85913ae)
/usr/local/go/src/pkg/runtime/zmalloc_linux_amd64.c:673 +0x38
[fp=0x2b1d69e84a00] makeslice1(0x739460, 0x6ada958, 0x85913ae, 0x2b1d69e84aa0)
/usr/local/go/src/pkg/runtime/slice.c:53 +0x45
[fp=0x2b1d69e84a30] growslice1(0x739460, 0xc313f27000, 0x6ada958, 0x6ada958, 0x6ada959, ...)
/usr/local/go/src/pkg/runtime/slice.c:212 +0x56
[fp=0x2b1d69e84a78] runtime.growslice(0x739460, 0xc313f27000, 0x6ada958, 0x6ada958, 0x1, ...)
/usr/local/go/src/pkg/runtime/slice.c:179 +0x9f
[fp=0x2b1d69e84b58] github.com/couchbaselabs/query/expression.(*ArrayRange).eval(0xc2007e07c0, 0xc2007e2960, 0x3, 0x3, 0xc200815180, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/func_array.go:950 +0x44b
[fp=0x2b1d69e84be0] github.com/couchbaselabs/query/expression.(*nAryBase).evaluate(0xc2007e07c0, 0xc20017fa10, 0xc2007e07c0, 0xc20016bd10, 0xc2007e0a40, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/n_ary.go:39 +0x19e
[fp=0x2b1d69e84c40] github.com/couchbaselabs/query/expression.(*ArrayRange).Evaluate(0xc2007e07c0, 0xc20016bd10, 0xc2007e0a40, 0xc20062b7b0, 0xc200815180, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/expression/func_array.go:894 +0x78
[fp=0x2b1d69e84e20] github.com/couchbaselabs/query/execution.(*InitialProject).processTerms(0xc200815000, 0xc20016a700, 0xc2007e0a40, 0xc200815180, 0xc2006a06d8, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:100 +0x397
[fp=0x2b1d69e84e88] github.com/couchbaselabs/query/execution.(*InitialProject).processItem(0xc200815000, 0xc20016a700, 0xc2007e0a40, 0xc200815180, 0xd0b401, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:84 +0x4b1
[fp=0x2b1d69e84f08] github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:142 +0x25a
[fp=0x2b1d69e84f20] sync.(*Once).Do(0xc200815050, 0x2b1d69e84f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
[fp=0x2b1d69e84f60] github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815000, 0xc20062c640, 0xc200815000, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
[fp=0x2b1d69e84f98] github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200815000, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
[fp=0x2b1d69e84fa0] runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1223
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 1 [IO wait]:
net.runtime_pollWait(0x2b1d69381f00, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc200180230, 0xb, 0xc200136f90)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).accept(0xc2001801b0, 0x9bc7c8, 0x0, 0xc200136f90, 0xb, ...)
/usr/local/go/src/pkg/net/fd_unix.go:385 +0x2c1
net.(*TCPListener).AcceptTCP(0xc2000002d0, 0x562186, 0x2b1d6870fd00, 0x562186)
/usr/local/go/src/pkg/net/tcpsock_posix.go:229 +0x45
net.(*TCPListener).Accept(0xc2000002d0, 0xc20016c300, 0xc200000040, 0xc2001802d0, 0x0, ...)
/usr/local/go/src/pkg/net/tcpsock_posix.go:239 +0x25
net/http.(*Server).Serve(0xc20016c2b0, 0xc20015be40, 0xc2000002d0, 0x0, 0x0, ...)
/usr/local/go/src/pkg/net/http/server.go:1542 +0x85
net/http.(*Server).ListenAndServe(0xc20016c2b0, 0x0, 0xc200136360)
/usr/local/go/src/pkg/net/http/server.go:1532 +0x9e
github.com/couchbaselabs/query/server/http.(*HttpEndpoint).ListenAndServe(0xc20016c2a0, 0x80b6c0, 0xcfebd8)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_endpoint.go:36 +0x2a
main.main()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/main/main.go:61 +0x547

goroutine 2 [syscall]:

goroutine 261 [chan receive]:
github.com/couchbaselabs/query/server/http.(*HttpEndpoint).ServeHTTP(0xc20016c2a0, 0xc20015b480, 0xc20077a000, 0xc20015cc30)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_endpoint.go:46 +0xb3
net/http.serverHandler.ServeHTTP(0xc20016c2b0, 0xc20015b480, 0xc20077a000, 0xc20015cc30)
/usr/local/go/src/pkg/net/http/server.go:1517 +0x16c
net/http.(*conn).serve(0xc2001802d0)
/usr/local/go/src/pkg/net/http/server.go:1096 +0x765
created by net/http.(*Server).Serve
/usr/local/go/src/pkg/net/http/server.go:1564 +0x266

goroutine 5 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 6 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 7 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc200691910, 0x2b1d69d91ea0)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc2006918c0, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
github.com/couchbaselabs/query/server.(*Server).serviceRequest(0xc20016c1e0, 0xc20013bbe0, 0xc2007c5000)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:127 +0x412
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:80 +0x6c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 8 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 9 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 10 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 11 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 12 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 13 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 14 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 15 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 16 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 17 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 18 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 19 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 20 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 21 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 22 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 23 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 24 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 25 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 26 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 27 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 28 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 29 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 30 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 31 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 32 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 33 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 34 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 35 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 36 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 37 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 38 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 39 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 40 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 41 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 42 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 43 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 44 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 45 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 46 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 47 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 48 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 49 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 50 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 51 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 52 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 53 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 54 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 55 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 56 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 57 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 58 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 59 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 60 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 61 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 62 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 63 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 64 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 65 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 66 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 67 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 68 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 69 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 70 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 71 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 72 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 73 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 74 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 75 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 76 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 77 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 78 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 79 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 80 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 81 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 82 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 83 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 84 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 85 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 86 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 87 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 88 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 89 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 90 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 91 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 92 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 93 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 94 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 95 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 96 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 97 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 98 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 99 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 100 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 101 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 102 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 103 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 104 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 105 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 106 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 107 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 108 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 109 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 110 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 111 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 112 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 113 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 114 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 115 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 116 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 117 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 118 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 119 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 120 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 121 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 122 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 123 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 124 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 125 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 126 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 127 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 128 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 129 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 130 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 131 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 132 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 133 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 134 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 135 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 136 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 137 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 138 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 139 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 140 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 141 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 142 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 143 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 144 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 145 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 146 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 147 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 148 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 149 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 150 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 151 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 152 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 153 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 154 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 155 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 156 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 157 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 158 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 159 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 160 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 161 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 162 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 163 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 164 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 165 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 166 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 167 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 168 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 169 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 170 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 171 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 172 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 173 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 174 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 175 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 176 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 177 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 178 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 179 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 180 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 181 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 182 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 183 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 184 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 185 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 186 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 187 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 188 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 189 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 190 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 191 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 192 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 193 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 194 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 195 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 196 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 197 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 198 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 199 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 200 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 201 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 202 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 203 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 204 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 205 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 206 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 207 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 208 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 209 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 210 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 211 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 212 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 213 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 214 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 215 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 216 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 217 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 218 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 219 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 220 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 221 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 222 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 223 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 224 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 225 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 226 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 227 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 228 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 229 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 230 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 231 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 232 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 233 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 234 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 235 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 236 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 237 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 238 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 239 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 240 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 241 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 242 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 243 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 244 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 245 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 246 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 247 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 248 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 249 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 250 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 251 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 252 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 253 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 254 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 255 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 256 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 257 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 258 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 259 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 260 [chan receive]:
github.com/couchbaselabs/query/server.(*Server).doServe(0xc20016c1e0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:79 +0x3c
created by github.com/couchbaselabs/query/server.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/server.go:73 +0x4a

goroutine 262 [IO wait]:
net.runtime_pollWait(0x2b1d69381e60, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2001802c0, 0xb, 0xc200136f90)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc200180240, 0xc200604000, 0x8000, 0x8000, 0x0, ...)
/usr/local/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc200000040, 0xc200604000, 0x8000, 0x8000, 0xb1, ...)
/usr/local/go/src/pkg/net/net.go:123 +0xc3
io.Copy(0xc2005b4390, 0xc200000318, 0xc2005b4090, 0xc200000040, 0x161, ...)
/usr/local/go/src/pkg/io/io.go:348 +0x1c6
net/http.func·004()
/usr/local/go/src/pkg/net/http/server.go:162 +0x66
created by net/http.(*conn).closeNotify
/usr/local/go/src/pkg/net/http/server.go:168 +0x1c6

goroutine 263 [chan receive]:
github.com/couchbaselabs/query/server/http.func·001()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:93 +0x39
created by github.com/couchbaselabs/query/server/http.newHttpRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:95 +0x587

goroutine 305 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc200691870, 0x2b1d69e91f70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200691820, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 282 [chan receive]:
github.com/couchbaselabs/query/server/http.func·001()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:93 +0x39
created by github.com/couchbaselabs/query/server/http.newHttpRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:95 +0x587

goroutine 301 [chan receive]:
github.com/couchbaselabs/query/server/http.func·001()
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:93 +0x39
created by github.com/couchbaselabs/query/server/http.newHttpRequest
/root/tuq/gocode/src/github.com/couchbaselabs/query/server/http/http_request.go:95 +0x587

goroutine 306 [select]:
github.com/couchbaselabs/query/execution.func·010()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:78 +0x3fc
sync.(*Once).Do(0xc200180950, 0x2b1d69e8ef70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Parallel).RunOnce(0xc200180900, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:87 +0x6c
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 316 [semacquire]:
sync.runtime_Semacquire(0xc200815054)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc200815050)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc200815050, 0x2b1d69e8df30)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815000, 0xc20062c640, 0xc200815000, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200815000, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·002
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:126 +0xfc

goroutine 308 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc2006919b0, 0x2b1d69e8cf70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200691960, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 309 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc200691a50, 0x2b1d69e8bf70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200691a00, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 310 [select]:
github.com/couchbaselabs/query/execution.func·022()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:82 +0x522
sync.(*Once).Do(0xc200691af0, 0x2b1d69e8af70)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*Sequence).RunOnce(0xc200691aa0, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:91 +0x6c
created by github.com/couchbaselabs/query/execution.func·010
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/parallel.go:67 +0x287

goroutine 312 [select]:
github.com/couchbaselabs/query/execution.func·002()
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:139 +0x2af
sync.(*Once).Do(0xc2008152d0, 0x2b1d69e89f30)
/usr/local/go/src/pkg/sync/once.go:40 +0x7e
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815280, 0xc20062c600, 0xc200815280, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*FinalProject).RunOnce(0xc200815280, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_final.go:38 +0x62
created by github.com/couchbaselabs/query/execution.func·022
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/sequence.go:72 +0x34d

goroutine 317 [semacquire]:
sync.runtime_Semacquire(0xc200815054)
/usr/local/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*Mutex).Lock(0xc200815050)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xbb
sync.(*Once).Do(0xc200815050, 0x2b1d69e88f30)
/usr/local/go/src/pkg/sync/once.go:37 +0x4b
github.com/couchbaselabs/query/execution.(*base).runConsumer(0xc200815000, 0xc20062c640, 0xc200815000, 0xc200815180, 0x0, ...)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/base.go:150 +0x75
github.com/couchbaselabs/query/execution.(*InitialProject).RunOnce(0xc200815000, 0xc200815180, 0x0, 0x0)
/root/tuq/gocode/src/github.com/couchbaselabs/query/execution/project_initial.go:42 +0x62
created by github.com/couchbaselabs/query/execution.func·