[MB-6633] querying for views during rebalancing through node's REST api before it has any active vbucket return ""error":"no_active_vbuckets" Created: 12/Sep/12  Updated: 10/Jan/13  Resolved: 21/Sep/12

Status: Closed
Project: Couchbase Server
Component/s: ns_server, view-engine
Affects Version/s: None
Fix Version/s: 2.0-beta-2
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Iryna Mironava Assignee: Aleksey Kondratenko
Resolution: Fixed Votes: 0
Labels: 2.0-beta-release-notes, regression
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: build 1717
centos
1 bucket, 200K items

Attachments: GZip Archive 10.3.121.104-8091-diag.txt.gz     GZip Archive 10.3.121.105-8091-diag.txt.gz    

 Description   
test
-t viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_incrementaly,skip_rebalance=true

steps to repro:
1. load data
2. start rebalance 1->2
3. wait some time, (in my case rebalance is going on 15 mins) try to see documents or query view from node which is adding to cluster
no_active_vbuckets error appears
iryna@ubuntu:~/couchbase/testrunner$ curl -v 'http://10.3.121.105:8092/default/_design/test_view-df0f1d8/_view/test_view-df0f1d8?debug=true&start_key=%5B2008%2C7%2Cnull%5D&connection_timeout=60000'
* About to connect() to 10.3.121.105 port 8092 (#0)
* Trying 10.3.121.105...
* connected
* Connected to 10.3.121.105 (10.3.121.105) port 8092 (#0)
> GET /default/_design/test_view-df0f1d8/_view/test_view-df0f1d8?debug=true&start_key=%5B2008%2C7%2Cnull%5D&connection_timeout=60000 HTTP/1.1
> User-Agent: curl/7.26.0
> Host: 10.3.121.105:8092
> Accept: */*
>
< HTTP/1.1 404 Object Not Found
< Server: MochiWeb/1.0 (Any of you quaids got a smint?)
< Date: Wed, 12 Sep 2012 22:07:23 GMT
< Content-Type: text/plain;charset=utf-8
< Content-Length: 106
< Cache-Control: must-revalidate
<
{"error":"no_active_vbuckets","reason":"Cannot execute view query since the node has no active vbuckets"}
* Connection #0 to host 10.3.121.105 left intact
* Closing connection #0


 Comments   
Comment by Farshid Ghods (Inactive) [ 12/Sep/12 ]
looks like a blocker to me , is this regression on 1717 ? have you been able to reproduce ths on earlier builds ?
Comment by Iryna Mironava [ 12/Sep/12 ]
first seen on 1715
Comment by Farshid Ghods (Inactive) [ 12/Sep/12 ]
ok this could be related to ep-engine changes?

what do you see in diags ?
can you do more investigations
Comment by Iryna Mironava [ 12/Sep/12 ]
it doesn't seem to be an ep-engine problem, we can query views from nodes which was in cluster successfully, the error appears only if we query node that we try to add. In logs i don't see something wrong, it is building replicas.
Comment by Farshid Ghods (Inactive) [ 12/Sep/12 ]
Aliaksey,

I dont think this is a regression but the use cases is too simple ( doing view queries from a node that is being rebalanced in but its still doing replica building and does not have an active vbucket )
Comment by Aleksey Kondratenko [ 12/Sep/12 ]
I need 100% clarity here. Does that happen _after_ rebalance or _during_ rebalance.

I'm asking because sometimes we need quite a lot of time to start moving any active vbuckets into new node. So if it's not after, then it's simple duplicate of MB-6440.
Comment by Farshid Ghods (Inactive) [ 13/Sep/12 ]
MB-6440 says the error only appears for a few second after rebalancing this bug says it happens even after 10 minutes post rebalancing
Comment by Iryna Mironava [ 13/Sep/12 ]
it happens during rebalance, but for much longer time window, 10-15 minutes i cannot see documents and query views. Is it a duplicate?
Comment by Aleksey Kondratenko [ 13/Sep/12 ]
Yes it's duplicate.

It is reasonably simple to fix, _but_ it may have some quite bad effects on clients.
Comment by Aleksey Kondratenko [ 13/Sep/12 ]
BTW this is not AFAIK a regression. Do you want me to fix this ASAP?
Comment by Farshid Ghods (Inactive) [ 13/Sep/12 ]
moving this to beta refresh so once beta is shipped we will merge it
Comment by Aleksey Kondratenko [ 14/Sep/12 ]
Some fix is coming. But now I understand there's another cause of this. I think enabling consistent views will fix other and potentially more prolonged it time cause.
Comment by kzeller [ 17/Sep/12 ]
Add to beta release notes as : If a cluster is being rebalanced, and a node does not yet have any active
        vBuckets, you will receive the error
        <literal>"error":"no_active_vbuckets"</literal> when you query a view
       via the REST API.
Comment by Aleksey Kondratenko [ 21/Sep/12 ]
merged fix
Generated at Thu Aug 28 18:19:29 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.