[MB-6736] Querying a view on a single node gives inconsistent_state error if bucket is empty Created: 26/Sep/12  Updated: 31/Jan/14  Resolved: 26/Sep/12

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.0
Fix Version/s: 2.0-beta-2
Security Level: Public

Type: Bug Priority: Major
Reporter: Deepkaran Salooja Assignee: Aleksey Kondratenko
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: build 1762 or repo sync + latest couchdb master

Attachments: GZip Archive 127.0.0.1-9000-diag.txt.gz    

 Description   

This is reproducible on both build 1762 or repo sync + latest couchdb master

Steps to reproduce:
1. Create default bucket(1 replica) and 1024 vbuckets.
2. Do no load any data
3. Create a view

curl -v -X PUT -H 'Content-Type: application/json' 'http://Administrator:asdasd@127.0.0.1:9500/default/_design/dev_d1' \
   -d '{"views":{"v3":{"map":"function(doc){\nemit(doc._id,doc.n2);\n}"}}}'
bash> curl -v -X PUT -H 'Content-Type: application/json' 'http://Administrator:asdasd@127.0.0.1:9500/default/_design/dev_d1' \
> -d '{"views":{"v3":{"map":"function(doc){\nemit(doc._id,doc.n2);\n}"}}}'
* About to connect() to 127.0.0.1 port 9500 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 9500 (#0)
* Server auth using Basic with user 'Administrator'
> PUT /default/_design/dev_d1 HTTP/1.1
> Authorization: Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=
> User-Agent: curl/7.21.4 (x86_64-unknown-linux-gnu) libcurl/7.21.4 OpenSSL/1.0.1 zlib/1.2.3.4
> Host: 127.0.0.1:9500
> Accept: */*
> Content-Type: application/json
> Content-Length: 67
>
< HTTP/1.1 201 Created
< Server: MochiWeb/1.0 (Any of you quaids got a smint?)
< Location: http://127.0.0.1:9500/default/_design/dev_d1
< Date: Wed, 26 Sep 2012 10:22:40 GMT
< Content-Type: text/plain;charset=utf-8
< Content-Length: 34
< Cache-Control: must-revalidate
<
{"ok":true,"id":"_design/dev_d1"}
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0


4. Query the view
curl -X GET 'http://Administrator:asdasd@127.0.0.1:9500/default/_design/dev_d1/_view/v3?stale=ok'

* About to connect() to 127.0.0.1 port 9500 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 9500 (#0)
* Server auth using Basic with user 'Administrator'
> GET /default/_design/dev_d1/_view/v3?stale=ok HTTP/1.1
> Authorization: Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=
> User-Agent: curl/7.21.4 (x86_64-unknown-linux-gnu) libcurl/7.21.4 OpenSSL/1.0.1 zlib/1.2.3.4
> Host: 127.0.0.1:9500
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Server: MochiWeb/1.0 (Any of you quaids got a smint?)
< Date: Wed, 26 Sep 2012 10:29:32 GMT
< Content-Type: text/plain;charset=utf-8
< Content-Length: 48
< Cache-Control: must-revalidate
<
{"error":"error","reason":"inconsistent_state"}
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0

Following error can be seen in the logs:

[views:debug,2012-09-26T15:54:34.340,n_0@127.0.0.1:<0.12439.0>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.
[views:debug,2012-09-26T15:54:39.348,n_0@127.0.0.1:<0.12439.0>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.
[views:debug,2012-09-26T15:54:44.356,n_0@127.0.0.1:<0.12439.0>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.
[views:debug,2012-09-26T15:54:49.364,n_0@127.0.0.1:<0.12439.0>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.
[views:debug,2012-09-26T15:54:54.373,n_0@127.0.0.1:<0.12439.0>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.
[views:debug,2012-09-26T15:54:59.380,n_0@127.0.0.1:<0.12439.0>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.
[views:debug,2012-09-26T15:55:04.389,n_0@127.0.0.1:<0.12439.0>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.
[views:debug,2012-09-26T15:55:09.399,n_0@127.0.0.1:<0.12439.0>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.
[views:debug,2012-09-26T15:55:14.408,n_0@127.0.0.1:<0.12439.0>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.


Diagnostics are attached.

 Comments   
Comment by Filipe Manana [ 26/Sep/12 ]
Latest master is currently broken in many ways.

Looking at logs:


[ns_server:info,2012-09-26T15:54:28.534,n_0@127.0.0.1:<0.12483.0>:compaction_daemon:check_all_dbs_exist:1271]Skipping compaction of bucket `default` since at least database `defa
[ns_server:debug,2012-09-26T15:54:28.535,n_0@127.0.0.1:compaction_daemon:compaction_daemon:handle_info:309]Finished compaction iteration.
[ns_server:debug,2012-09-26T15:54:28.536,n_0@127.0.0.1:compaction_daemon:compaction_daemon:schedule_next_compaction:1204]Finished compaction too soon. Next run will be in 30s
[couchdb:info,2012-09-26T15:54:34.335,n_0@127.0.0.1:<0.12439.0>:couch_log:info:39]Set view `default`, group `_design/dev_d1`, missing partitions: [0,1,2,3,4,5,6,7,8,9,10,11,12,13
[views:debug,2012-09-26T15:54:34.340,n_0@127.0.0.1:<0.12439.0>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.

Seems like some vbucket databases are not being created.
Don't which component is causing this, but it's certainly not the view engine, as this was working before build.

The list of suspects, from Peter's reply to a thread:

==================
couchdb master...
8f678a3 Damien Katz, CBD-453 Exclude non-utf8 doc ids from views
b945921 Aliaksey Kandratsenka, MB-6612: exclude opening dbs from
all_known_databases_with_prefix
2613486 Aliaksey Kandratsenka, MB-6612: just store pid in couch server ets
tables
fa5b6fe Aliaksey Kandratsenka, MB-6612: open databases synchronously
7587aa0 Aliaksey Kandratsenka, MB-6612: stop maintaining open dbs lru

==================ep-engine master...
c0084a7 Chiyoung Seo, MB-6676 Don't abort for communication failures with
mccouch
31c007f Chiyoung Seo, MB-6714 Create a new checkpoint for the non-empty
open checkpoint
6306271 Chiyoung Seo, MB-6714 Wake up the flusher after creating a
checkpoint by force


You need to redirect this to someone else or find yourself which commit in which of those components broke stuff.
Comment by Aleksey Kondratenko [ 26/Sep/12 ]
I don't know what you guys are talking about. It works on my box. Please double check your environment
Comment by Aleksey Kondratenko [ 26/Sep/12 ]
actually indeed. it does not work
Comment by Aleksey Kondratenko [ 26/Sep/12 ]
interesting. Simpler fix for 6612 was also merged (succefully!) on top of main fix. And naturally, caused problems.
Comment by Aleksey Kondratenko [ 26/Sep/12 ]
Revert uploaded to: http://review.couchbase.org/#/c/21113/
Comment by Thuan Nguyen [ 26/Sep/12 ]
Integrated in github-couchdb-preview #508 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/508/])
    MB-6736: Revert "MB-6612: exclude opening dbs from..." (Revision 804ef001394269d3c02e3eb1ef05ac3a02f86c54)

     Result = SUCCESS
Farshid Ghods :
Files :
* src/couchdb/couch_server.erl
Generated at Wed Sep 17 21:59:06 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.