[MB-6640] Indexing progress stuck at 100% Created: 13/Sep/12  Updated: 09/Jan/13  Resolved: 13/Sep/12

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.0-beta
Fix Version/s: 2.0-beta-2
Security Level: Public

Type: Bug Priority: Major
Reporter: Karan Kumar Assignee: Filipe Manana
Resolution: Duplicate Votes: 0
Labels: system-test
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 1717

Attachments: Zip Archive 10.6.2.11-9132012-diag.zip     Zip Archive 10.6.2.7-9132012-diag.zip     PNG File Screen Shot 2012-09-13 at 1.24.13 AM.png    

 Description   

The view engine is still indexing the ddoc and both the node report it as 100%.

While it finished indexing at 8PM from couchdb logs its still showing indexing at 100% at 1:30AM, after 5hrs.

Also, I see that the indexes are still growing in size on disk, both replica and main index. (The cluster is completely idle, no front end load, no tap load)

Are indexes ready to be queries? Are we calculating stats for these indexes?

Node 7, the progress shown at 19:06 is 100%
[couchdb:debug,2012-09-12T20:58:13.048,ns_1@10.6.2.7:couch_task_status:couch_log:debug:36]New task status for <0.18201.3>: [{changes_done,7079},
                                  {design_documents,[<<"_design/d8">>]},
                                  {indexer_type,main},
                                  {initial_build,false},
                                  {progress,100},
                                  {set,<<"default">>},
                                  {signature,
                                      <<"02dcc6636d5e6f2bb68fd0fc592ba9a0">>},
                                  {started_on,1347508674},
                                  {total_changes,7079},
                                  {type,indexer},
                                  {updated_on,1347508693}]

Node 11
[couchdb:debug,2012-09-12T20:58:03.987,ns_1@10.6.2.11:couch_task_status:couch_log:debug:36]New task status for <0.28296.3>: [{changes_done,5116},
                                  {design_documents,[<<"_design/d8">>]},
                                  {indexer_type,main},
                                  {initial_build,false},
                                  {progress,100},
                                  {set,<<"default">>},
                                  {signature,
                                      <<"02dcc6636d5e6f2bb68fd0fc592ba9a0">>},
                                  {started_on,1347508670},
                                  {total_changes,5116},
                                  {type,indexer},
                                  {updated_on,1347508683}]


 Comments   
Comment by Filipe Manana [ 13/Sep/12 ]
It's not stuck, for the initial index build we simply don't have a way to report meaningful progress when it reaches the final phase, staying at 100%.

I don't know why things are so slow in the test environment, I've never seen this locally, specially for such small datasets.

This relates to http://www.couchbase.com/issues/browse/CBD-74.
Comment by Filipe Manana [ 13/Sep/12 ]
Actually it's not initial phase, still it can happen and it's known, explained in CBD-74.

This is also same symptom as MB-6629, terrible slowdown in the system.
Comment by Filipe Manana [ 13/Sep/12 ]
Same as in MB-6629
Generated at Wed May 22 03:37:24 CDT 2013 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.