Couchbase Server

[windows] Initial index never finishes if keep loading data

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: 2.0-beta-2
  • Fix Version/s: 2.0
  • Component/s: 3rd-party, view-engine
  • Security Level: Public
  • Labels:
    None
  • Environment:

Description

4 nodes cluster/ 8M items / 1 default bucket/ 5 ddocs/ 1 view per ddoc
data loaded by mcsoda
views are:
curl -v -X PUT -H 'Content-Type: application/json' 'http://10.1.3.146:8092/default/_design/view0' -d '{"views": { "view0":{"map":"function(doc){\nemit(doc.key,doc.country);\n}"}}}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://10.1.3.146:8092/default/_design/view1' -d '{"views": { "view1":{"map":"function(doc){\nemit(doc.key,doc.key_num);\n}"}}}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://10.1.3.146:8092/default/_design/view2' -d '{"views": { "view2":{"map":"function(doc){\nemit(doc.key,doc.name);\n}"}}}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://10.1.3.146:8092/default/_design/view3' -d '{"views": { "view3":{"map":"function(doc){\nemit(doc.key,doc.email);\n}"}}}'
curl -v -X PUT -H 'Content-Type: application/json' 'http://10.1.3.146:8092/default/_design/view4' -d '{"views": { "view4":{"map":"function(doc){\nemit(doc.key,doc.city);\n}"}}}'

Steps:
1. Start load items
2. Having 300K create views
3. Observe view indexes. Indexes are building, and reached 100%, but then index is already 2 hours on 100%. Total_changes is increased for the same pid, and index doesn't finish. Physically I see index files.


output of some _active_tasks:

[{"pid":"<0.228.0>","limit":4,"running":4,"started_on":1352881114,"type":"couch_main_index_barrier","updated_on":1352881985,"waiting":1},{"pid":"<0.229.0>","limit":2,"running":2,"started_on":1352881114,"type":"couch_replica_index_barrier","updated_on":1352881985,"waiting":3},{"pid":"<0.25593.0>","changes_done":299044,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d",
"started_on":1352881956,"total_changes":299044,"type":"indexer","updated_on":1352889782},
{"pid":"<0.25596.0>","changes_done":289133,"design_documents":["_design/view1"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":289133,"type":"indexer","updated_on":1352889851},{"pid":"<0.25599.0>","changes_done":306227,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":306227,"type":"indexer","updated_on":1352889794},{"pid":"<0.25602.0>","changes_done":297028,"design_documents":["_design/view0"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":297028,"type":"indexer","updated_on":1352889871},{"pid":"<0.25730.0>","design_documents":["_design/view2"],"indexer_type":"replica","set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25736.0>","changes_done":292103,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"total_changes":292103,"type":"indexer","updated_on":1352889880},{"pid":"<0.25739.0>","design_documents":["_design/view3"],"indexer_type":"replica","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25743.0>","changes_done":292306,"design_documents":["_design/view3"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"total_changes":292306,"type":"indexer","updated_on":1352889731},{"pid":"<0.25848.0>","design_documents":["_design/view4"],"indexer_type":"main","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985},{"pid":"<0.25867.0>","design_documents":["_design/view4"],"indexer_type":"replica","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985}]

[{"pid":"<0.228.0>","limit":4,"running":4,"started_on":1352881114,"type":"couch_main_index_barrier","updated_on":1352881985,"waiting":1},{"pid":"<0.229.0>","limit":2,"running":2,"started_on":1352881114,"type":"couch_replica_index_barrier","updated_on":1352881985,"waiting":3},{"pid":"<0.25593.0>","changes_done":357450,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":357450,"type":"indexer","updated_on":1352890263},{"pid":"<0.25596.0>","changes_done":348670,"design_documents":["_design/view1"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":348670,"type":"indexer","updated_on":1352890619},{"pid":"<0.25599.0>","changes_done":345303,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":345303,"type":"indexer","updated_on":1352890232},{"pid":"<0.25602.0>","changes_done":335241,"design_documents":["_design/view0"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":335241,"type":"indexer","updated_on":1352890235},{"pid":"<0.25730.0>","design_documents":["_design/view2"],"indexer_type":"replica","set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25736.0>","changes_done":324299,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"total_changes":324299,"type":"indexer","updated_on":1352890231},{"pid":"<0.25739.0>","design_documents":["_design/view3"],"indexer_type":"replica","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25743.0>","changes_done":365889,"design_documents":["_design/view3"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"total_changes":365889,"type":"indexer","updated_on":1352890878},{"pid":"<0.25848.0>","design_documents":["_design/view4"],"indexer_type":"main","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985},{"pid":"<0.25867.0>","design_documents":["_design/view4"],"indexer_type":"replica","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985}]

[{"pid":"<0.228.0>","limit":4,"running":4,"started_on":1352881114,"type":"couch_main_index_barrier","updated_on":1352881985,"waiting":1},{"pid":"<0.229.0>","limit":2,"running":2,"started_on":1352881114,"type":"couch_replica_index_barrier","updated_on":1352881985,"waiting":3},{"pid":"<0.25593.0>","changes_done":357450,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":357450,"type":"indexer","updated_on":1352890263},{"pid":"<0.25596.0>","changes_done":348670,"design_documents":["_design/view1"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":348670,"type":"indexer","updated_on":1352890619},{"pid":"<0.25599.0>","changes_done":345303,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":345303,"type":"indexer","updated_on":1352890232},{"pid":"<0.25602.0>","changes_done":335241,"design_documents":["_design/view0"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":335241,"type":"indexer","updated_on":1352890235},{"pid":"<0.25730.0>","design_documents":["_design/view2"],"indexer_type":"replica","set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25736.0>","changes_done":324299,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"total_changes":324299,"type":"indexer","updated_on":1352890231},{"pid":"<0.25739.0>","design_documents":["_design/view3"],"indexer_type":"replica","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25743.0>","changes_done":365889,"design_documents":["_design/view3"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"total_changes":365889,"type":"indexer","updated_on":1352890878},{"pid":"<0.25848.0>","design_documents":["_design/view4"],"indexer_type":"main","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985},{"pid":"<0.25867.0>","design_documents":["_design/view4"],"indexer_type":"replica","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985}]

[{"pid":"<0.228.0>","limit":4,"running":4,"started_on":1352881114,"type":"couch_main_index_barrier","updated_on":1352881985,"waiting":1},{"pid":"<0.229.0>","limit":2,"running":2,"started_on":1352881114,"type":"couch_replica_index_barrier","updated_on":1352881985,"waiting":3},{"pid":"<0.25593.0>","changes_done":379175,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":379175,"type":"indexer","updated_on":1352890975},{"pid":"<0.25596.0>","changes_done":348670,"design_documents":["_design/view1"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":348670,"type":"indexer","updated_on":1352890619},{"pid":"<0.25599.0>","changes_done":367664,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":367664,"type":"indexer","updated_on":1352890955},{"pid":"<0.25602.0>","changes_done":362416,"design_documents":["_design/view0"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":362416,"type":"indexer","updated_on":1352891267},{"pid":"<0.25730.0>","design_documents":["_design/view2"],"indexer_type":"replica","set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25736.0>","changes_done":349987,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"total_changes":349987,"type":"indexer","updated_on":1352890909},{"pid":"<0.25739.0>","design_documents":["_design/view3"],"indexer_type":"replica","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25743.0>","changes_done":365889,"design_documents":["_design/view3"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"total_changes":365889,"type":"indexer","updated_on":1352890878},{"pid":"<0.25848.0>","design_documents":["_design/view4"],"indexer_type":"main","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985},{"pid":"<0.25867.0>","design_documents":["_design/view4"],"indexer_type":"replica","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985}]

[{"pid":"<0.228.0>","limit":4,"running":4,"started_on":1352881114,"type":"couch_main_index_barrier","updated_on":1352881985,"waiting":1},{"pid":"<0.229.0>","limit":2,"running":2,"started_on":1352881114,"type":"couch_replica_index_barrier","updated_on":1352881985,"waiting":3},{"pid":"<0.25593.0>","changes_done":400448,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":400448,"type":"indexer","updated_on":1352891802},{"pid":"<0.25596.0>","changes_done":370119,"design_documents":["_design/view1"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":370119,"type":"indexer","updated_on":1352891467},{"pid":"<0.25599.0>","changes_done":367664,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":367664,"type":"indexer","updated_on":1352890955},{"pid":"<0.25602.0>","changes_done":362416,"design_documents":["_design/view0"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":362416,"type":"indexer","updated_on":1352891267},{"pid":"<0.25730.0>","design_documents":["_design/view2"],"indexer_type":"replica","set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25736.0>","changes_done":355207,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"total_changes":355207,"type":"indexer","updated_on":1352891750},{"pid":"<0.25739.0>","design_documents":["_design/view3"],"indexer_type":"replica","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25743.0>","changes_done":375590,"design_documents":["_design/view3"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"total_changes":375590,"type":"indexer","updated_on":1352891723},{"pid":"<0.25848.0>","design_documents":["_design/view4"],"indexer_type":"main","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985},{"pid":"<0.25867.0>","design_documents":["_design/view4"],"indexer_type":"replica","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985}]


[{"pid":"<0.228.0>","limit":4,"running":4,"started_on":1352881114,"type":"couch_main_index_barrier","updated_on":1352881985,"waiting":1},{"pid":"<0.229.0>","limit":2,"running":2,"started_on":1352881114,"type":"couch_replica_index_barrier","updated_on":1352881985,"waiting":3},{"pid":"<0.25593.0>","changes_done":400448,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":400448,"type":"indexer","updated_on":1352891802},{"pid":"<0.25596.0>","changes_done":370119,"design_documents":["_design/view1"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":370119,"type":"indexer","updated_on":1352891467},{"pid":"<0.25599.0>","changes_done":389626,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":389626,"type":"indexer","updated_on":1352891876},{"pid":"<0.25602.0>","changes_done":384506,"design_documents":["_design/view0"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":384506,"type":"indexer","updated_on":1352892179},{"pid":"<0.25730.0>","design_documents":["_design/view2"],"indexer_type":"replica","set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25736.0>","changes_done":355207,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"total_changes":355207,"type":"indexer","updated_on":1352891750},{"pid":"<0.25739.0>","design_documents":["_design/view3"],"indexer_type":"replica","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25743.0>","changes_done":375590,"design_documents":["_design/view3"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"total_changes":375590,"type":"indexer","updated_on":1352891723},{"pid":"<0.25848.0>","design_documents":["_design/view4"],"indexer_type":"main","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985},{"pid":"<0.25867.0>","design_documents":["_design/view4"],"indexer_type":"replica","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985}]

[{"pid":"<0.228.0>","limit":4,"running":4,"started_on":1352881114,"type":"couch_main_index_barrier","updated_on":1352881985,"waiting":1},{"pid":"<0.229.0>","limit":2,"running":2,"started_on":1352881114,"type":"couch_replica_index_barrier","updated_on":1352881985,"waiting":3},{"pid":"<0.25593.0>","changes_done":405805,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":405805,"type":"indexer","updated_on":1352892516},{"pid":"<0.25596.0>","changes_done":375952,"design_documents":["_design/view1"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352881956,"total_changes":375952,"type":"indexer","updated_on":1352892366},{"pid":"<0.25599.0>","changes_done":395533,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":395533,"type":"indexer","updated_on":1352892568},{"pid":"<0.25602.0>","changes_done":384506,"design_documents":["_design/view0"],"indexer_type":"replica","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352881956,"total_changes":384506,"type":"indexer","updated_on":1352892179},{"pid":"<0.25730.0>","design_documents":["_design/view2"],"indexer_type":"replica","set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25736.0>","changes_done":376175,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352881970,"total_changes":376175,"type":"indexer","updated_on":1352892483},{"pid":"<0.25739.0>","design_documents":["_design/view3"],"indexer_type":"replica","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"type":"blocked_indexer","updated_on":1352881970},{"pid":"<0.25743.0>","changes_done":397184,"design_documents":["_design/view3"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352881970,"total_changes":397184,"type":"indexer","updated_on":1352892468},{"pid":"<0.25848.0>","design_documents":["_design/view4"],"indexer_type":"main","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985},{"pid":"<0.25867.0>","design_documents":["_design/view4"],"indexer_type":"replica","set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352881985,"type":"blocked_indexer","updated_on":1352881985}]

attaching logs

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Iryna Mironava added a comment -
https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/diag-135.txt.gz
https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/diag-146.txt.gz
https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/diag-147.txt.gz
https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/diag-151.txt.gz

https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/cbcollect-135.zip
https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/cbcollect-146.zip
https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/cbcollect-147.zip
https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/cbcollect-151.zip

Show
Iryna Mironava added a comment - https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/diag-135.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/diag-146.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/diag-147.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/diag-151.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/cbcollect-135.zip https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/cbcollect-146.zip https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/cbcollect-147.zip https://s3.amazonaws.com/bugdb/jira/MB-7180/aa8da72b/cbcollect-151.zip
Hide
Permalink
Filipe Manana added a comment -
Hi Iryna,

I don't see any errors anywhere, nor blocked (deadlocked) indexer processes from diag - they're all in runnable state but somehow Erlang VM is not scheduling them.

I do see there's lots of vbucket database compaction going on.

If you stop the load, does it finish eventually?

Somehow I think there might be a high level issue with Erlang VM, perhaps you can try replacing the startup option +S 128:128 with +A 16 and see if it still happens?
Show
Filipe Manana added a comment - Hi Iryna, I don't see any errors anywhere, nor blocked (deadlocked) indexer processes from diag - they're all in runnable state but somehow Erlang VM is not scheduling them. I do see there's lots of vbucket database compaction going on. If you stop the load, does it finish eventually? Somehow I think there might be a high level issue with Erlang VM, perhaps you can try replacing the startup option +S 128:128 with +A 16 and see if it still happens?
Hide
Permalink
Farshid Ghods added a comment -
This is Iryna attempt to reproduce MB-7113 in small scale. so far we dont see memcached or elang timeout even though there are 5 ddocs.

Iryna,
as we discussed let's move this test to ec2 environment , and put data and index files on two different drives so that indexing and data compaction don't compete.

if you have the cluster running anyways it is worth waiting after load is stopped so that data compaction does not compete with index anymore.

Show
Farshid Ghods added a comment - This is Iryna attempt to reproduce MB-7113 in small scale. so far we dont see memcached or elang timeout even though there are 5 ddocs. Iryna, as we discussed let's move this test to ec2 environment , and put data and index files on two different drives so that indexing and data compaction don't compete. if you have the cluster running anyways it is worth waiting after load is stopped so that data compaction does not compete with index anymore.
Hide
Permalink
Farshid Ghods added a comment -
Filipe,

cluster is available here : http://10.1.3.146:8091/index.html#sec=buckets
Adminstrator/password
Show
Farshid Ghods added a comment - Filipe, cluster is available here : http://10.1.3.146:8091/index.html#sec=buckets Adminstrator/password
Hide
Permalink
Filipe Manana added a comment -
Farshid, looking at the cluster now, the indexers finished and nothing seems blocked.
It seems someone finished the process(es) loading data in parallel?
Show
Filipe Manana added a comment - Farshid, looking at the cluster now, the indexers finished and nothing seems blocked. It seems someone finished the process(es) loading data in parallel?
Hide
Permalink
Iryna Mironava added a comment -
load was stopped, index continued adding by 3K-5K to total_changes at least two hours more. will try with +A 16
Show
Iryna Mironava added a comment - load was stopped, index continued adding by 3K-5K to total_changes at least two hours more. will try with +A 16
Hide
Permalink
Iryna Mironava added a comment -
tried with +A 16, index wasn't stuck for long time. Just 15 minutes it was updating total_changes having 100%

[{"pid":"<0.239.0>","limit":4,"running":4,"started_on":1352963383,"type":"couch_main_index_barrier","updated_on":1352966734,"waiting":1},{"pid":"<0.240.0>","limit":2,"running":0,"started_on":1352963383,"type":"couch_replica_index_barrier","updated_on":1352963383,"waiting":0},{"pid":"<0.19681.1>","design_documents":["_design/view3"],"indexer_type":"main","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352966734,"type":"blocked_indexer","updated_on":1352966734},{"pid":"<0.19691.1>","changes_done":245851,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352966734,"total_changes":245851,"type":"indexer","updated_on":1352967207},{"pid":"<0.19695.1>","changes_done":238044,"design_documents":["_design/view4"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352966734,"total_changes":238044,"type":"indexer","updated_on":1352967190},{"pid":"<0.19698.1>","changes_done":235101,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352966734,"total_changes":235101,"type":"indexer","updated_on":1352967200},{"pid":"<0.19701.1>","changes_done":241506,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352966734,"total_changes":241506,"type":"indexer","updated_on":1352967196},{"pid":"<0.27162.1>","bucket":"default","original_target":{"type":"bucket"},"progress":95,"started_on":1352967054,"total_vbuckets":513,"trigger_type":"scheduled","type":"bucket_compaction","updated_on":1352967208,"vbuckets_done":488}]


[{"pid":"<0.239.0>","limit":4,"running":4,"started_on":1352963383,"type":"couch_main_index_barrier","updated_on":1352966734,"waiting":1},{"pid":"<0.240.0>","limit":2,"running":0,"started_on":1352963383,"type":"couch_replica_index_barrier","updated_on":1352963383,"waiting":0},{"pid":"<0.19681.1>","design_documents":["_design/view3"],"indexer_type":"main","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352966734,"type":"blocked_indexer","updated_on":1352966734},{"pid":"<0.19691.1>","changes_done":255921,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352966734,"total_changes":255921,"type":"indexer","updated_on":1352967225},{"pid":"<0.19695.1>","changes_done":253255,"design_documents":["_design/view4"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352966734,"total_changes":253255,"type":"indexer","updated_on":1352967223},{"pid":"<0.19698.1>","changes_done":245305,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352966734,"total_changes":245305,"type":"indexer","updated_on":1352967227},{"pid":"<0.19701.1>","changes_done":258002,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352966734,"total_changes":258002,"type":"indexer","updated_on":1352967234}]

[{"pid":"<0.239.0>","limit":4,"running":4,"started_on":1352963383,"type":"couch_main_index_barrier","updated_on":1352966734,"waiting":1},{"pid":"<0.240.0>","limit":2,"running":0,"started_on":1352963383,"type":"couch_replica_index_barrier","updated_on":1352963383,"waiting":0},{"pid":"<0.19681.1>","design_documents":["_design/view3"],"indexer_type":"main","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352966734,"type":"blocked_indexer","updated_on":1352966734},{"pid":"<0.19691.1>","changes_done":257733,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352966734,"total_changes":257733,"type":"indexer","updated_on":1352967236},{"pid":"<0.19695.1>","changes_done":257863,"design_documents":["_design/view4"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352966734,"total_changes":257863,"type":"indexer","updated_on":1352967234},{"pid":"<0.19698.1>","changes_done":258398,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352966734,"total_changes":258398,"type":"indexer","updated_on":1352967252},{"pid":"<0.19701.1>","changes_done":258002,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352966734,"total_changes":258002,"type":"indexer","updated_on":1352967234}]
Show
Iryna Mironava added a comment - tried with +A 16, index wasn't stuck for long time. Just 15 minutes it was updating total_changes having 100% [{"pid":"<0.239.0>","limit":4,"running":4,"started_on":1352963383,"type":"couch_main_index_barrier","updated_on":1352966734,"waiting":1},{"pid":"<0.240.0>","limit":2,"running":0,"started_on":1352963383,"type":"couch_replica_index_barrier","updated_on":1352963383,"waiting":0},{"pid":"<0.19681.1>","design_documents":["_design/view3"],"indexer_type":"main","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352966734,"type":"blocked_indexer","updated_on":1352966734},{"pid":"<0.19691.1>","changes_done":245851,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352966734,"total_changes":245851,"type":"indexer","updated_on":1352967207},{"pid":"<0.19695.1>","changes_done":238044,"design_documents":["_design/view4"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352966734,"total_changes":238044,"type":"indexer","updated_on":1352967190},{"pid":"<0.19698.1>","changes_done":235101,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352966734,"total_changes":235101,"type":"indexer","updated_on":1352967200},{"pid":"<0.19701.1>","changes_done":241506,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352966734,"total_changes":241506,"type":"indexer","updated_on":1352967196},{"pid":"<0.27162.1>","bucket":"default","original_target":{"type":"bucket"},"progress":95,"started_on":1352967054,"total_vbuckets":513,"trigger_type":"scheduled","type":"bucket_compaction","updated_on":1352967208,"vbuckets_done":488}] [{"pid":"<0.239.0>","limit":4,"running":4,"started_on":1352963383,"type":"couch_main_index_barrier","updated_on":1352966734,"waiting":1},{"pid":"<0.240.0>","limit":2,"running":0,"started_on":1352963383,"type":"couch_replica_index_barrier","updated_on":1352963383,"waiting":0},{"pid":"<0.19681.1>","design_documents":["_design/view3"],"indexer_type":"main","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352966734,"type":"blocked_indexer","updated_on":1352966734},{"pid":"<0.19691.1>","changes_done":255921,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352966734,"total_changes":255921,"type":"indexer","updated_on":1352967225},{"pid":"<0.19695.1>","changes_done":253255,"design_documents":["_design/view4"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352966734,"total_changes":253255,"type":"indexer","updated_on":1352967223},{"pid":"<0.19698.1>","changes_done":245305,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352966734,"total_changes":245305,"type":"indexer","updated_on":1352967227},{"pid":"<0.19701.1>","changes_done":258002,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352966734,"total_changes":258002,"type":"indexer","updated_on":1352967234}] [{"pid":"<0.239.0>","limit":4,"running":4,"started_on":1352963383,"type":"couch_main_index_barrier","updated_on":1352966734,"waiting":1},{"pid":"<0.240.0>","limit":2,"running":0,"started_on":1352963383,"type":"couch_replica_index_barrier","updated_on":1352963383,"waiting":0},{"pid":"<0.19681.1>","design_documents":["_design/view3"],"indexer_type":"main","set":"default","signature":"10c44ad8ced8fa36c0b4c6bf60059c19","started_on":1352966734,"type":"blocked_indexer","updated_on":1352966734},{"pid":"<0.19691.1>","changes_done":257733,"design_documents":["_design/view1"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"e4e45966441bf87daf6d75e9ebdbb66d","started_on":1352966734,"total_changes":257733,"type":"indexer","updated_on":1352967236},{"pid":"<0.19695.1>","changes_done":257863,"design_documents":["_design/view4"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"bb8abaa226ed0ded4d13a6757514b19c","started_on":1352966734,"total_changes":257863,"type":"indexer","updated_on":1352967234},{"pid":"<0.19698.1>","changes_done":258398,"design_documents":["_design/view0"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"7056d17d8d01fa89fafc2f192f6540db","started_on":1352966734,"total_changes":258398,"type":"indexer","updated_on":1352967252},{"pid":"<0.19701.1>","changes_done":258002,"design_documents":["_design/view2"],"indexer_type":"main","initial_build":true,"progress":100,"set":"default","signature":"a95bff4f72d6836f5795ad4f1000d744","started_on":1352966734,"total_changes":258002,"type":"indexer","updated_on":1352967234}]
Hide
Permalink
Farshid Ghods added a comment -
Filipe/Alk/Yaseen/Damien,

looks like on windows we get a much better performance with +A option . on the other hand i dont think we ever saw erlang crash on windows as we saw on linux. linux has 64-bit erlang and windows has the 32-bit

i have asked Iryna to run more rebalance and index tests on windows with +A16 option to see if such crash is reproducible if not i suggest we ship windows with +A16 and continue running lots of tests with this option
Show
Farshid Ghods added a comment - Filipe/Alk/Yaseen/Damien, looks like on windows we get a much better performance with +A option . on the other hand i dont think we ever saw erlang crash on windows as we saw on linux. linux has 64-bit erlang and windows has the 32-bit i have asked Iryna to run more rebalance and index tests on windows with +A16 option to see if such crash is reproducible if not i suggest we ship windows with +A16 and continue running lots of tests with this option
Hide
Permalink
Filipe Manana added a comment -
Erlang issue with scheduler threads.
Show
Filipe Manana added a comment - Erlang issue with scheduler threads.
Hide
Permalink
Steve Yen added a comment -
bug scrub...

Iryna, please re-run test with 16:16 scheduler threads.
Show
Steve Yen added a comment - bug scrub... Iryna, please re-run test with 16:16 scheduler threads.
Hide
Permalink
Steve Yen added a comment -
bug-scrub: moved to 2.0.1, this is an async-thread-related issue
Show
Steve Yen added a comment - bug-scrub: moved to 2.0.1, this is an async-thread-related issue
Hide
Permalink
Iryna Mironava added a comment -
tried the same with 16:16, index is quicker, indexes were not stuck on 100%. (build 1954 with +S 16:16)
Show
Iryna Mironava added a comment - tried the same with 16:16, index is quicker, indexes were not stuck on 100%. (build 1954 with +S 16:16)
Hide
Permalink
Dipti Borkar added a comment -
2.0.1 may include additional improvements
Show
Dipti Borkar added a comment - 2.0.1 may include additional improvements

People

  • Assignee:
    Filipe Manana
    Reporter:
    Iryna Mironava
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: