Couchbase Server

Odd warmup result after shutdown followed by restart

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 2.0-developer-preview-4
  • Fix Version/s: 2.0-beta
  • Component/s: couchbase-bucket
  • Security Level: Public
  • Labels:
    None
  • Environment:
    Linux

Description

Have 2-node cluster on Linux, and run memcachetest to insert 1M items. After rebalance, it has 2M items on the 2-node cluster. Kill both memcached, and then restart the 2 nodes. When warmup completes, I see some odd result. One node #1, it has ~54% active resident ratio and 0% replica resident ratio; on node #2, it has 0% active resident ratio and ~52% replica resident ratio. In ns_server log, I see message "Enough MB of data loaded to enable traffic" from maybeEnableTraffic(). I am not sure if it has anything to do with the odd result.

On node#2, it seems that it loaded all replica vbuckets before load stops. I am wondering whether it is good idea to load active vbuckets before replica ones during warmup.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Liang Guo added a comment -
Node n_0 active_items 500K replica_items 500K, active_resident 51.4%, replica_resident 0%
Node n_1 active_items 500K replica_items 500K, active_resident 0%, replica_resident 52.8%

In n_1 log, from my debug trace, I saw StoredValue::ejectValue was called a lot before vbcukets states were set to active from dead after warmup completed. This is why 0% active resident ratio, nonResident is 500K while numItems is 500K.
Show
Liang Guo added a comment - Node n_0 active_items 500K replica_items 500K, active_resident 51.4%, replica_resident 0% Node n_1 active_items 500K replica_items 500K, active_resident 0%, replica_resident 52.8% In n_1 log, from my debug trace, I saw StoredValue::ejectValue was called a lot before vbcukets states were set to active from dead after warmup completed. This is why 0% active resident ratio, nonResident is 500K while numItems is 500K.
Hide
Permalink
Liang Guo added a comment -
Items got ejected immediately after insertion due to memory constraint. Because warmup loads replica vbuckets before active ones on ns_1, then it ended up having 500K active_non_resident_items and therefore 0% active_resident. On n_0, it'd be the opposite, because all replica items got ejected while active ones remained resident. It is probably not a functional bug, but it is some design flaw because it would require a lot of BG fetches to bring active data back into memory.

This is evidence to the necessity of proposed optimization of loading active vbuckets before replica ones. In dgm, at least we would have some active items resident in memory.
Show
Liang Guo added a comment - Items got ejected immediately after insertion due to memory constraint. Because warmup loads replica vbuckets before active ones on ns_1, then it ended up having 500K active_non_resident_items and therefore 0% active_resident. On n_0, it'd be the opposite, because all replica items got ejected while active ones remained resident. It is probably not a functional bug, but it is some design flaw because it would require a lot of BG fetches to bring active data back into memory. This is evidence to the necessity of proposed optimization of loading active vbuckets before replica ones. In dgm, at least we would have some active items resident in memory.
Hide
Permalink
Liang Guo added a comment -
With CBD-369, I am seeing 38% active_resident_ratio and 12% replica_resident_ratio on both nodes after warmup completes. This is much better than 0% active_resident_ratio on n_1 before. I am closing the bug.
Show
Liang Guo added a comment - With CBD-369, I am seeing 38% active_resident_ratio and 12% replica_resident_ratio on both nodes after warmup completes. This is much better than 0% active_resident_ratio on n_1 before. I am closing the bug.
Hide
Permalink
Liang Guo added a comment -
CBD
http://review.couchbase.org/#change,17415
Show
Liang Guo added a comment - CBD http://review.couchbase.org/#change,17415

People

  • Assignee:
    Liang Guo
    Reporter:
    Liang Guo
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: