Couchbase Server

dispatcher stops persisting items to disk after a successful rebalance due to a race condition in the scheduling ( happens when rebalncing in more than 1 node)

Details

Description

This behavior have been seen while adding 2 or more nodes. steps to reproduce is
1- create a cluster of N nodes
2- add two nodes and rebalance
3- dispatcher does not seem to pick up the items from checkpoints and persist them to disk

because of this behavior we will have to keep all the closed checkpoints in memory and the node might run out of memory after a while depending on the number of mutations that happen in the cluster.


Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Farshid Ghods added a comment -
https://github.com/membase/ep-engine/commit/9a21b04ba1863a855acd61243ad26da2d6879c01

https://github.com/membase/ep-engine/commit/9a21b04ba1863a855acd61243ad26da2d6879c01

Show
Farshid Ghods added a comment - https://github.com/membase/ep-engine/commit/9a21b04ba1863a855acd61243ad26da2d6879c01 https://github.com/membase/ep-engine/commit/9a21b04ba1863a855acd61243ad26da2d6879c01
Hide
Permalink
Farshid Ghods added a comment -
due to this bug membase server will not be able to persist open checkpoints and it will keep them in the memory for a longer time and depending on the number of mutations the client might see temp OOM errors in that node.
Show
Farshid Ghods added a comment - due to this bug membase server will not be able to persist open checkpoints and it will keep them in the memory for a longer time and depending on the number of mutations the client might see temp OOM errors in that node.
Farshid Ghods made changes -
Field Original Value New Value
Assignee Chiyoung Seo [ chiyoung ]
Fix Version/s 1.7.2 [ 10203 ]
Hide
Permalink
Farshid Ghods added a comment -
https://github.com/membase/ep-engine/commit/9a21b04ba1863a855acd61243ad26da2d6879c01

https://github.com/membase/ep-engine/commit/9a21b04ba1863a855acd61243ad26da2d6879c01
Show
Farshid Ghods added a comment - https://github.com/membase/ep-engine/commit/9a21b04ba1863a855acd61243ad26da2d6879c01 https://github.com/membase/ep-engine/commit/9a21b04ba1863a855acd61243ad26da2d6879c01
Farshid Ghods made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Farshid Ghods made changes -
Summary dispatcher stops persisting items after a successful rebalance due to a race condition in the scheduling ( happens when rebalncing in more than 1 node) dispatcher stops persisting items to disk after a successful rebalance due to a race condition in the scheduling ( happens when rebalncing in more than 1 node)
Farshid Ghods made changes -
Labels 1.7.1-release-notes 1.7.1-release-notes 1.7.2-release-notes
Farshid Ghods made changes -
Component/s couchbase-bucket [ 10173 ]
Component/s ep_engine [ 10013 ]
Farshid Ghods made changes -
Status Resolved [ 5 ] Closed [ 6 ]

People

  • Assignee:
    Chiyoung Seo
    Reporter:
    Farshid Ghods
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: