Couchbase Server

After two nodes crashed, curr_items remained 0 after warmup for extended period of time

Details

  • Type: Bug Bug
  • Status: Reopened Reopened
  • Priority: Critical Critical
  • Resolution: Unresolved
  • Affects Version/s: 1.7 GA, 1.8.1
  • Fix Version/s: 2.1
  • Component/s: ns_server
  • Security Level: Public
  • Labels:

Description

we had two nodes crash at Tribal Crossing, possibly related to a disk space issue, but I don't think so.

After they crashed, the nodes warmed up relatively quickly, but immediately "discarded" their items. I say that because I see that they warmed up ~10m items, but the current item counts were both 0.

I tried shutting down the service and had to kill memcached manually (kill -9). Restarting it went through the same process of warming up and then nothing.

While I was looking around, I left it sit for a little while and magically all of the items came back. I seem to recall this bug previously where a node wouldn't be told to be active until all the nodes in the cluster were active...and it got into trouble when not all of the nodes restarted.

Diags for all nodes will be attached

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Perry Krug added a comment -
Full set of logs at \\corp-fs1\export_support_cases\bug_4030
Show
Perry Krug added a comment - Full set of logs at \\corp-fs1\export_support_cases\bug_4030
Hide
Permalink
Aleksey Kondratenko added a comment -
It _is_ ns_server issue caused by janitor needing all nodes to be up for vbuckets activation. We planned fix for 1.8.1 (now 1.8.2)
Show
Aleksey Kondratenko added a comment - It _is_ ns_server issue caused by janitor needing all nodes to be up for vbuckets activation. We planned fix for 1.8.1 (now 1.8.2)
Hide
Permalink
Aleksey Kondratenko added a comment -
Fix would land as part of fast warmup integration
Show
Aleksey Kondratenko added a comment - Fix would land as part of fast warmup integration
Hide
Permalink
Perry Krug added a comment -
Peter, can we get a second look at this one? We've seen this before, and the problem is that the janitor did not run until all nodes had joined the cluster and warmed up. I'm not sure we've fixed that already...
Show
Perry Krug added a comment - Peter, can we get a second look at this one? We've seen this before, and the problem is that the janitor did not run until all nodes had joined the cluster and warmed up. I'm not sure we've fixed that already...
Hide
Permalink
Aleksey Kondratenko added a comment -
Latest 2.0 will mark nodes as green and enable memcached traffic when all of them are up. So easy part is done.

Partial janitor (i.e. enabling traffic for some nodes when others are still down/warming up) is something that will unlikely be done soon
Show
Aleksey Kondratenko added a comment - Latest 2.0 will mark nodes as green and enable memcached traffic when all of them are up. So easy part is done. Partial janitor (i.e. enabling traffic for some nodes when others are still down/warming up) is something that will unlikely be done soon
Hide
Permalink
Perry Krug added a comment -
Thanks Alk...what's the difference in behavior (in this area) between 1.x and 2.0? It "sounds" like they're the same, no?

And this bug should still remain open until we fix the primary issue which is the partial janitor...correct?
Show
Perry Krug added a comment - Thanks Alk...what's the difference in behavior (in this area) between 1.x and 2.0? It "sounds" like they're the same, no? And this bug should still remain open until we fix the primary issue which is the partial janitor...correct?
Hide
Permalink
Aleksey Kondratenko added a comment -
1.8.1 will show node as green when ep-engine thinks it's warmed up. But confusingly it'll not be really ready. All vbuckets will be in state dead and curr_items will be 0.

2.0 fixes this confusion. Node is marked green when it's actually warmed up from user's perspective. I.e. right vbucket states are set and it'll serve clients traffic.

2.0 is still very conservative about only making vbucket state changes when all nodes are up and warmed up. Thats "impartial" janitor. Whether it's a bug or "lack of feature" is debatable. But I think main concern that users are confused by green-ness of nodes is resolved.
Show
Aleksey Kondratenko added a comment - 1.8.1 will show node as green when ep-engine thinks it's warmed up. But confusingly it'll not be really ready. All vbuckets will be in state dead and curr_items will be 0. 2.0 fixes this confusion. Node is marked green when it's actually warmed up from user's perspective. I.e. right vbucket states are set and it'll serve clients traffic. 2.0 is still very conservative about only making vbucket state changes when all nodes are up and warmed up. Thats "impartial" janitor. Whether it's a bug or "lack of feature" is debatable. But I think main concern that users are confused by green-ness of nodes is resolved.
Hide
Permalink
Aleksey Kondratenko added a comment -
Closing as fixed. We'll get to partial janitor some day in future which is feature we lack today, not bug we have IMHO
Show
Aleksey Kondratenko added a comment - Closing as fixed. We'll get to partial janitor some day in future which is feature we lack today, not bug we have IMHO
Hide
Permalink
Perry Krug added a comment -
Reopening this for the need for partial janitor. Recent customer had multiple nodes need to be hard-booted and none returned to service until all were warmed up
Show
Perry Krug added a comment - Reopening this for the need for partial janitor. Recent customer had multiple nodes need to be hard-booted and none returned to service until all were warmed up
Hide
Permalink
Steve Yen added a comment -
bug-scrub: moving out of 2.0, as this looks like a feature req.
Show
Steve Yen added a comment - bug-scrub: moving out of 2.0, as this looks like a feature req.
Hide
Permalink
Farshid Ghods added a comment -
in system testing we have noticed many times that if multiple nodes crash until all nodes are warmed up node status for those that are already warmed up appears as yellow.


user won't be able to understand which node has successfully warmed up from the console and if one node is actually not recovering or not warm up in a reasonable time they have to figure it out some other way ( cbstats ... )

another issue with this is that user won't be able to perform a fail over for 1 node even though N-1 nodes has warmed up already.

i am not sure if fixing this bug will impact cluster-restore functionality but something important to fix or suggest a workaround to the user ( by workaround i mean a documented , tested and supported set of commands )
Show
Farshid Ghods added a comment - in system testing we have noticed many times that if multiple nodes crash until all nodes are warmed up node status for those that are already warmed up appears as yellow. user won't be able to understand which node has successfully warmed up from the console and if one node is actually not recovering or not warm up in a reasonable time they have to figure it out some other way ( cbstats ... ) another issue with this is that user won't be able to perform a fail over for 1 node even though N-1 nodes has warmed up already. i am not sure if fixing this bug will impact cluster-restore functionality but something important to fix or suggest a workaround to the user ( by workaround i mean a documented , tested and supported set of commands )
Hide
Permalink
Mike Wiederhold added a comment -
Comments say this is an ns_server issue so I am removing couchbase-bucket from affected components. Please re-add if there is a couchbase-bucket task for this issue.
Show
Mike Wiederhold added a comment - Comments say this is an ns_server issue so I am removing couchbase-bucket from affected components. Please re-add if there is a couchbase-bucket task for this issue.

People

  • Assignee:
    Aleksey Kondratenko
    Reporter:
    Perry Krug
Vote (0)
Watch (3)

Dates

  • Created:
    Updated: