Couchbase Server

Disk not drain (windows)

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 2.0-beta
  • Component/s: couchbase-bucket
  • Security Level: Public
  • Labels:
  • Environment:
    2.0.0-1578, windows

Description

Please refer to the screenshot. For build 1578, disk did not drain.

> Loaded 7M items. Waited for them to be flushed. 3.5M active items stayed in the queue.

> Memory fragmentation was nearly 2.4G. May relate to disk too.

Node: 10.2.2.231 (vms)

Created issue:

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Steve Yen added a comment -
(seemed assigned to wrong folk)
Show
Steve Yen added a comment - (seemed assigned to wrong folk)
Hide
Permalink
Chiyoung Seo added a comment -
Ronnie,

When you have this kind of problems, please grab the diag log file and attach it to the bug. Otherwise, it is difficult for us to debug this issue.
Show
Chiyoung Seo added a comment - Ronnie, When you have this kind of problems, please grab the diag log file and attach it to the bug. Otherwise, it is difficult for us to debug this issue.
Hide
Permalink
Ronnie Sun added a comment -
attached diag logs
Show
Ronnie Sun added a comment - attached diag logs
Hide
Permalink
Ronnie Sun added a comment -
@chiyoung,

Attached diag logs.

The cluster is live, 10.2.2.231.

I'll leave it to you for debugging. Thanks
Show
Ronnie Sun added a comment - @chiyoung, Attached diag logs. The cluster is live, 10.2.2.231. I'll leave it to you for debugging. Thanks
Hide
Permalink
Chiyoung Seo added a comment -
The write dispatcher thread on 10.2.2.231 got an exception and terminated and that's the reason why disk write queue is not drained at all:

[ns_server:info,2012-08-19T14:27:38.036,ns_1@10.2.2.231:ns_port_memcached:ns_port_server:log:169]memcached<0.751.0>: 38 3: RW_Dispatcher: Caught an exception: MUTEX ERROR: Failed to destroy mutex: Invalid argument

Show
Chiyoung Seo added a comment - The write dispatcher thread on 10.2.2.231 got an exception and terminated and that's the reason why disk write queue is not drained at all: [ns_server:info,2012-08-19T14:27:38.036, ns_1@10.2.2.231 :ns_port_memcached:ns_port_server:log:169]memcached<0.751.0>: 38 3: RW_Dispatcher: Caught an exception: MUTEX ERROR: Failed to destroy mutex: Invalid argument
Hide
Permalink
Jin Lim added a comment -
http://review.couchbase.org/#/c/20109/
Show
Jin Lim added a comment - http://review.couchbase.org/#/c/20109/
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ep-engine-2-0 #409 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/409/])
    MB-6323 Remove a race condition in Dispatcher wake method (Revision 5ea867c641d33eb31b57bdb709e8513a3a57152a)

     Result = SUCCESS
Jin Lim :
Files :
* src/bgfetcher.hh
* src/flusher.cc
* src/dispatcher.cc
* src/dispatcher.hh
Show
Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #409 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/409/ ])      MB-6323 Remove a race condition in Dispatcher wake method (Revision 5ea867c641d33eb31b57bdb709e8513a3a57152a)      Result = SUCCESS Jin Lim : Files : * src/bgfetcher.hh * src/flusher.cc * src/dispatcher.cc * src/dispatcher.hh
Hide
Permalink
Chiyoung Seo added a comment -
This happened it again with the latest fix.
Show
Chiyoung Seo added a comment - This happened it again with the latest fix.
Hide
Permalink
Chiyoung Seo added a comment -
Ronnie, please attach the diag file.
Show
Chiyoung Seo added a comment - Ronnie, please attach the diag file.
Hide
Permalink
Ronnie Sun added a comment -
Node 10-2-2-232 went down

Attached diag logs.
Show
Ronnie Sun added a comment - Node 10-2-2-232 went down Attached diag logs.
Hide
Permalink
Jin Lim added a comment -
Please reattache the diag logs. Don't see them. Thanks!
Show
Jin Lim added a comment - Please reattache the diag logs. Don't see them. Thanks!
Hide
Permalink
Jin Lim added a comment -
Got the logs. Both write commit failure and mutex error occurred again. Looking into it.
Show
Jin Lim added a comment - Got the logs. Both write commit failure and mutex error occurred again. Looking into it.
Hide
Permalink
Chiyoung Seo added a comment -
http://review.couchbase.org/#/c/20322/
Show
Chiyoung Seo added a comment - http://review.couchbase.org/#/c/20322/
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ep-engine-2-0 #419 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/419/])
    MB-6323 Ignore EINVAL from pthread_mutex/cond_destroy() (Revision 92e8151fcd8ea25b0799a5f9f59d1459f1aaea48)
MB-6323 attempt to open db with initial rev = 1 if file is not found (Revision aea990ac4f589f785b283d8b98e7d09d6e140884)

     Result = SUCCESS
Jin Lim :
Files :
* src/mutex.cc
* Makefile.am
* src/syncobject.hh

Jin Lim :
Files :
* src/couch-kvstore/couch-kvstore.cc
Show
Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #419 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/419/ ])      MB-6323 Ignore EINVAL from pthread_mutex/cond_destroy() (Revision 92e8151fcd8ea25b0799a5f9f59d1459f1aaea48) MB-6323 attempt to open db with initial rev = 1 if file is not found (Revision aea990ac4f589f785b283d8b98e7d09d6e140884)      Result = SUCCESS Jin Lim : Files : * src/mutex.cc * Makefile.am * src/syncobject.hh Jin Lim : Files : * src/couch-kvstore/couch-kvstore.cc
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ep-engine-2-0 #421 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/421/])
    MB-6323 return TMPFAIL for failed bgfetch instead of assert (Revision 4bf1696c646d8a7706d29de86d6814b0ff1004bf)

     Result = SUCCESS
Jin Lim :
Files :
* src/ep.cc
Show
Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #421 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/421/ ])      MB-6323 return TMPFAIL for failed bgfetch instead of assert (Revision 4bf1696c646d8a7706d29de86d6814b0ff1004bf)      Result = SUCCESS Jin Lim : Files : * src/ep.cc

People

  • Assignee:
    Jin Lim
    Reporter:
    Ronnie Sun
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: