Couchbase Server

Disk not drain (windows)

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 2.0-beta
  • Component/s: couchbase-bucket
  • Security Level: Public
  • Labels:
  • Environment:
    2.0.0-1578, windows

Description

Please refer to the screenshot. For build 1578, disk did not drain.

> Loaded 7M items. Waited for them to be flushed. 3.5M active items stayed in the queue.

> Memory fragmentation was nearly 2.4G. May relate to disk too.

Node: 10.2.2.231 (vms)

Created issue:

Activity

Ascending order - Click to sort in descending order
Ronnie Sun made changes -
Field Original Value New Value
Labels pblock
Hide
Permalink
Steve Yen added a comment -
(seemed assigned to wrong folk)
Show
Steve Yen added a comment - (seemed assigned to wrong folk)
Steve Yen made changes -
Assignee Farshid Ghods [ farshid ] Chiyoung Seo [ chiyoung ]
Steve Yen made changes -
Environment 2.0.0-1578 2.0.0-1578, windows
Hide
Permalink
Chiyoung Seo added a comment -
Ronnie,

When you have this kind of problems, please grab the diag log file and attach it to the bug. Otherwise, it is difficult for us to debug this issue.
Show
Chiyoung Seo added a comment - Ronnie, When you have this kind of problems, please grab the diag log file and attach it to the bug. Otherwise, it is difficult for us to debug this issue.
Chiyoung Seo made changes -
Assignee Chiyoung Seo [ chiyoung ] Ronnie Sun [ ronnie ]
Steve Yen made changes -
Priority Major [ 3 ] Blocker [ 1 ]
Hide
Permalink
Ronnie Sun added a comment -
attached diag logs
Show
Ronnie Sun added a comment - attached diag logs
Ronnie Sun made changes -
Attachment Archive.zip [ 14509 ]
Hide
Permalink
Ronnie Sun added a comment -
@chiyoung,

Attached diag logs.

The cluster is live, 10.2.2.231.

I'll leave it to you for debugging. Thanks
Show
Ronnie Sun added a comment - @chiyoung, Attached diag logs. The cluster is live, 10.2.2.231. I'll leave it to you for debugging. Thanks
Ronnie Sun made changes -
Assignee Ronnie Sun [ ronnie ] Chiyoung Seo [ chiyoung ]
Hide
Permalink
Chiyoung Seo added a comment -
The write dispatcher thread on 10.2.2.231 got an exception and terminated and that's the reason why disk write queue is not drained at all:

[ns_server:info,2012-08-19T14:27:38.036,ns_1@10.2.2.231:ns_port_memcached:ns_port_server:log:169]memcached<0.751.0>: 38 3: RW_Dispatcher: Caught an exception: MUTEX ERROR: Failed to destroy mutex: Invalid argument

Show
Chiyoung Seo added a comment - The write dispatcher thread on 10.2.2.231 got an exception and terminated and that's the reason why disk write queue is not drained at all: [ns_server:info,2012-08-19T14:27:38.036, ns_1@10.2.2.231 :ns_port_memcached:ns_port_server:log:169]memcached<0.751.0>: 38 3: RW_Dispatcher: Caught an exception: MUTEX ERROR: Failed to destroy mutex: Invalid argument
Chiyoung Seo made changes -
Assignee Chiyoung Seo [ chiyoung ] Jin Lim [ jin ]
Chiyoung Seo made changes -
Sprint Status Current Sprint
Jin Lim made changes -
Status Open [ 1 ] In Progress [ 3 ]
Jin Lim made changes -
Status In Progress [ 3 ] Open [ 1 ]
Hide
Permalink
Jin Lim added a comment -
http://review.couchbase.org/#/c/20109/
Show
Jin Lim added a comment - http://review.couchbase.org/#/c/20109/
Jin Lim made changes -
Status Open [ 1 ] Resolved [ 5 ]
Fix Version/s 2.0 [ 10114 ]
Resolution Fixed [ 1 ]
Jin Lim made changes -
Comment [ A fix is pushed for review, http://review.couchbase.org/#/c/20109/ ]
Chiyoung Seo made changes -
Sprint Status Current Sprint
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ep-engine-2-0 #409 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/409/])
    MB-6323 Remove a race condition in Dispatcher wake method (Revision 5ea867c641d33eb31b57bdb709e8513a3a57152a)

     Result = SUCCESS
Jin Lim :
Files :
* src/bgfetcher.hh
* src/flusher.cc
* src/dispatcher.cc
* src/dispatcher.hh
Show
Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #409 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/409/ ])      MB-6323 Remove a race condition in Dispatcher wake method (Revision 5ea867c641d33eb31b57bdb709e8513a3a57152a)      Result = SUCCESS Jin Lim : Files : * src/bgfetcher.hh * src/flusher.cc * src/dispatcher.cc * src/dispatcher.hh
Hide
Permalink
Chiyoung Seo added a comment -
This happened it again with the latest fix.
Show
Chiyoung Seo added a comment - This happened it again with the latest fix.
Chiyoung Seo made changes -
Resolution Fixed [ 1 ]
Status Resolved [ 5 ] Reopened [ 4 ]
Sprint Status Current Sprint
Hide
Permalink
Chiyoung Seo added a comment -
Ronnie, please attach the diag file.
Show
Chiyoung Seo added a comment - Ronnie, please attach the diag file.
Hide
Permalink
Ronnie Sun added a comment -
Node 10-2-2-232 went down

Attached diag logs.
Show
Ronnie Sun added a comment - Node 10-2-2-232 went down Attached diag logs.
Ronnie Sun made changes -
Attachment ns_diag.zip [ 14588 ]
Hide
Permalink
Jin Lim added a comment -
Please reattache the diag logs. Don't see them. Thanks!
Show
Jin Lim added a comment - Please reattache the diag logs. Don't see them. Thanks!
Hide
Permalink
Jin Lim added a comment -
Got the logs. Both write commit failure and mutex error occurred again. Looking into it.
Show
Jin Lim added a comment - Got the logs. Both write commit failure and mutex error occurred again. Looking into it.
Peter Wansch made changes -
Fix Version/s 2.0-beta [ 10113 ]
Fix Version/s 2.0 [ 10114 ]
Component/s couchbase-bucket [ 10173 ]
Hide
Permalink
Chiyoung Seo added a comment -
http://review.couchbase.org/#/c/20322/
Show
Chiyoung Seo added a comment - http://review.couchbase.org/#/c/20322/
Chiyoung Seo made changes -
Status Reopened [ 4 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Chiyoung Seo made changes -
Sprint Status Current Sprint
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ep-engine-2-0 #419 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/419/])
    MB-6323 Ignore EINVAL from pthread_mutex/cond_destroy() (Revision 92e8151fcd8ea25b0799a5f9f59d1459f1aaea48)
MB-6323 attempt to open db with initial rev = 1 if file is not found (Revision aea990ac4f589f785b283d8b98e7d09d6e140884)

     Result = SUCCESS
Jin Lim :
Files :
* src/mutex.cc
* Makefile.am
* src/syncobject.hh

Jin Lim :
Files :
* src/couch-kvstore/couch-kvstore.cc
Show
Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #419 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/419/ ])      MB-6323 Ignore EINVAL from pthread_mutex/cond_destroy() (Revision 92e8151fcd8ea25b0799a5f9f59d1459f1aaea48) MB-6323 attempt to open db with initial rev = 1 if file is not found (Revision aea990ac4f589f785b283d8b98e7d09d6e140884)      Result = SUCCESS Jin Lim : Files : * src/mutex.cc * Makefile.am * src/syncobject.hh Jin Lim : Files : * src/couch-kvstore/couch-kvstore.cc
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ep-engine-2-0 #421 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/421/])
    MB-6323 return TMPFAIL for failed bgfetch instead of assert (Revision 4bf1696c646d8a7706d29de86d6814b0ff1004bf)

     Result = SUCCESS
Jin Lim :
Files :
* src/ep.cc
Show
Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #421 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/421/ ])      MB-6323 return TMPFAIL for failed bgfetch instead of assert (Revision 4bf1696c646d8a7706d29de86d6814b0ff1004bf)      Result = SUCCESS Jin Lim : Files : * src/ep.cc

People

  • Assignee:
    Jin Lim
    Reporter:
    Ronnie Sun
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: