Couchbase Server

rebalance-in failed with beam.smp resident set size ~20GB

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Critical Critical
  • Resolution: Cannot Reproduce
  • Affects Version/s: 2.0
  • Fix Version/s: 2.0
  • Component/s: ns_server
  • Security Level: Public
  • Labels:
  • Environment:
    Physical machines, CentOS 5.8, 32GB RAM, 4 cores, 2 x SSD
    Build 1956
    10M x 2KB items, 1 ddoc, 8 views, 4 -> 3

Description

I've never seen such a mess in logs, but apart from all issues memory consumption looks really weird:

# ps -eo rss,vsize,args | grep beam
19867592 27554248 /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -S 16:16 -sbt u -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progname erl -- -home /opt/couchbase -- -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -noshell -noinput -noshell -noinput -run ns_bootstrap -- -name ns_1@10.2.1.66 -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" -ns_server pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server nodefile "/opt/couchbase/var/lib/couchbase/couchbase-server.node" -ns_server cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie"

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Pavel Paulau added a comment -
Diags:
https://s3.amazonaws.com/bugdb/jira/MB-7247/740e62ca/10.2.1.65.diags.zip
https://s3.amazonaws.com/bugdb/jira/MB-7247/740e62ca/10.2.1.66.diags.zip
https://s3.amazonaws.com/bugdb/jira/MB-7247/740e62ca/10.2.1.67.diags.zip
https://s3.amazonaws.com/bugdb/jira/MB-7247/740e62ca/10.2.1.68.diags.zip
Show
Pavel Paulau added a comment - Diags: https://s3.amazonaws.com/bugdb/jira/MB-7247/740e62ca/10.2.1.65.diags.zip https://s3.amazonaws.com/bugdb/jira/MB-7247/740e62ca/10.2.1.66.diags.zip https://s3.amazonaws.com/bugdb/jira/MB-7247/740e62ca/10.2.1.67.diags.zip https://s3.amazonaws.com/bugdb/jira/MB-7247/740e62ca/10.2.1.68.diags.zip
Hide
Permalink
Pavel Paulau added a comment -
atop:
https://s3.amazonaws.com/bugdb/jira/MB-7247/740e62ca/10.2.1.66.atop
Show
Pavel Paulau added a comment - atop: https://s3.amazonaws.com/bugdb/jira/MB-7247/740e62ca/10.2.1.66.atop
Hide
Permalink
Farshid Ghods added a comment -
by any chance did you run a query without a limit against this cluster ?

symptoms are similar to http://www.couchbase.com/issues/browse/MB-7199
Show
Farshid Ghods added a comment - by any chance did you run a query without a limit against this cluster ? symptoms are similar to http://www.couchbase.com/issues/browse/MB-7199
Hide
Permalink
Pavel Paulau added a comment -
Nope, all our queries have limit (10 or 30).

This is regular rebalance test...
Show
Pavel Paulau added a comment - Nope, all our queries have limit (10 or 30). This is regular rebalance test...
Hide
Permalink
Farshid Ghods added a comment -
it would be good if you can get the core dumps of this process so that alk/filipe can look at the file and find out how all that memory is used
Show
Farshid Ghods added a comment - it would be good if you can get the core dumps of this process so that alk/filipe can look at the file and find out how all that memory is used
Hide
Permalink
Pavel Paulau added a comment -
Too late.
Show
Pavel Paulau added a comment - Too late.
Hide
Permalink
Aleksey Kondratenko added a comment -
Logs are useless here. Logs are rotated past interesting point when problem started about 15:30. From atop I can only see that initial bump in process size appears to coincide with massive increase in dirty pages count.

Erlang's crash dump would help a lot here.
Show
Aleksey Kondratenko added a comment - Logs are useless here. Logs are rotated past interesting point when problem started about 15:30. From atop I can only see that initial bump in process size appears to coincide with massive increase in dirty pages count. Erlang's crash dump would help a lot here.
Hide
Permalink
Steve Yen added a comment -
from bug-scrub mtg -- assigning back to @pavel - please try to capture more info the next time you see this.
Show
Steve Yen added a comment - from bug-scrub mtg -- assigning back to @pavel - please try to capture more info the next time you see this.
Hide
Permalink
Steve Yen added a comment -
from perf-sync mtg, moving back to 2.0 to help track this better.

build 2.0.0-1954 had the 16:16 scheduler threads change, so this is the first time Pavel's tested 16:16 configuration.

Pavel, can you re-run this experiment and see how reproducible this is?
Thanks,
Steve

Show
Steve Yen added a comment - from perf-sync mtg, moving back to 2.0 to help track this better. build 2.0.0-1954 had the 16:16 scheduler threads change, so this is the first time Pavel's tested 16:16 configuration. Pavel, can you re-run this experiment and see how reproducible this is? Thanks, Steve
Hide
Permalink
Steve Yen added a comment -
resolving this for now as next build 1966/1967 didn't have this issue; let's re-open if it occurs again and add the usual collect-info.

Show
Steve Yen added a comment - resolving this for now as next build 1966/1967 didn't have this issue; let's re-open if it occurs again and add the usual collect-info.

People

  • Assignee:
    Pavel Paulau
    Reporter:
    Pavel Paulau
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: