Couchbase 4.1.0-5005 - Unavailable for Accessing Documents

Couchbase Server
, ,
1

Hello -

Let me start with our setup. We have cluster with 3 nodes, details below:
Total RAM in cluster: 137 GB
No. of Buckets: 2 (1-Couchbase bucket type, 1-memcache bucket type)
Cache Metadata for Couchbase bucket type is Value Ejection
No. of replica’s: 2
Auto Compaction Settings: Couchbase Default (30% on database, view, index)
Index: We have primary index using GSI on couchbase bucket

We have recently started facing an issue on production average 6 times per day. Couchbase not responding to queries. We are accessing Couchbase using Java SDK v2.3.4.

We perform very basic operation like insert, getAndTouch, remove, and replace. SDK env build like-

CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder()
                        .viewEndpoints(0)
                        .searchEndpoints(0)
                        .queryEndpoints(0)
                        .kvTimeout(10000)
                        .build();

We are trying figure out, why couchbase becomes non-responsive?

Java SDK Exception: Happens a lot

rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: com.couchbase.client.core.message.kv.ReplaceResponse.class
        at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:109) ~[rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:73) ~[rxjava-1.1.8.jar:1.1.8]
        ... 17 common frames omitted
Wrapped by: com.couchbase.client.java.error.TemporaryFailureException: null
        at com.couchbase.client.java.CouchbaseAsyncBucket$19.call(CouchbaseAsyncBucket.java:613) ~[java-client-2.3.4.jar:na]
        at com.couchbase.client.java.CouchbaseAsyncBucket$19.call(CouchbaseAsyncBucket.java:592) ~[java-client-2.3.4.jar:na]
        at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69) ~[rxjava-1.1.8.jar:1.1.8]
        at rx.observers.Subscribers$5.onNext(Subscribers.java:235) ~[rxjava-1.1.8.jar:1.1.8]
        at rx.internal.producers.SingleProducer.request(SingleProducer.java:65) ~[rxjava-1.1.8.jar:1.1.8]
        at rx.Subscriber.setProducer(Subscriber.java:211) ~[rxjava-1.1.8.jar:1.1.8]
        at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102) ~[rxjava-1.1.8.jar:1.1.8]
        at rx.Subscriber.setProducer(Subscriber.java:205) ~[rxjava-1.1.8.jar:1.1.8]
        at rx.subjects.AsyncSubject.onCompleted(AsyncSubject.java:106) ~[rxjava-1.1.8.jar:1.1.8]
        at com.couchbase.client.core.endpoint.AbstractGenericHandler.completeResponse(AbstractGenericHandler.java:349) ~[core-io-1.3.4.jar:na]
        at com.couchbase.client.core.endpoint.AbstractGenericHandler.access$000(AbstractGenericHandler.java:67) ~[core-io-1.3.4.jar:na]
        at com.couchbase.client.core.endpoint.AbstractGenericHandler$1.call(AbstractGenericHandler.java:367) ~[core-io-1.3.4.jar:na]
        at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) ~[rxjava-1.1.8.jar:1.1.8]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_79]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_79]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) ~[na:1.7.0_79]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) ~[na:1.7.0_79]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_79]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_79]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_79]

System Logs Details:

info.log

[ns_server:info,2017-02-28T11:18:04.463-08:00,ns_1@xxxxxxxxxxxx:<0.12891.8039>:compaction_new_daemon:spawn_scheduled_kv_compactor:467]Start compaction of vbuckets for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2017-02-28T11:18:04.677-08:00,ns_1@xxxxxxxxxxxx:<0.17057.8039>:compaction_new_daemon:spawn_scheduled_views_compactor:493]Start compaction of indexes for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2017-02-28T11:18:15.881-08:00,ns_1@xxxxxxxxxxxx:ns_config_rep<0.1532.0>:ns_config_rep:do_pull:347]Pulling config from: 'ns_1@xxxxxxxxxxxx'

[ns_server:info,2017-02-28T11:18:26.981-08:00,ns_1@xxxxxxxxxxxx:<0.16794.8039>:compaction_new_daemon:spawn_master_db_compactor:791]Start compaction of master db for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2017-02-28T11:18:34.470-08:00,ns_1@xxxxxxxxxxxx:<0.15212.8039>:compaction_new_daemon:spawn_scheduled_kv_compactor:467]Start compaction of vbuckets for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2017-02-28T11:18:34.685-08:00,ns_1@xxxxxxxxxxxx:<0.17259.8039>:compaction_new_daemon:spawn_scheduled_views_compactor:493]Start compaction of indexes for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2017-02-28T11:18:56.985-08:00,ns_1@xxxxxxxxxxxx:<0.15873.8039>:compaction_new_daemon:spawn_master_db_compactor:791]Start compaction of master db for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2017-02-28T11:18:59.912-08:00,ns_1@xxxxxxxxxxxx:ns_config_rep<0.1532.0>:ns_config_rep:do_pull:347]Pulling config from: 'ns_1@xxxxxxxxxxxx'

[ns_server:info,2017-02-28T11:19:04.478-08:00,ns_1@xxxxxxxxxxxx:<0.17436.8039>:compaction_new_daemon:spawn_scheduled_kv_compactor:467]Start compaction of vbuckets for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2017-02-28T11:19:04.690-08:00,ns_1@xxxxxxxxxxxx:<0.19348.8039>:compaction_new_daemon:spawn_scheduled_views_compactor:493]Start compaction of indexes for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2017-02-28T11:19:14.483-08:00,ns_1@xxxxxxxxxxxx:ns_config_rep<0.1532.0>:ns_config_rep:do_pull:347]Pulling config from: 'ns_1@xxxxxxxxxxxx'

[ns_server:info,2017-02-28T11:19:26.988-08:00,ns_1@xxxxxxxxxxxx:<0.15893.8039>:compaction_new_daemon:spawn_master_db_compactor:791]Start compaction of master db for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2017-02-28T11:19:34.484-08:00,ns_1@xxxxxxxxxxxx:<0.17328.8039>:compaction_new_daemon:spawn_scheduled_kv_compactor:467]Start compaction of vbuckets for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:info,2017-02-28T11:19:34.696-08:00,ns_1@xxxxxxxxxxxx:<0.18076.8039>:compaction_new_daemon:spawn_scheduled_views_compactor:493]Start compaction of indexes for bucket cas_tgt with config:

babysitter.log

[ns_server:info,2017-02-28T08:46:35.244-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:35.033911-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 3302 ms

[ns_server:info,2017-02-28T08:46:36.265-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:36.054575-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 3006 ms

[ns_server:info,2017-02-28T08:46:38.099-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:37.888702-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 3872 ms
memcached<0.75.0>: 2017-02-28T08:46:37.889043-08:00 WARNING 125: Slow COMPACT_DB operation on connection (127.0.0.1:32931 => 127.0.0.1:11209): 3921 ms

[ns_server:info,2017-02-28T08:46:38.680-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:38.469591-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 3430 ms

[ns_server:info,2017-02-28T08:46:40.192-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:39.981970-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 3924 ms

[ns_server:info,2017-02-28T08:46:41.389-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:41.178905-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 3287 ms

[ns_server:info,2017-02-28T08:46:43.157-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:42.946088-08:00 WARNING 125: Slow COMPACT_DB operation on connection (127.0.0.1:32931 => 127.0.0.1:11209): 5054 ms
memcached<0.75.0>: 2017-02-28T08:46:42.948599-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 4476 ms

[ns_server:info,2017-02-28T08:46:44.720-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:44.509188-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 4523 ms

[ns_server:info,2017-02-28T08:46:46.548-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:46.337256-08:00 WARNING 78: Slow GAT operation on connection (10.0.12.19:58872 => 10.0.11.34:11210): 1094 ms
memcached<0.75.0>: 2017-02-28T08:46:46.352616-08:00 WARNING 84: Slow GAT operation on connection (10.0.11.20:34691 => 10.0.11.34:11210): 1005 ms
memcached<0.75.0>: 2017-02-28T08:46:46.359100-08:00 WARNING 74: Slow GAT operation on connection (10.0.12.19:58832 => 10.0.11.34:11210): 1075 ms

[ns_server:info,2017-02-28T08:46:47.498-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:47.287781-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 4337 ms
memcached<0.75.0>: 2017-02-28T08:46:47.289129-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 6108 ms
memcached<0.75.0>: 2017-02-28T08:46:47.289798-08:00 WARNING 125: Slow COMPACT_DB operation on connection (127.0.0.1:32931 => 127.0.0.1:11209): 4340 ms

[ns_server:info,2017-02-28T08:46:49.092-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:48.881786-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 4368 ms

[ns_server:info,2017-02-28T08:46:51.349-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:51.138972-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 3721 ms
memcached<0.75.0>: 2017-02-28T08:46:51.146086-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 3729 ms

[ns_server:info,2017-02-28T08:46:52.337-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:52.126317-08:00 WARNING 125: Slow COMPACT_DB operation on connection (127.0.0.1:32931 => 127.0.0.1:11209): 4712 ms

[ns_server:info,2017-02-28T08:46:53.647-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:53.436322-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 4551 ms

[ns_server:info,2017-02-28T08:46:55.352-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:55.141433-08:00 WARNING 76: Slow COMPACT_DB operation on connection (127.0.0.1:34445 => 127.0.0.1:11209): 3990 ms
memcached<0.75.0>: 2017-02-28T08:46:55.142311-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 3991 ms

[ns_server:info,2017-02-28T08:46:56.317-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:56.106211-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 3969 ms

[ns_server:info,2017-02-28T08:46:57.307-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:57.097034-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 3658 ms

[ns_server:info,2017-02-28T08:46:58.235-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:58.024770-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 2879 ms

[ns_server:info,2017-02-28T08:47:00.206-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:46:59.996053-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 3887 ms
memcached<0.75.0>: 2017-02-28T08:46:59.999696-08:00 WARNING 76: Slow COMPACT_DB operation on connection (127.0.0.1:34445 => 127.0.0.1:11209): 4853 ms

[ns_server:info,2017-02-28T08:47:01.399-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:01.188709-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 4089 ms

[ns_server:info,2017-02-28T08:47:02.760-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:02.550075-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 4521 ms

[ns_server:info,2017-02-28T08:47:04.853-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:04.642258-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 4639 ms
memcached<0.75.0>: 2017-02-28T08:47:04.698065-08:00 WARNING 76: Slow COMPACT_DB operation on connection (127.0.0.1:34445 => 127.0.0.1:11209): 4697 ms

[ns_server:info,2017-02-28T08:47:06.338-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:06.127775-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 4936 ms

[ns_server:info,2017-02-28T08:47:07.307-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:07.096158-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 4542 ms

[ns_server:info,2017-02-28T08:47:09.348-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:09.137562-08:00 WARNING 76: Slow COMPACT_DB operation on connection (127.0.0.1:34445 => 127.0.0.1:11209): 4436 ms
memcached<0.75.0>: 2017-02-28T08:47:09.141259-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 4496 ms

[ns_server:info,2017-02-28T08:47:10.332-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:10.121702-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 3990 ms

[ns_server:info,2017-02-28T08:47:10.997-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:10.787110-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 3688 ms

[ns_server:info,2017-02-28T08:47:13.334-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:13.123259-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 3979 ms
memcached<0.75.0>: 2017-02-28T08:47:13.125767-08:00 WARNING 76: Slow COMPACT_DB operation on connection (127.0.0.1:34445 => 127.0.0.1:11209): 3985 ms

[ns_server:info,2017-02-28T08:47:14.553-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:14.342363-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 4217 ms

[ns_server:info,2017-02-28T08:47:15.718-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:15.507511-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 4717 ms

[ns_server:info,2017-02-28T08:47:17.818-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:17.607928-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 4482 ms
memcached<0.75.0>: 2017-02-28T08:47:17.613103-08:00 WARNING 76: Slow COMPACT_DB operation on connection (127.0.0.1:34445 => 127.0.0.1:11209): 4485 ms

[ns_server:info,2017-02-28T08:47:19.611-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:19.400601-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 3890 ms
memcached<0.75.0>: 2017-02-28T08:47:19.401276-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 5046 ms

[ns_server:info,2017-02-28T08:47:21.437-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:21.226561-08:00 WARNING 76: Slow COMPACT_DB operation on connection (127.0.0.1:34445 => 127.0.0.1:11209): 3611 ms
memcached<0.75.0>: 2017-02-28T08:47:21.230533-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 3619 ms

[ns_server:info,2017-02-28T08:47:22.315-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:22.105014-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 2701 ms

[ns_server:info,2017-02-28T08:47:23.027-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:22.816711-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 3413 ms

[ns_server:info,2017-02-28T08:47:24.525-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:24.314897-08:00 WARNING 76: Slow COMPACT_DB operation on connection (127.0.0.1:34445 => 127.0.0.1:11209): 3085 ms
memcached<0.75.0>: 2017-02-28T08:47:24.317219-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 3083 ms

[ns_server:info,2017-02-28T08:47:26.586-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:26.375888-08:00 WARNING 81: Slow COMPACT_DB operation on connection (127.0.0.1:40203 => 127.0.0.1:11209): 3556 ms
memcached<0.75.0>: 2017-02-28T08:47:26.376666-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 4269 ms

[ns_server:info,2017-02-28T08:47:28.652-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:28.441802-08:00 WARNING 76: Slow COMPACT_DB operation on connection (127.0.0.1:34445 => 127.0.0.1:11209): 4124 ms

[ns_server:info,2017-02-28T08:47:28.898-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:28.687798-08:00 WARNING 77: Slow COMPACT_DB operation on connection (127.0.0.1:36648 => 127.0.0.1:11209): 4368 ms

[ns_server:info,2017-02-28T08:47:30.487-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T08:47:30.276462-08:00 WARNING 158: Slow COMPACT_DB operation on connection (127.0.0.1:52365 => 127.0.0.1:11209): 3897 ms

[ns_server:info,2017-02-28T09:17:02.049-08:00,babysitter_of_ns_1@127.0.0.1:<0.75.0>:ns_port_server:log:210]memcached<0.75.0>: 2017-02-28T09:17:01.760541-08:00 WARNING 114: Slow REPLACE operation on connection (10.0.10.20:49857 => 10.0.11.34:11210): 18446744073618 ms

debug.log

[ns_server:debug,2017-02-28T11:54:34.212-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_scheduler_message:1251]Starting compaction (compact_views) for the following buckets:
[<<"cas_tgt">>]
[ns_server:info,2017-02-28T11:54:34.217-08:00,ns_1@xxxxxxxxxxxx:<0.14271.8040>:compaction_new_daemon:spawn_scheduled_views_compactor:493]Start compaction of indexes for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2017-02-28T11:54:34.217-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_compactors_exit:1292]Finished compaction iteration.
[ns_server:debug,2017-02-28T11:54:34.217-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_scheduler:schedule_next:60]Finished compaction for compact_views too soon. Next run will be in 30s
[ns_server:debug,2017-02-28T11:54:34.947-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_scheduler_message:1251]Starting compaction (compact_kv) for the following buckets:
[<<"cas_tgt">>]
[ns_server:info,2017-02-28T11:54:34.948-08:00,ns_1@xxxxxxxxxxxx:<0.16830.8040>:compaction_new_daemon:spawn_scheduled_kv_compactor:467]Start compaction of vbuckets for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2017-02-28T11:54:34.952-08:00,ns_1@xxxxxxxxxxxx:<0.14733.8040>:compaction_new_daemon:bucket_needs_compaction:913]`cas_tgt` data size is 94285391280, disk size is 122901629952
[ns_server:debug,2017-02-28T11:54:34.952-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_compactors_exit:1292]Finished compaction iteration.
[ns_server:debug,2017-02-28T11:54:34.952-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_scheduler:schedule_next:60]Finished compaction for compact_kv too soon. Next run will be in 30s
[ns_server:info,2017-02-28T11:54:40.297-08:00,ns_1@xxxxxxxxxxxx:ns_config_rep<0.1532.0>:ns_config_rep:do_pull:347]Pulling config from: 'ns_1@xxxxxxxxxxxx'

[ns_server:debug,2017-02-28T11:54:56.251-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_scheduler_message:1251]Starting compaction (compact_master) for the following buckets:
[<<"cas_tgt">>]
[ns_server:info,2017-02-28T11:54:56.252-08:00,ns_1@xxxxxxxxxxxx:<0.16644.8040>:compaction_new_daemon:spawn_master_db_compactor:791]Start compaction of master db for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2017-02-28T11:54:56.253-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_compactors_exit:1292]Finished compaction iteration.
[ns_server:debug,2017-02-28T11:54:56.253-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_scheduler:schedule_next:60]Finished compaction for compact_master too soon. Next run will be in 30s
[ns_server:debug,2017-02-28T11:55:04.218-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_scheduler_message:1251]Starting compaction (compact_views) for the following buckets:
[<<"cas_tgt">>]
[ns_server:info,2017-02-28T11:55:04.222-08:00,ns_1@xxxxxxxxxxxx:<0.14811.8040>:compaction_new_daemon:spawn_scheduled_views_compactor:493]Start compaction of indexes for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2017-02-28T11:55:04.223-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_compactors_exit:1292]Finished compaction iteration.
[ns_server:debug,2017-02-28T11:55:04.223-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_scheduler:schedule_next:60]Finished compaction for compact_views too soon. Next run will be in 30s
[ns_server:debug,2017-02-28T11:55:04.953-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_scheduler_message:1251]Starting compaction (compact_kv) for the following buckets:
[<<"cas_tgt">>]
[ns_server:info,2017-02-28T11:55:04.954-08:00,ns_1@xxxxxxxxxxxx:<0.20270.8040>:compaction_new_daemon:spawn_scheduled_kv_compactor:467]Start compaction of vbuckets for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2017-02-28T11:55:04.958-08:00,ns_1@xxxxxxxxxxxx:<0.17990.8040>:compaction_new_daemon:bucket_needs_compaction:913]`cas_tgt` data size is 94285599006, disk size is 122973895680
[ns_server:debug,2017-02-28T11:55:04.959-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_compactors_exit:1292]Finished compaction iteration.
[ns_server:debug,2017-02-28T11:55:04.959-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_scheduler:schedule_next:60]Finished compaction for compact_kv too soon. Next run will be in 30s
[ns_server:debug,2017-02-28T11:55:26.255-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_scheduler_message:1251]Starting compaction (compact_master) for the following buckets:
[<<"cas_tgt">>]
[ns_server:info,2017-02-28T11:55:26.256-08:00,ns_1@xxxxxxxxxxxx:<0.13091.8040>:compaction_new_daemon:spawn_master_db_compactor:791]Start compaction of master db for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2017-02-28T11:55:26.257-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_compactors_exit:1292]Finished compaction iteration.
[ns_server:debug,2017-02-28T11:55:26.257-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_scheduler:schedule_next:60]Finished compaction for compact_master too soon. Next run will be in 30s
[ns_server:debug,2017-02-28T11:55:34.224-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_scheduler_message:1251]Starting compaction (compact_views) for the following buckets:
[<<"cas_tgt">>]
[ns_server:info,2017-02-28T11:55:34.231-08:00,ns_1@xxxxxxxxxxxx:<0.13763.8040>:compaction_new_daemon:spawn_scheduled_views_compactor:493]Start compaction of indexes for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2017-02-28T11:55:34.232-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_compactors_exit:1292]Finished compaction iteration.
[ns_server:debug,2017-02-28T11:55:34.232-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_scheduler:schedule_next:60]Finished compaction for compact_views too soon. Next run will be in 30s
[ns_server:debug,2017-02-28T11:55:34.960-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_scheduler_message:1251]Starting compaction (compact_kv) for the following buckets:
[<<"cas_tgt">>]
[ns_server:info,2017-02-28T11:55:34.962-08:00,ns_1@xxxxxxxxxxxx:<0.17660.8040>:compaction_new_daemon:spawn_scheduled_kv_compactor:467]Start compaction of vbuckets for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2017-02-28T11:55:34.966-08:00,ns_1@xxxxxxxxxxxx:<0.19631.8040>:compaction_new_daemon:bucket_needs_compaction:913]`cas_tgt` data size is 94285827164, disk size is 123054369792
[ns_server:debug,2017-02-28T11:55:34.966-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_compactors_exit:1292]Finished compaction iteration.
[ns_server:debug,2017-02-28T11:55:34.966-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_scheduler:schedule_next:60]Finished compaction for compact_kv too soon. Next run will be in 30s
[ns_server:info,2017-02-28T11:55:36.145-08:00,ns_1@xxxxxxxxxxxx:ns_config_rep<0.1532.0>:ns_config_rep:do_pull:347]Pulling config from: 'ns_1@xxxxxxxxxxxx'

[ns_server:info,2017-02-28T11:55:54.437-08:00,ns_1@xxxxxxxxxxxx:ns_config_rep<0.1532.0>:ns_config_rep:do_pull:347]Pulling config from: 'ns_1@xxxxxxxxxxxx'

[ns_server:debug,2017-02-28T11:55:56.258-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_scheduler_message:1251]Starting compaction (compact_master) for the following buckets:
[<<"cas_tgt">>]
[ns_server:info,2017-02-28T11:55:56.259-08:00,ns_1@xxxxxxxxxxxx:<0.13535.8040>:compaction_new_daemon:spawn_master_db_compactor:791]Start compaction of master db for bucket cas_tgt with config:
[{database_fragmentation_threshold,{30,undefined}},
 {view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2017-02-28T11:55:56.262-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_new_daemon:process_compactors_exit:1292]Finished compaction iteration.
[ns_server:debug,2017-02-28T11:55:56.262-08:00,ns_1@xxxxxxxxxxxx:compaction_new_daemon<0.1683.0>:compaction_scheduler:schedule_next:60]Finished compaction for compact_master too soon. Next run will be in 30s

Please let me know, if you need further details.

Regards,
Jeeva

1 Like
2

By the we have approximately:

  • On RAM 79 GB data
  • On Disk 371 GB data

Regards,
Jeeva

3

Couchbase team, looking forward to any insights here?

Regards,
Jeeva

4

It been a 5 days, can anyone shed some light here?

Regards,
Jeeva