Compaction process being killed even after disabling Time Period?

Hello,

I used to have the auto-compaction period set from 3am to 6am for my DB. I could see in the log that every morning at 6am the memcached process was killed and the bucket reloaded. I was under the impression that this was caused by the auto-compaction not having enough time to finish and decided to disable the time-period for auto-compaction. I can see in the bucket stats that auto-compaction now does happen continuously, but memcached is still killed everyday at precisely 6am.

So I have two questions:

- Am I right in assuming that the process is killed because of the auto-compaction period time-out (I've pasted the log below)?
- If so, could there be a bug where if you disable the auto-compaction time-period the compaction is effectivelly done at anytime, but the time-out would still apply?

Here's the log that I get at 6 am everyday:

Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 255. Restarting. Messages: Sun Sep 22 06:00:11.716865 Eastern Daylight Time 3: (bucket) Trying to connect to mccouch: "127.0.0.1:11213"
Sun Sep 22 06:00:11.716865 Eastern Daylight Time 3: (bucket) Connected to mccouch: "127.0.0.1:11213"
Sun Sep 22 06:00:40.508636 Eastern Daylight Time 3: Extension support isn't implemented in this version of bucket_engine
Sun Sep 22 06:00:40.583652 Eastern Daylight Time 3: (bucket) Failed to load mutation log, falling back to key dump
Sun Sep 22 06:10:32.870741 Eastern Daylight Time 3: (bucket) metadata loaded in 0h:10m:21s
Sun Sep 22 06:14:51.561558 Eastern Daylight Time 3: (bucket) Total memory use reached to the low water mark, stop warmup
Sun Sep 22 06:14:51.561558 Eastern Daylight Time 3: (bucket) Engine warmup is complete, request to stop loading remaining database
Sun Sep 22 06:14:51.562546 Eastern Daylight Time 3: (bucket) Canceling loading database, warmup has completed
Sun Sep 22 06:14:51.572588 Eastern Daylight Time 3: (bucket) warmup completed in 0h:14m:39s
Mon Sep 23 01:11:25.785808 Eastern Daylight Time 3: 42532 Closing connection due to read error: Unknown error

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

and then:

Control connection to memcached on 'ns_1@10.144.5.77' disconnected: {badmatch,
{error,
closed}}

1 Answer

« Back to question.

Sorry, should have thought of providing those details in the first place.

Couchbase version is 2.1.0 enterprise edition (build-718) running on a windows server 2012 virtual server (64 bits, 16 gig ram).

Couchbase is configured with 14 GB (used 10.6 GB atm) quota with a single bucket that has a quota of 14GB. Bucket contains only non-critical data and is not replicated/distributed. The bucket has 1.3M items at any time with a TTL of 3 days. At night when the DB is mostly idle, a process does a touch on about 75% of the items to keep them alive for 3 more days (basically for the items that have been served during the day). The process of doing those touches causes a lot of fragmentation (see http://www.couchbase.com/issues/browse/MB-6981 ) and auto-compaction kicks-in rapidly when we start this operation.

Let me know if I should provide more details.