[Docker] CB Server 7.0.2 buckets stuck after VM reboot

In our (individual) dev machines we are using official EE docker image, in a docker machine. The virtual machine has 4 cores and 8Go of RAM.

We have 3 buckets.
The problem occurs when we have a bucket with a lot of documents ( ~1.2M ) in one of the buckets, and we reboot the virtual machine. The CB server starts, says it warms up the buckets, yet nothing happen in any of the buckets. It says “loading keys X/Y” but the number doesn’t evolve.

Also sometimes one of the bucket appears green, yet no document are available. Indexes are green with 0 items in it.

Logs don’t say anything interesting. How can we work to troubleshoot that issue ?

Rebooting once more got me this log message after a while :


Service ‘indexer’ exited with status 2. Restarting. Messages: github.com/couchbase/indexing/secondary/indexer.(*indexer).handleStorageWarmupDone(0xc00008c700, 0x1557720, 0xc0086ae120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6853 +0xd14 github.com/couchbase/indexing/secondary/indexer.(*indexer).handleWorkerMsgs(0xc00008c700, 0x1557720, 0xc0086ae120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1387 +0x15d0 github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc00008c700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1034 +0x74d github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc00011f7d0, 0x4, 0x13312f3, 0xf, 0xc000137e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:466 +0x221a main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1427

EDIT 2 : Looks like disabling the auto-failover gives me a lot more information about what’s happening, I have some logs now !

EDIT 3 : So when I visit the “documents” tab, it shows an alert with Error with document: some_document_id : 404 - Object Not Found and does that for every documents in the bucket (I’m on a bucket with only a few dozen docs in dev)

EDIT 4 : one more log


Compactor for database liskexplorer (pid [{type,database}, {important,true}, {name,<<“liskexplorer”>>}, {fa, {#Fun<compaction_daemon.4.113866204>, [<<“liskexplorer”>>, {config, {30,undefined}, {30,undefined}, undefined,false,false, {daemon_config,30,131072, 20971520}}, false, {[{type,bucket}]}]}}]) terminated unexpectedly: {{bad_return_value, {stop, {error, {bucket_create_error, {memcached_error, not_stored, undefined}}}}}, {gen_server, call, [{‘ns_memcached-liskexplorer’, ‘ns_1@’}, {raw_stats, <<“diskinfo”>>, #Fun<compaction_daemon.18.113866204>, {<<“0”>>, <<“0”>>}}, 180000]}} (repeated 1 times, last seen 29.233904 secs ago)

EDIT 5 : Ok docker logs are more interesting. Thousands of chown: changing ownership of <some_path>': Operation not permitted