I am running Couchbase Server 7.1.1 in Ubuntu inside docker and this problem happens regularly (approximately bi-weekly in our CI environment).
I have also observed “it” (or similar symptom) running on my development machine inside a docker container in Windows/WSL and natively in Windows.
It can be recovered by deleting all indices (@2i folder) and restarting the couchbase service.
I have the logs and the dmp file, if there is an interest to look into this, where can I upload them?
I can see SIGSEGV: segmentation violation in indexer.log and the error.log is showing failure to connect to the index service:
[ns_server:error,2022-10-08T06:20:04.087Z,ns_1@127.0.0.1:service_agent-index<0.31453.1>:service_agent:terminate:259]Terminating abnormally
[ns_server:error,2022-10-08T06:21:07.476Z,ns_1@127.0.0.1:service_agent-index<0.12999.2>:service_agent:handle_info:238]Lost json rpc connection for service index, reason shutdown. Terminating.
[ns_server:error,2022-10-08T06:21:07.476Z,ns_1@127.0.0.1:service_agent-index<0.12999.2>:service_agent:terminate:259]Terminating abnormally
[ns_server:error,2022-10-08T06:24:33.161Z,ns_1@127.0.0.1:service_agent-index<0.16965.2>:service_agent:handle_info:238]Lost json rpc connection for service index, reason shutdown. Terminating.
[ns_server:error,2022-10-08T06:24:33.162Z,ns_1@127.0.0.1:service_agent-index<0.16965.2>:service_agent:terminate:259]Terminating abnormally
[ns_server:error,2022-10-08T06:24:33.521Z,ns_1@127.0.0.1:service_status_keeper_worker<0.567.0>:rest_utils:get_json:62]Request to (indexer) getIndexStatus with headers [{“If-None-Match”,
“843e514a7c4a9c48”}] failed: {error,
{econnrefused,
============
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x7f65efaa1cd2]
runtime stack:
runtime.throw({0x1110cae?, 0x1d765e0?})
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/panic.go:992 +0x71
runtime.sigpanic()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/signal_unix.go:802 +0x3a9
goroutine 486 [syscall]:
runtime.cgocall(0xe95381, 0xc0050b0b48)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/cgocall.go:157 +0x5c fp=0xc0050b0b20 sp=0xc0050b0ae8 pc=0x40979c
github.com/couchbase/indexing/secondary/fdb._Cfunc_fdb_get_kv(0x7f6564608540, 0xc00804aaa0, 0x4b, 0xc00ac258b8, 0xc00c5106c0)
_cgo_gotypes.go:736 +0x4c fp=0xc0050b0b48 sp=0xc0050b0b20 pc=0xb0058c
github.com/couchbase/indexing/secondary/fdb.(*KVStore).GetKV.func1(0xc004a871d0?, 0x413405?, 0xc0050b0bf8?, 0x456c49?, 0xc0050b0bf8?)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/kv.go:35 +0xbb fp=0xc0050b0bb0 sp=0xc0050b0b48 pc=0xb0e63b
github.com/couchbase/indexing/secondary/fdb.(*KVStore).GetKV(0xc004a871d0, {0xc00804aaa0, 0x4b, 0x3?})
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/kv.go:35 +0x10c fp=0xc0050b0c60 sp=0xc0050b0bb0 pc=0xb0e2ec
github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).insertPrimaryIndex(0xc0000a6800, {0xc00804aaa0, 0x4b, 0x50}, {0xc01710c1e0, 0x4b, 0x50}, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:427 +0x225 fp=0xc0050b0de0 sp=0xc0050b0c60 pc=0xd15f05
github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).insert(0xc0000a6800, {0xc00804aaa0?, 0x1?, 0x50?}, {0xc01710c280?, 0x51?, 0x60?}, {0xc01710c1e0, 0x4b, 0x50}, …)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:409 +0x105 fp=0xc0050b0e78 sp=0xc0050b0de0 pc=0xd15c05
github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).handleCommandsWorker(0xc0000a6800, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:357 +0x288 fp=0xc0050b0fc0 sp=0xc0050b0e78 pc=0xd15468
github.com/couchbase/indexing/secondary/indexer.NewForestDBSlice.func1()
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:170 +0x2a fp=0xc0050b0fe0 sp=0xc0050b0fc0 pc=0xd1492a
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0050b0fe8 sp=0xc0050b0fe0 pc=0x46e301
created by github.com/couchbase/indexing/secondary/indexer.NewForestDBSlice
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:170 +0xe3a