Couchbase community 4.5 Indexer crash with Nessus Scanner

Couchbase Server
1

Hi,

We are using Couchbase Community version. We upgraded from community 4 to 4.5 and started seeing indexer crash when Nessus Scanner runs in our host. The Nessus scanner never had any issues or crashes in couchbase in community 4.0

The stack trace in indexer logs starts with “runtime error: makeslice: len out of range” and finally it goes to OOM and crashes. I tried the Couchbase gitter channel and I was suggested to post in couchbase forum for inputs to see if its a bug.

2018-01-16T21:27:29.006+00:00 [Error] panic in PeerPipe.doReceive() : runtime error: makeslice: len out of range
2018-01-16T21:27:29.006+00:00 [Error] /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:165 +0x24
github.com/couchbase/gometa/common.(*PeerPipe).doReceive.func1(0xc82830a060)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:192 +0xf8
panic(0xf72840, 0xc827bfc950)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:426 +0x4e9
github.com/couchbase/gometa/common.(*PeerPipe).readBytes(0xc82830a060, 0x1b252d3132333435, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:255 +0x8c
github.com/couchbase/gometa/common.(*PeerPipe).doReceive(0xc82830a060)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:211 +0x373
created by github.com/couchbase/gometa/common.NewPeerPipe
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:60 +0x142

2018-01-16T21:27:29.007+00:00 [Error] LeaderSyncProxy.updateAcceptEpochAfterQuorum(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
2018-01-16T21:27:29.007+00:00 [Error] LeaderServer:startProxy(): Leader Fail to synchronization with follower (TCP conn = 10.1.114.123:44636)
2018-01-16T21:27:34.069+00:00 [Error] panic in PeerPipe.doReceive() : runtime error: makeslice: len out of range
2018-01-16T21:27:34.070+00:00 [Error] /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:165 +0x24
github.com/couchbase/gometa/common.(*PeerPipe).doReceive.func1(0xc82454b6e0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:192 +0xf8
panic(0xf72840, 0xc82401c460)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:426 +0x4e9
github.com/couchbase/gometa/common.(*PeerPipe).readBytes(0xc82454b6e0, 0x1b252d3132333435, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:255 +0x8c
github.com/couchbase/gometa/common.(*PeerPipe).doReceive(0xc82454b6e0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:211 +0x373
created by github.com/couchbase/gometa/common.NewPeerPipe
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:60 +0x142

2018-01-16T21:27:34.070+00:00 [Error] LeaderSyncProxy.updateAcceptEpochAfterQuorum(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
2018-01-16T21:27:34.070+00:00 [Error] LeaderServer:startProxy(): Leader Fail to synchronization with follower (TCP conn = 10.1.114.123:46830)
2018-01-16T21:27:34.094+00:00 [Info] [Queryport “:9101”] connection “10.1.114.123:48886” doReceive() …
2018-01-16T21:27:36.570+00:00 [Info] Indexer::ReadMemstats Time Taken 473.590212ms
2018-01-16T21:27:36.616+00:00 [Info] Indexer::monitorMemUsage ManualGC Time Taken 42.804449ms
2018-01-16T21:27:36.958+00:00 [Error] panic in PeerPipe.doReceive() : runtime error: makeslice: len out of range
2018-01-16T21:27:36.958+00:00 [Error] /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:165 +0x24

Does anyone has any thoughts or have seen this issue in their implementation where Nessus Scanner is running?

Looking forward for some inputs.

Thanks
Mahesh

2

This is a known issue that has been fixed in Couchbase Server 5.0.0 CE.

3

Thank You Patrick for the update. Couchbase 5.0 would be a major upgrade for us at this point. But will be looking into upgrading it in near future. For now I am guessing there is no other latest 4.5.1-2844 Community Edition (build-2844) available . We are on build 2844. I will also test if Nessus scanner breaks anything in Couchbase CE 5.0 in our POC environments.

-Mahesh