We’re having some trouble with a one-node cluster which is setup on a vagrant box running in a docker-container. I’m not really sure where to start debugging this, any help would be great.
2016-05-31T18:04:12.866+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash] 2016-05-31T18:04:12.935+00:00 [Info] Indexer::NewIndexer Status INIT 2016-05-31T18:04:13.145+00:00 [Info] Setting maxcpus = 4 2016-05-31T18:04:13.146+00:00 [Info] Setting buffer block size to 16384 bytes 2016-05-31T18:04:13.146+00:00 [Info] Setting log level to Info 2016-05-31T18:04:13.146+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2016-05-31T18:04:13.235+00:00 [Info] [Queryport ":9101"] started ... 2016-05-31T18:04:13.25+00:00 [Info] New settings received: {"indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":400,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.memory_quota":268435456} 2016-05-31T18:04:13.25+00:00 [Info] Setting maxcpus = 4 2016-05-31T18:04:13.25+00:00 [Info] Setting log level to Info 2016-05-31T18:04:13.459+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2016-05-31T18:04:13.465+00:00 [Info] ClustMgr:handleSetLocalValue Key IndexerId Value 4a:ff:39:d0:e4:27:d0:1 2016-05-31T18:04:13.468+00:00 [Info] Indexer Id 4a:ff:39:d0:e4:27:d0:1 2016-05-31T18:04:13.493+00:00 [Info] KVSender::sendShutdownTopic Projector 10.0.2.15:9999 Topic MAINT_STREAM_TOPIC_4a:ff:39:d0:e4:27:d0:1 2016-05-31T18:04:13.507+00:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 10.0.2.15:9999 Topic MAINT_STREAM_TOPIC_4a:ff:39:d0:e4:27:d0:1. Err proto: protobuf.Error: illegal tag 0 (wire type 0) 2016-05-31T18:04:13.507+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received proto: protobuf.Error: illegal tag 0 (wire type 0) from 10.0.2.15:9999 2016-05-31T18:04:14.509+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received proto: protobuf.Error: illegal tag 0 (wire type 0) 2016-05-31T18:04:14.509+00:00 [Error] Indexer::closeAllStreams Stream MAINT_STREAM Error from Projector proto: protobuf.Error: illegal tag 0 (wire type 0). Retrying. 2016-05-31T18:04:19.515+00:00 [Info] KVSender::sendShutdownTopic Projector 10.0.2.15:9999 Topic MAINT_STREAM_TOPIC_4a:ff:39:d0:e4:27:d0:1 2016-05-31T18:04:19.516+00:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 10.0.2.15:9999 Topic MAINT_STREAM_TOPIC_4a:ff:39:d0:e4:27:d0:1. Err proto: protobuf.Error: illegal tag 0 (wire type 0)
The last 2 messages just repeat over and over in the indexer.log
Here’s some from the error.log
[ns_server:error,2016-06-02T15:59:33.709Z,ns_1@10.0.2.15:index_status_keeper_worker<0.652.0>:index_rest:get_json:45]R equest to http://127.0.0.1:9102/getIndexStatus failed: {error, {econnrefused, [{lhttpc_client, send_request,1, [{file, "/home/couchbase/jenkins/workspace/sherlock-unix/couchd b/src/lhttpc/lhttpc_client.erl"}, {line,220}]}, {lhttpc_client, execute,9, [{file, "/home/couchbase/jenkins/workspace/sherlock-unix/couchd b/src/lhttpc/lhttpc_client.erl"}, {line,169}]}, {lhttpc_client, request,9, [{file, "/home/couchbase/jenkins/workspace/sherlock-unix/couchd b/src/lhttpc/lhttpc_client.erl"}, {line,92}]}]}}
More digging reveals (in projector.log)
2016-06-02T15:50:35.559+00:00 [Error] pram[:9999] listen failed listen tcp :9999: bind: address already in use
We’re running other Java services locally and it looks like it might be JMX collision