Recover Indexer Machine

Couchbase Server
1

HI,
Yesterday I had removed a data node and then added a new one
Now My Indexer Machine is Not Working. It Indexes and then stops indexing. Can You please suggest me ways to recreate Index

2

Hi @ronagrawal,

To recreate indexes (I assume you mean N1QL indexes), you can run the CREATE INDEX statements again. But more to the point, maybe you’d like to share your error log information so we can figure out why the indexing isn’t working?

3

I deleted one data node and then I added one.
Now it Seems my Index Machine is Not Working.
Following is the log Error:

Node ‘ns_1@xx’ saw that node 'ns_1@xx went down. Details: [{nodedown_reason,
net_tick_timeout}]

Node ‘ns_1@xx’ saw that node ‘ns_1@xxcompute.amazonaws.com’ came up. Tags: []

Service ‘memcached’ exited with status 137. Restarting. Messages: 2017-04-10T15:57:10.777436Z WARNING (mt_flights) DCP (Consumer)

Mutation Queue Waiting For Node Alloc for 226410 Milliseconds Vbucket 121

Service ‘projector’ exited with status 1. Restarting. Messages: github.com/couchbase/indexing/secondary/projector.(*Projector).handleRequest(0xc208047180, 0x7fa6807fc718, 0xc26b589d20, 0xc2080e0005) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/adminport.go:94 +0x59f created by github.com/couchbase/indexing/secondary/projector.(*Projector).mainAdminPort /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/adminport.go:70 +0x1018 [goport] 2017/04/12 08:16:39 /opt/couchbase/bin/projector terminated: exit status 2

4

Hi Ronagrawal,

First of all, thank you for your feedback!

  1. Can you please tell us which version of Couchbase you are using?
  2. Can you describe briefly your approach of removing and adding a new node in the cluster? Did you rebalance the cluster after this operation?
  3. Is there any way you can send us full logs for your cluster? Using cbcollect_info tool

Thank you and regards,
Don Pinto
Couchbase Server PM

5

Couchbase Version : Version: 4.1.1-5914 Enterprise Edition (build-5914)

  1. I removed one data node. Later It asked for rebalancing… I did that. It failed. I again initiated the process. It worked.
  2. However, my Index Machine stopped Working. It indexed few documents and then it stopped indexing.
  3. Today I added a new machine for Indexing.However It is stuck after completing 77%

The log will have ip address as well. Still I will generate the log.
Please Wait And thank you for the reply

Server Nodes Image


Data Bucket

6

@ronagrawal, Please make sure you set a reasonable memory quota for the index service, depending on how much you are trying to index, rather than the default 256MB(can be modified via UI->Settings).

Also, it would be a good idea to upgrade to 4.5.x. It is much more robust and performs better.