Hi we are running Couchbase Community Edition 5.1.1 and are seeing issues with the Indexes failing and needing to be rebuilt.
In a number of cases over a number of different machines our application will fails in a variety ways because queries that are expected to work return no data.
If we drop all of the indexes and recreate them then the problem is fixed. But we really want to know why this is happening and to prevent it from happening to start with.
Are there any known issues that might be contributing to this problem?
We have had another occurrence of this problem and I’ve been able to find some potentially relevant errors in the Couchbase log files. The key snippet of error log seems to be;
Service ‘indexer’ exited with status 2. Restarting. Messages:
'Error status code: -1, Error in REMOVE on a database file ‘c:\Program Files\Couchbase\Server\var\lib\couchbase\data@2i\kakapo-bird_idx_record_chips_birdID_5621171458353600489_0.index\data.fdb.0’, errno = 32: 'The process cannot access the file because it is being used by another process.
Is this a “normal” error message or does it indicate the problem we are having with indexes failing?
Google found this issue in the bug database: https://issues.couchbase.com/browse/MB-17215
That seems to indicate that this sort of an error is a major, but the bug report says that it was fixed in 4.5, and we’re using 5.1.1.
Hi @richard.perfect, It would help us to investigate these failures if you can provide the following information-
- OS Version
- How many nodes in the cluster do you have, and what is the layout of services
- What kind of operations were going on when you hit this issue? Were the indexes being deleted ?
- How many documents are there in the bucket on which the above index is built?
- Can you upload logs ?
Thanks @richard.perfect. I have sent you an email. Pls send me the Google drive link to the log files.
Hi Mihir, - I sent through a link to the log files a few days ago. Were you able to download them ok?
Hi @richard.perfect, I was able to download the logs fine. I have gone through them and I do see the errors you mentioned. Unfortunately, the indexer logs have rotated, and I cannot see the logs for the period where the indexer service was restarted. It would be great if you can take a cbcollect and share the logs if the issue is seen again. In the meanwhile, I am also trying to reproduce the issue in house. I will keep you posted. Thanks.