Lost indexes spontaneously

Couchbase Server
1

4 days ago, When I was on a vacation, My service goes down and I get No index available on keyspace myapp that matches your query. Use CREATE INDEX or CREATE PRIMARY INDEX to create an index, or check that your expected index is online. error in couchbase

I have 3 nodes and 1 replicate for each index, I checked that there is no hardware and network failure in the system and all 3 nodes are UP and operating, I checked Indexes page of Couchbase admin panel and figured out

  1. In the first bucket all indexes lost and dropped (No body dropped indexes it’s dropped spontaneously)
  2. In the second bucket 3 of 6 indexes lost and dropped spontaneously
  3. In the third bucket all indexes exists and noting is wrong

Is there any known issue causes index lost? We have no hardware and network failure and no software update and changes, I am using Community Edition 6.6.0 build 7909

2

@socketman2016 , the only time index service can cleanup an index on its own is it detects a storage corruption. In that case, you should see messages like “Detected storage corruption for index [xxx], partition id [xxx]. Starting cleanup.” in the UI->Logs.

1 Like
3

I have no Detected storage corruption in logs

  1. What is a a storage corruption? is it a hardware failure?
  2. I have Deleting old data files of bucket "xxx" log, What is it?
  3. I have many of this log too
Client-side error-report for user "root" on node 'ns_1@xxx.xxx.xxx.xxx':
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36
Got unhandled javascript error:
name: TypeError;
message: Cannot read property 'trim' of undefined;
stack: TypeError: Cannot read property 'trim' of undefined
at canCreateBlankQuery (http://xxx.xxx.xxx.xxx:8091/_p/ui/query/qw_query_service.js:775:35)
at queryController.hasNextResult [as hasNext] (http://xxx.xxx.xxx.xxx:8091/_p/ui/query/qw_query_service.js:784:7)
at fn0 (eval at compile (http://xxx.xxx.xxx.xxx:8091/ui/libs/angular.js:1:1), <anonymous>:4:349)
at expressionInputWatch (http://xxx.xxx.xxx.xxx:8091/ui/libs/angular.js:16669:31)
at Scope.$digest (http://xxx.xxx.xxx.xxx:8091/ui/libs/angular.js:18372:34)
at Scope.$apply (http:/xxx.xxx.xxx.xxx:8091/ui/libs/angular.js:18649:24)
at http://xxx.xxx.xxx.xxx:8091/ui/libs/angular.js:20574:36
at completeOutstandingRequest (http://xxx.xxx.xxx.xxx:8091/ui/libs/angular.js:6428:10)
at http://xxx.xxx.xxx.xxx:8091/ui/libs/angular.js:6707:7;
4
  1. What is a a storage corruption? is it a hardware failure?

The index storage corruption can be a result of a software bug and/or a hardware failure. That doesn’t seem to be the case here.

  1. I have Deleting old data files of bucket "xxx" log, What is it?

Not sure. But this message is related to bucket data file, not the index data files.

We’ll need to take a look at the indexer.log to see what is causing the indexes to get cleaned up. Is the application dropping/recreating the bucket by any chance?

5

Is the application dropping/recreating the bucket by any chance?

NO

6

that means I lost data? OMG!

7

Where is indexer.log?

8

It can be some old data cleanup that was pending. May not be actual data loss, but cannot be sure. Need to check the logs.
CC @drigby.

9

On linux, the default lot directory is:

/opt/couchbase/var/lib/couchbase/logs

For more information:
https://docs.couchbase.com/server/current/manage/manage-logging/manage-logging.html

10

I have ‘Deleting old data files of bucket “xxx”’ log, What is it?

Can you post the complete log message? If necessary you can redact bucket names.

11

Hi @socketman2016 ,

Hoping that everything is good on your side.

Please let up know if any help is needed on this.

Thanks.