Service 'index' exited with status 134

paralen · June 28, 2022, 7:29am

Hi,
I run Couchbase 7.1.0 on Ubuntu 20.04. The following errors occurs:

Service 'index' exited with status 134. Restarting. Messages:
2022-06-27T13:08:13.287+00:00 [Info] RebalanceServiceManager::rebalanceJanitor Running Periodic Cleanup
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7f80db219a80)
2022-06-27T13:08:14.897+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 15031182372585180544 FlushTime 173ns CommitTime 3.160754ms TotalFlushTime 1.4252ms TotalCommitTime 5m43.882682393s
2022-06-27T13:08:14.897+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 15031182372585180544 Creating New Snapshot SnapshotInfo: seqnos: 223, 0, 69883 committed:true
2022-06-27T13:08:14.898+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 15031182372585180544 PartitionId: 0 SliceId: 0 Crc64: 833398431925170828 (SnapshotInfo: seqnos: 223, 0, 69883 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 39.957733ms SnapOpenDur 134.563µs
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp before terminating.

I have two nodes and 200 documents. This error is causing com.couchbase.client.core.error.InternalServerFailureException: Internal Couchbase Server error in my java app sometimes.

amit.kulkarni · June 30, 2022, 4:45am

CC @sduvuru for forestdb related question.

sduvuru · June 30, 2022, 10:50pm

Looks like some memory corruption. Can you run forestdb_dump on the forestdb file and see if the disk image looks ok?

paralen · July 4, 2022, 6:34pm

forestdb_dump:

2022-07-04T18:32:30.545+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 0). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.545+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 1). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 38, SB No.: 2). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 3). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Crash Detected: 2064 non-block aligned bytes discarded from a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp'

2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Crash Detected: Last Block not DBHEADER 0 in a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp'

2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Error in READ on a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp', errno = 2: 'No such file or directory'

Unable to open /opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp

paralen · July 6, 2022, 3:37pm

another crash:

Service 'index' exited with status 134. Restarting. Messages:
2022-07-06T15:24:43.968+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 17532055274852714362 PartitionId: 0 SliceId: 0 Crc64: 833398431925170828 (SnapshotInfo: seqnos: 2, 0, 146036 committed:true) SnapType FORCE_COMMIT SnapAligned true SnapCreateDur 14.075703ms SnapOpenDur 55.282µs
2022-07-06T15:24:43.981+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 12084728265744008952 FlushTime 305ns CommitTime 194.802µs TotalFlushTime 0s TotalCommitTime 17.987216926s
2022-07-06T15:24:43.981+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 12084728265744008952 Creating New Snapshot SnapshotInfo: seqnos: 1, 0, 146036 committed:true
2022-07-06T15:24:43.981+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 12084728265744008952 PartitionId: 0 SliceId: 0 Crc64: 833398431925170828 (SnapshotInfo: seqnos: 1, 0, 146036 committed:true) SnapType FORCE_COMMIT SnapAligned true SnapCreateDur 12.338927ms SnapOpenDur 31.423µs
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7fecfcb4f490)
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/94f72b8b-717a-4e23-e39eac87-213f8425.dmp before terminating.

I think this bug was introduced in couchbase 7.x

paralen · July 13, 2022, 2:48pm

Is there any solution to the problem? There is always at least one crash per day.
It’s impossible to downgrade (7.1 → 6.6) for us because of scopes / collections.

voskobovich · December 8, 2022, 12:05pm

I have the same problem on Community Edition 7.1.1 build 3175

Service 'index' exited with status 134. Restarting. Messages:
2022-12-05T21:53:10.962+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 10649006615221593611 PartitionId: 0 SliceId: 0 Crc64: 2656193918834593421 (SnapshotInfo: seqnos: 86, 0, 2262 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 20.653743ms SnapOpenDur 27.777µs
2022-12-05T21:53:10.963+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 18322477779647034417 FlushTime 238ns CommitTime 376.973µs TotalFlushTime 7.514243ms TotalCommitTime 15.590804117s
2022-12-05T21:53:10.963+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 18322477779647034417 Creating New Snapshot SnapshotInfo: seqnos: 11, 11, 21854 committed:true
2022-12-05T21:53:10.963+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 18322477779647034417 PartitionId: 0 SliceId: 0 Crc64: 2656193918834593421 (SnapshotInfo: seqnos: 11, 11, 21854 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 17.850179ms SnapOpenDur 30.491µs
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7fa73a21c650)
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/ff12b14d-2b76-41c9-8d079cbb-8d2ec0a2.dmp before terminating.

@sduvuru @amit.kulkarni have you any updates?

voskobovich · December 18, 2022, 1:21pm

Hi @varun.velamuri!

Can you help me with this thread?

I am still getting the following error:

Service 'index' exited with status 134. Restarting. Messages:
2022-12-17T23:36:05.537+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 10682370526375687935 PartitionId: 0 SliceId: 0 Crc64: 16428671364733376948 (SnapshotInfo: seqnos: 1187, 0, 67607 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 118.393072ms SnapOpenDur 66.423µs
2022-12-17T23:36:05.537+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 2977984481364261990 FlushTime 222ns CommitTime 432.79µs TotalFlushTime 60.897316ms TotalCommitTime 13.284413481s
2022-12-17T23:36:05.537+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 2977984481364261990 Creating New Snapshot SnapshotInfo: seqnos: 1166, 0, 67604 committed:true
2022-12-17T23:36:05.537+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 2977984481364261990 PartitionId: 0 SliceId: 0 Crc64: 16428671364733376948 (SnapshotInfo: seqnos: 1166, 0, 67604 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 115.916262ms SnapOpenDur 23.924µs
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7f00b2207150)
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp before terminating.

The result after running forestdb_dump

root@468afc6cb3c6:/opt/couchbase/var/lib/couchbase# forestdb_dump /opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 0). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 1). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 6f, SB No.: 2). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 3). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.575+00:00 [ERRO][FDB] Crash Detected: 3656 non-block aligned bytes discarded from a database file '/opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp'

2022-12-18T13:19:47.575+00:00 [ERRO][FDB] Crash Detected: Last Block not DBHEADER 20 in a database file '/opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp'

2022-12-18T13:19:47.604+00:00 [ERRO][FDB] Error in READ on a database file '/opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp', errno = 2: 'No such file or directory'

Unable to open /opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp

varun.velamuri · December 19, 2022, 5:15am

@voskobovich I am not sure of what is causing this issue. In general, ForestDB (the storage engine for community edition) has limitations with regard to the support for large keys. Do you have any large keys in your index - Like, the indexed entries are more than 2KB? In such cases, you need to rework your index to make it smaller.

I will re-direct this question to one of our ForestDB experts for a better answer.

Thanks,
Varun

voskobovich · December 20, 2022, 8:40pm

@varun.velamuri thanks for your answer.

I can’t exactly answer your question. How can I find the max size of the document in all buckets and all collections? Maybe couchbase dashboard can show this?

I found two links with info about the max size of the key and document in ForestBD: first and second

The maximum sizes of a key, metadata, and value are 64KB, 64KB, and 4GB, respectively. They are all simply processed as binary objects.

How did you find out about 2 KB per entity?

varun.velamuri · December 22, 2022, 4:03am

@voskobovich What I mean is indexed entry. If you have a composite index with many fields, then each indexed entry will be large.

You can calculate the size of an indexed entry approximately by doing data_size / items_count stats - Both these stats are available per index on UI and also on indexer_stats.log. If there are many composite fields and the indexed entry is going to grow larger, then you may have to rework your index.

Thanks,
Varun

voskobovich · December 22, 2022, 2:14pm

@varun.velamuri Thanks for the clarification.

Max size of the index entry in my server is 50 bytes. Most likely this is not the problem.

Two days ago, I saw that the index service had 99 MB of RAM left out of 512 MB. I added 256 MB. A total of 768 MB of RAM for the index service. Since then, there have been no such errors.
Can out-of-memory cause the errors described above?

varun.velamuri · December 22, 2022, 9:50pm

@voskobovich I am glad that it solved the issue. Unfortunately, I have little knowledge of how ForestDB storage engine works to comment if this issue is due to out-of-memory.

Probably @sduvuru can share more insights.

sduvuru · January 3, 2023, 3:26pm

Forestdb does not handle memory failures well. This does create random memory corruption and crashes. I think the right sizing of memory will resolve those issues.

jonathan · September 8, 2023, 1:20pm

I have the exact same issue on 7.1.5.
Crashing once or twice a day.
I have lot of RAM left for indexes (5GB).
Indexes are quite small (17-20K items).
Key sizes are under 2KB.

I’m seeing this issue since quite a lot of time but it’s just now affecting my application’s behavior.
This is seriously impacting reliability of my application.

Error log :

Service 'index' exited with status 134. Restarting. Messages:
2023-09-08T10:07:18.104+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 3068491716512286044 PartitionId: 0 SliceId: 0 Crc64: 4870296443227363729 (SnapshotInfo: seqnos: 19, 19, 3756951 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 16.09978ms SnapOpenDur 17.592µs
2023-09-08T10:07:18.106+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 18072388016328274219 FlushTime 218ns CommitTime 217.724µs TotalFlushTime 0s TotalCommitTime 7.263702701s
2023-09-08T10:07:18.106+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 18072388016328274219 Creating New Snapshot SnapshotInfo: seqnos: 8, 5, 3756950 committed:true
2023-09-08T10:07:18.106+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 18072388016328274219 PartitionId: 0 SliceId: 0 Crc64: 4870296443227363729 (SnapshotInfo: seqnos: 8, 5, 3756950 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 111.960441ms SnapOpenDur 34.995µs
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/4b0b291e-c1de-48fa-93b436bb-82f5e3e8.dmp before terminating.
Service 'index' exited with status 134. Restarting. Messages:
2023-09-08T10:07:18.104+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 3068491716512286044 PartitionId: 0 SliceId: 0 Crc64: 4870296443227363729 (SnapshotInfo: seqnos: 19, 19, 3756951 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 16.09978ms SnapOpenDur 17.592µs
2023-09-08T10:07:18.106+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 18072388016328274219 FlushTime 218ns CommitTime 217.724µs TotalFlushTime 0s TotalCommitTime 7.263702701s
2023-09-08T10:07:18.106+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 18072388016328274219 Creating New Snapshot SnapshotInfo: seqnos: 8, 5, 3756950 committed:true
2023-09-08T10:07:18.106+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 18072388016328274219 PartitionId: 0 SliceId: 0 Crc64: 4870296443227363729 (SnapshotInfo: seqnos: 8, 5, 3756950 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 111.960441ms SnapOpenDur 34.995µs
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/4b0b291e-c1de-48fa-93b436bb-82f5e3e8.dmp before terminating.

forestdb_dump :

2023-09-08T13:05:09.275+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 0). Note: this message might be a false alarm if upgrade is running.
2023-09-08T13:05:09.275+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 1). Note: this message might be a false alarm if upgrade is running.
2023-09-08T13:05:09.275+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 2). Note: this message might be a false alarm if upgrade is running.
2023-09-08T13:05:09.276+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 3). Note: this message might be a false alarm if upgrade is running.
2023-09-08T13:05:09.276+00:00 [ERRO][FDB] Crash Detected: 1040 non-block aligned bytes discarded from a database file 'f305cba1-1f0c-46c2-c78ed2a7-94714f47.dmp'

2023-09-08T13:05:09.276+00:00 [ERRO][FDB] Crash Detected: Last Block not DBHEADER 0 in a database file 'f305cba1-1f0c-46c2-c78ed2a7-94714f47.dmp'

2023-09-08T13:05:09.277+00:00 [ERRO][FDB] Error in READ on a database file 'f305cba1-1f0c-46c2-c78ed2a7-94714f47.dmp', errno = 2: 'No such file or directory'

Unable to open f305cba1-1f0c-46c2-c78ed2a7-94714f47.dmp

@sduvuru can I provide any more info to you to diagnose the issue ?

snowvil84 · October 4, 2023, 4:26am

Settings → Auto-Compaction → Index Fragmentation

Circular write mode with day + time interval trigger → Append-only write mode with index fragmentation level trigger

After changing to the above option, the issue with the Indexer Service terminating abnormally was resolved.

I’m not sure, but it seems that the problem occurs while performing circular write mode.

jonathan · November 12, 2023, 5:50pm

Thanks a lot for your message. I just changed this setting. I’ll let you know if it solves the issue for me too.
Thanks again for taking the time to help me

paralen · November 13, 2023, 3:23pm

Be careful. We tried this setting and our couchbase instance went out of space and crashed (after few hours).

jonathan · November 16, 2023, 10:32am

I’ve had the same issue on 7.2.2 with indexer crashing because of RAM.
On another server on 7.1.5 no issue on this side, however it didn’t fix the issue completely. It happens less often for sure but it still did happen this morning.
I’m monitoring this with a few variations in my config but if I can’t find any way of getting it to work I’ll make a new dedicated thread on that matter.
I’ll consider changing database if this doesn’t get solved because for the time being I don’t have a stable software because of this issue…