Hi,
I run Couchbase 7.1.0 on Ubuntu 20.04. The following errors occurs:
Service 'index' exited with status 134. Restarting. Messages:
2022-06-27T13:08:13.287+00:00 [Info] RebalanceServiceManager::rebalanceJanitor Running Periodic Cleanup
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7f80db219a80)
2022-06-27T13:08:14.897+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 15031182372585180544 FlushTime 173ns CommitTime 3.160754ms TotalFlushTime 1.4252ms TotalCommitTime 5m43.882682393s
2022-06-27T13:08:14.897+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 15031182372585180544 Creating New Snapshot SnapshotInfo: seqnos: 223, 0, 69883 committed:true
2022-06-27T13:08:14.898+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 15031182372585180544 PartitionId: 0 SliceId: 0 Crc64: 833398431925170828 (SnapshotInfo: seqnos: 223, 0, 69883 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 39.957733ms SnapOpenDur 134.563µs
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp before terminating.
I have two nodes and 200 documents. This error is causing com.couchbase.client.core.error.InternalServerFailureException: Internal Couchbase Server error in my java app sometimes.
CC @sduvuru for forestdb related question.
Looks like some memory corruption. Can you run forestdb_dump on the forestdb file and see if the disk image looks ok?
forestdb_dump:
2022-07-04T18:32:30.545+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 0). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.545+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 1). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 38, SB No.: 2). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 3). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Crash Detected: 2064 non-block aligned bytes discarded from a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp'
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Crash Detected: Last Block not DBHEADER 0 in a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp'
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Error in READ on a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp', errno = 2: 'No such file or directory'
Unable to open /opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp
another crash:
Service 'index' exited with status 134. Restarting. Messages:
2022-07-06T15:24:43.968+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 17532055274852714362 PartitionId: 0 SliceId: 0 Crc64: 833398431925170828 (SnapshotInfo: seqnos: 2, 0, 146036 committed:true) SnapType FORCE_COMMIT SnapAligned true SnapCreateDur 14.075703ms SnapOpenDur 55.282µs
2022-07-06T15:24:43.981+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 12084728265744008952 FlushTime 305ns CommitTime 194.802µs TotalFlushTime 0s TotalCommitTime 17.987216926s
2022-07-06T15:24:43.981+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 12084728265744008952 Creating New Snapshot SnapshotInfo: seqnos: 1, 0, 146036 committed:true
2022-07-06T15:24:43.981+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 12084728265744008952 PartitionId: 0 SliceId: 0 Crc64: 833398431925170828 (SnapshotInfo: seqnos: 1, 0, 146036 committed:true) SnapType FORCE_COMMIT SnapAligned true SnapCreateDur 12.338927ms SnapOpenDur 31.423µs
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7fecfcb4f490)
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/94f72b8b-717a-4e23-e39eac87-213f8425.dmp before terminating.
I think this bug was introduced in couchbase 7.x
Is there any solution to the problem? There is always at least one crash per day.
It’s impossible to downgrade (7.1 → 6.6) for us because of scopes / collections.
I have the same problem on Community Edition 7.1.1 build 3175
Service 'index' exited with status 134. Restarting. Messages:
2022-12-05T21:53:10.962+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 10649006615221593611 PartitionId: 0 SliceId: 0 Crc64: 2656193918834593421 (SnapshotInfo: seqnos: 86, 0, 2262 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 20.653743ms SnapOpenDur 27.777µs
2022-12-05T21:53:10.963+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 18322477779647034417 FlushTime 238ns CommitTime 376.973µs TotalFlushTime 7.514243ms TotalCommitTime 15.590804117s
2022-12-05T21:53:10.963+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 18322477779647034417 Creating New Snapshot SnapshotInfo: seqnos: 11, 11, 21854 committed:true
2022-12-05T21:53:10.963+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 18322477779647034417 PartitionId: 0 SliceId: 0 Crc64: 2656193918834593421 (SnapshotInfo: seqnos: 11, 11, 21854 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 17.850179ms SnapOpenDur 30.491µs
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7fa73a21c650)
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/ff12b14d-2b76-41c9-8d079cbb-8d2ec0a2.dmp before terminating.
@sduvuru @amit.kulkarni have you any updates?
Hi @varun.velamuri!
Can you help me with this thread?
I am still getting the following error:
Service 'index' exited with status 134. Restarting. Messages:
2022-12-17T23:36:05.537+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 10682370526375687935 PartitionId: 0 SliceId: 0 Crc64: 16428671364733376948 (SnapshotInfo: seqnos: 1187, 0, 67607 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 118.393072ms SnapOpenDur 66.423µs
2022-12-17T23:36:05.537+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 2977984481364261990 FlushTime 222ns CommitTime 432.79µs TotalFlushTime 60.897316ms TotalCommitTime 13.284413481s
2022-12-17T23:36:05.537+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 2977984481364261990 Creating New Snapshot SnapshotInfo: seqnos: 1166, 0, 67604 committed:true
2022-12-17T23:36:05.537+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 2977984481364261990 PartitionId: 0 SliceId: 0 Crc64: 16428671364733376948 (SnapshotInfo: seqnos: 1166, 0, 67604 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 115.916262ms SnapOpenDur 23.924µs
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7f00b2207150)
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp before terminating.
The result after running forestdb_dump
root@468afc6cb3c6:/opt/couchbase/var/lib/couchbase# forestdb_dump /opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 0). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 1). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 6f, SB No.: 2). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 3). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.575+00:00 [ERRO][FDB] Crash Detected: 3656 non-block aligned bytes discarded from a database file '/opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp'
2022-12-18T13:19:47.575+00:00 [ERRO][FDB] Crash Detected: Last Block not DBHEADER 20 in a database file '/opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp'
2022-12-18T13:19:47.604+00:00 [ERRO][FDB] Error in READ on a database file '/opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp', errno = 2: 'No such file or directory'
Unable to open /opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp
@voskobovich I am not sure of what is causing this issue. In general, ForestDB (the storage engine for community edition) has limitations with regard to the support for large keys. Do you have any large keys in your index - Like, the indexed entries are more than 2KB? In such cases, you need to rework your index to make it smaller.
I will re-direct this question to one of our ForestDB experts for a better answer.
Thanks,
Varun
@varun.velamuri thanks for your answer.
I can’t exactly answer your question. How can I find the max size of the document in all buckets and all collections? Maybe couchbase dashboard can show this?
I found two links with info about the max size of the key and document in ForestBD: first and second
The maximum sizes of a key, metadata, and value are 64KB, 64KB, and 4GB, respectively. They are all simply processed as binary objects.
How did you find out about 2 KB per entity?
@voskobovich What I mean is indexed entry. If you have a composite index with many fields, then each indexed entry will be large.
You can calculate the size of an indexed entry approximately by doing data_size / items_count stats - Both these stats are available per index on UI and also on indexer_stats.log. If there are many composite fields and the indexed entry is going to grow larger, then you may have to rework your index.
Thanks,
Varun
@varun.velamuri Thanks for the clarification.
Max size of the index entry in my server is 50 bytes. Most likely this is not the problem.
Two days ago, I saw that the index service had 99 MB of RAM left out of 512 MB. I added 256 MB. A total of 768 MB of RAM for the index service. Since then, there have been no such errors.
Can out-of-memory cause the errors described above?
@voskobovich I am glad that it solved the issue. Unfortunately, I have little knowledge of how ForestDB storage engine works to comment if this issue is due to out-of-memory.
Probably @sduvuru can share more insights.
Forestdb does not handle memory failures well. This does create random memory corruption and crashes. I think the right sizing of memory will resolve those issues.