Hi,
I run Couchbase 7.1.0 on Ubuntu 20.04. The following errors occurs:
Service 'index' exited with status 134. Restarting. Messages:
2022-06-27T13:08:13.287+00:00 [Info] RebalanceServiceManager::rebalanceJanitor Running Periodic Cleanup
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7f80db219a80)
2022-06-27T13:08:14.897+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 15031182372585180544 FlushTime 173ns CommitTime 3.160754ms TotalFlushTime 1.4252ms TotalCommitTime 5m43.882682393s
2022-06-27T13:08:14.897+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 15031182372585180544 Creating New Snapshot SnapshotInfo: seqnos: 223, 0, 69883 committed:true
2022-06-27T13:08:14.898+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 15031182372585180544 PartitionId: 0 SliceId: 0 Crc64: 833398431925170828 (SnapshotInfo: seqnos: 223, 0, 69883 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 39.957733ms SnapOpenDur 134.563µs
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp before terminating.
I have two nodes and 200 documents. This error is causing com.couchbase.client.core.error.InternalServerFailureException: Internal Couchbase Server error in my java app sometimes.
2022-07-04T18:32:30.545+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 0). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.545+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 1). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 38, SB No.: 2). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 3). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Crash Detected: 2064 non-block aligned bytes discarded from a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp'
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Crash Detected: Last Block not DBHEADER 0 in a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp'
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Error in READ on a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp', errno = 2: 'No such file or directory'
Unable to open /opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp
Is there any solution to the problem? There is always at least one crash per day.
It’s impossible to downgrade (7.1 → 6.6) for us because of scopes / collections.
root@468afc6cb3c6:/opt/couchbase/var/lib/couchbase# forestdb_dump /opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 0). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 1). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 6f, SB No.: 2). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.574+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 3). Note: this message might be a false alarm if upgrade is running.
2022-12-18T13:19:47.575+00:00 [ERRO][FDB] Crash Detected: 3656 non-block aligned bytes discarded from a database file '/opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp'
2022-12-18T13:19:47.575+00:00 [ERRO][FDB] Crash Detected: Last Block not DBHEADER 20 in a database file '/opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp'
2022-12-18T13:19:47.604+00:00 [ERRO][FDB] Error in READ on a database file '/opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp', errno = 2: 'No such file or directory'
Unable to open /opt/couchbase/var/lib/couchbase/crash/a890dd85-bdc5-4122-f9fb80b7-c5001612.dmp
@voskobovich I am not sure of what is causing this issue. In general, ForestDB (the storage engine for community edition) has limitations with regard to the support for large keys. Do you have any large keys in your index - Like, the indexed entries are more than 2KB? In such cases, you need to rework your index to make it smaller.
I will re-direct this question to one of our ForestDB experts for a better answer.
I can’t exactly answer your question. How can I find the max size of the document in all buckets and all collections? Maybe couchbase dashboard can show this?
I found two links with info about the max size of the key and document in ForestBD: first and second
The maximum sizes of a key, metadata, and value are 64KB, 64KB, and 4GB, respectively. They are all simply processed as binary objects.
@voskobovich What I mean is indexed entry. If you have a composite index with many fields, then each indexed entry will be large.
You can calculate the size of an indexed entry approximately by doing data_size / items_count stats - Both these stats are available per index on UI and also on indexer_stats.log. If there are many composite fields and the indexed entry is going to grow larger, then you may have to rework your index.
Max size of the index entry in my server is 50 bytes. Most likely this is not the problem.
Two days ago, I saw that the index service had 99 MB of RAM left out of 512 MB. I added 256 MB. A total of 768 MB of RAM for the index service. Since then, there have been no such errors.
Can out-of-memory cause the errors described above?
@voskobovich I am glad that it solved the issue. Unfortunately, I have little knowledge of how ForestDB storage engine works to comment if this issue is due to out-of-memory.
Forestdb does not handle memory failures well. This does create random memory corruption and crashes. I think the right sizing of memory will resolve those issues.
I have the exact same issue on 7.1.5.
Crashing once or twice a day.
I have lot of RAM left for indexes (5GB).
Indexes are quite small (17-20K items).
Key sizes are under 2KB.
I’m seeing this issue since quite a lot of time but it’s just now affecting my application’s behavior.
This is seriously impacting reliability of my application.
2023-09-08T13:05:09.275+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 0). Note: this message might be a false alarm if upgrade is running.
2023-09-08T13:05:09.275+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 1). Note: this message might be a false alarm if upgrade is running.
2023-09-08T13:05:09.275+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 2). Note: this message might be a false alarm if upgrade is running.
2023-09-08T13:05:09.276+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 3). Note: this message might be a false alarm if upgrade is running.
2023-09-08T13:05:09.276+00:00 [ERRO][FDB] Crash Detected: 1040 non-block aligned bytes discarded from a database file 'f305cba1-1f0c-46c2-c78ed2a7-94714f47.dmp'
2023-09-08T13:05:09.276+00:00 [ERRO][FDB] Crash Detected: Last Block not DBHEADER 0 in a database file 'f305cba1-1f0c-46c2-c78ed2a7-94714f47.dmp'
2023-09-08T13:05:09.277+00:00 [ERRO][FDB] Error in READ on a database file 'f305cba1-1f0c-46c2-c78ed2a7-94714f47.dmp', errno = 2: 'No such file or directory'
Unable to open f305cba1-1f0c-46c2-c78ed2a7-94714f47.dmp
@sduvuru can I provide any more info to you to diagnose the issue ?
Thanks a lot for your message. I just changed this setting. I’ll let you know if it solves the issue for me too.
Thanks again for taking the time to help me
I’ve had the same issue on 7.2.2 with indexer crashing because of RAM.
On another server on 7.1.5 no issue on this side, however it didn’t fix the issue completely. It happens less often for sure but it still did happen this morning.
I’m monitoring this with a few variations in my config but if I can’t find any way of getting it to work I’ll make a new dedicated thread on that matter.
I’ll consider changing database if this doesn’t get solved because for the time being I don’t have a stable software because of this issue…
Did you find any solution for that? We are experiencing the same issue and also do have lots of RAM left with small indexes and keys. For us it is happening multiple times a day and it heavily affects our application by now.
