Couchbase hangs with error in indexer

Couchbase Server
1

Hello,

every while, our Couchbase (CE 7.2.2) hangs with all indexes staying in rebuild mode. We can’t find a cause…

Did someone have similar problems or have a clue what is wrong ?

Thank you !

In log there are errors like this:

Service ‘index’ exited with status 2. Restarting. Messages:
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.20.6/go/src/net/http/transport.go:1943 +0x4e fp=0xc004d3dce8 sp=0xc004d3dc88 pc=0x6f566e
bufio.(*Reader).fill(0xc0062f1440)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.20.6/go/src/bufio/bufio.go:106 +0xff fp=0xc004d3dd20 sp=0xc004d3dce8 pc=0x662f1f
bufio.(*Reader).Peek(0xc0062f1440, 0x1)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.20.6/go/src/bufio/bufio.go:144 +0x5d fp=0xc004d3dd40 sp=0xc004d3dd20 pc=0x66307d
net/http.(*persistConn).readLoop(0xc00671aa20)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.20.6/go/src/net/http/transport.go:2107 +0x1ac fp=0xc004d3dfc8 sp=0xc004d3dd40 pc=0x6f648c
net/http.(*Transport).dialConn.func5()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.20.6/go/src/net/http/transport.go:1765 +0x26 fp=0xc004d3dfe0 sp=0xc004d3dfc8 pc=0x6f4c26
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.20.6/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc004d3dfe8 sp=0xc004d3dfe0 pc=0x4732e1
created by net/http.(*Transport).dialConn
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.20.6/go/src/net/http/transport.go:1765 +0x16ea

Service ‘index’ exited with status 134. Restarting. Messages:
2024-01-04T10:21:34.582+00:00 [Info] CompactionDaemon: Finished compacting index instance:10229859520340259200
2024-01-04T10:21:34.582+00:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8928177170886921143 Data:62702612, Disk:182693888, ExtraSnapshotData:0, Fragmentation:65.67886715509607%, mode : circular
2024-01-04T10:21:34.582+00:00 [Info] CompactionDaemon: Compacting index instance:8928177170886921143
2024-01-04T10:21:34.597+00:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 99194. Slice Id 0, IndexInstId 8928177170886921143, IndexDefnId 10376223828848498437
2024-01-04T10:21:34.598+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/i4c_#primary_8928177170886921143_0.index/data.fdb.17
assertion failed [item->txn_id == txn->txn_id] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/wal.cc:1255 (0x65652d383230392d != (nil))
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/7825ec07-eb1c-48a8-515cc1a6-0638936c.dmp before terminating.

2

When I search issues.couchbase.com for: text ~ “go/src/net/http/transport.go:1943”, I find Loading... which is closed as a duplicate of
Loading..., which is closed as fixed in 7.2.4 (it also points to Log in - Couchbase database and Log in - Couchbase database )

1 Like
3

Thank you mreiche!
We will check those tickets. Unfortunately i don’t have access to last two JIRA links.
Also I guess 7.2.4 is not released yet, since last version i can find is 7.2.3… We will try our luck with 7.2.3 for now…

4

The issue is related running out of memory. The fix is related to what happens when a critical memory threshold is reached. So allowing more memory might help avoid the issue.

5

Are there any specific settings to increase memory for this ? Because the database has allocated 16GB of RAM and by the seeable statistics it uses only a few hundred of MB…

6

@matko , you can check how much memory quota has been allocated to the index service under UI->Settings (documentation).

7

Thank you for your answer deepkaran.

I think 12GB should be much than enough ?
image

8

hi @matko
I see that the 2nd crash is a storage crash in ForestDB. can you share the crash dump file (whose path is mentioned in the message)?
For the first crash, might need to look at a larger stack trace. can you collect logs for the index node and upload them too?

regarding memory settings, you can refer to Monitor Indexes | Couchbase Docs; these stats will show the current usage by indices created; if you the RAM Used/Remaining remaining is low, maybe you can try increasing the memory quota for index service

9

Thank you @dhruvilshah98 !

Here is mentioned dump file:
7825ec07-eb1c-48a8-515cc1a6-0638936c.zip (39.9 KB)

Can you please point me, where to find logs for index node ?

10

thanks!
info about log collection can be found at Manage Logging | Couchbase Docs

11

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.