Hi,
We have a Couchbase node (Couchbase Server Community Edition 7.1.1 build 3175) on Ubuntu 20.04 (with RAM 16GB, Disk > 100GB), and a Python FastAPI application on another Ubuntu.
When querying Couchbase in the APP, sometimes we encounter a Broken Pipe error message thrown by the Couchbase Python SDK. Upon checking the log on the Couchbase server, we observe multiple Service index exited with status 134, restarting message at the time of the error occurrence.
It would be highly appreciated if someone could offer a potential solution to this issue.
Couchbase Log
Service 'index' exited with status 134. Restarting. Messages:
InstanceId: 4652489839859031334 PartitionId: 0 Endpoints: [:9105]
InstanceId: 1972920516647907182 PartitionId: 0 Endpoints: [:9105]
InstanceId: 18036580604677429716 PartitionId: 0 Endpoints: [:9105]
InstanceId: 16287220193423244923 PartitionId: 0 Endpoints: [:9105]
InstanceId: 538494393000860933 PartitionId: 0 Endpoints: [:9105]
InstanceId: 9875337341189051107 PartitionId: 0 Endpoints: [:9105]
InstanceId: 12321076852208181193 PartitionId: 0 Endpoints: [:9105]
InstanceId: 11911987649747087861 PartitionId: 0 Endpoints: [:9105]
InstanceId: 14270152523609621505 PartitionId: 0 Endpoints: [:9105]
InstanceId: 10070622683191153988 PartitionId: 0 Endpoints: [:9105]
InstanceId: 9864256008861178045 PartitionId: 0 Endpoints: [:9105]
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7ffb246971d0)
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/a995a1a0-f1c2-4cc3-14fe3ab1-75203d23.dmp before terminating.
Application Log
<ec=5, category=couchbase.common, message=internal_server_failure (5), context=QueryErrorContext({
'client_context_id': 'bd53d0c6-404b-4978-8b70-c8fa94ec0e3c',
'context_type': 'QueryErrorContext',
'first_error_code': 5000,
'first_error_message': ' write tcp 127.0.0.1:37612->127.0.0.1:9101: write: '
'broken pipe from [127.0.0.1:9101] - cause: write tcp '
'127.0.0.1:37612->127.0.0.1:9101: write: broken pipe '
'from [127.0.0.1:9101]',
'http_body': '{"requestID": "10e9e81e-6326-4798-873c-da5ce2c782ce", '
'"clientContextID": "bd53d0c6-404b-4978-8b70-c8fa94ec0e3c", '
'"signature": {}, '
'"results": [ ], "errors": [{"code":5000,"msg":" write tcp '
'127.0.0.1:37612->127.0.0.1:9101: write: broken pipe from '
'[127.0.0.1:9101] - cause: write tcp '
'127.0.0.1:37612->127.0.0.1:9101: write: broken pipe from '
'[127.0.0.1:9101]"}], "status": "errors" } ',
'http_status': 500,
'last_dispatched_from': '172.20.0.15:38816',
'last_dispatched_to': '10.100.150.142:8093',
'method': 'POST',
'parameters': '{"client_context_id":"bd53d0c6-404b-4978-8b70-c8fa94ec0e3c","metrics":false,"scan_consistency":"request_plus","statement":"\n'
' SELECT something FROM `bucket`.`scope`.`collection` WHERE...'
' ","timeout":"74500ms"}',
'path': '/query/service',
'retry_attempts': 0,
'statement': 'SELECT something FROM `bucket`.`scope`.`collection` WHERE...'
}
), C Source=/home/ec2-user/workspace/python/sdk/python-manylinux-wheel-pipeline/couchbase-python-client/src/n1ql.cxx:323>