@abhinav thanks for looking at this.
We have identified the root cause of this issue which was a memory pressure.
We’ve set the memory quota for FTS service to 10000MB and this issue started to happen when the FTS memory usage per node was approaching 8000MB.
Closer look at FTS logs revealed a lot of warnings like this:
2022-06-22T15:23:05.928+00:00 [INFO] app_herder: indexing over indexQuota: 7680000000, memUsed: 8453991341, preIndexingMemory: 34933, indexes: 21, waiting: 43
2022-06-22T15:23:05.928+00:00 [INFO] app_herder: indexing over indexQuota: 7680000000, memUsed: 8453991341, preIndexingMemory: 34933, indexes: 21, waiting: 43
which after some deduction allowed to realize that FTS doesn’t allow indexes to grow up to entire quota, but only allows them up to 75% of the quota (reserving the remainin memory chunk for other needs???).
So, once we increased the quota to 12000MB, the issue is now gone.
However, this high memory usage is quite concerning:
- the bucket for an index currently only has around 22000 records, using just 272MB of space; of them only around 8000 records are eligible for the FTS index.
- the index size on disk fluctuates around 32-36MB, which is fine for this small amount of data.
- the number of data modification operations per second is around 20 with seldom spikes to 1500.
- the number of search queries is extremely small, like a few runs per minute at most.
- the FTS service memory usage reported by dashboard is 8-9GB on each of the 3 nodes in the cluster. This is clearly strange having the small amount of data and index size on disk.
Index definition attached below. As you can see, we only index/store a couple of fields, all these fields are controlled for max length (current maximum is 51 characters, average 26 characters).
(Note that FTS log size on each node currently exceeds 23MB of data; collecting logs via dashboard would take ages; but I could share a smaller extract of the latest logs if you point me as to which files and how much log lines (e.g. 10000) would be enough to look into this).
Could you suggest what we could be doing wrong with the index definition?
{
"type": "fulltext-index",
"name": "device-meta-devices-list-v1",
"uuid": "1ff6d1afddce0312",
"sourceType": "couchbase",
"sourceName": "device-meta",
"planParams": {
"maxPartitionsPerPIndex": 32,
"indexPartitions": 32,
"numReplicas": 1
},
"params": {
"doc_config": {
"docid_prefix_delim": "",
"docid_regexp": "",
"mode": "type_field",
"type_field": "doctype"
},
"mapping": {
"analysis": {},
"default_analyzer": "keyword",
"default_datetime_parser": "disabled",
"default_field": "_all",
"default_mapping": {
"dynamic": false,
"enabled": false
},
"default_type": "_default",
"index_dynamic": false,
"store_dynamic": false,
"type_field": "_type",
"types": {
"device": {
"dynamic": false,
"enabled": true,
"properties": {
"group_id": {
"dynamic": false,
"enabled": true,
"fields": [
{
"index": true,
"name": "group_id",
"type": "text"
}
]
},
"name": {
"dynamic": false,
"enabled": true,
"fields": [
{
"docvalues": true,
"index": true,
"name": "name",
"type": "text"
}
]
},
"owner": {
"dynamic": false,
"enabled": true,
"fields": [
{
"docvalues": true,
"index": true,
"name": "owner",
"type": "text"
}
]
},
"repo_id": {
"dynamic": false,
"enabled": true,
"fields": [
{
"index": true,
"name": "repo_id",
"type": "text"
}
]
},
"tag": {
"dynamic": false,
"enabled": true,
"fields": [
{
"index": true,
"name": "tag",
"type": "text"
}
]
},
"uuid": {
"dynamic": false,
"enabled": true,
"fields": [
{
"index": true,
"name": "uuid",
"type": "text"
}
]
}
}
}
}
},
"store": {
"indexType": "scorch"
}
},
"sourceParams": null
}
