Currently running on couchbase-server-community-6.6.0 and we’ve noticed that the search service is at max utilization. We do not have any Full Text Search indexes or queries. We are unable to turn off the service without doing a swap rebalance of new nodes. Is there anything we can do to mitigate this issue?
This is quite likely due to a memory leak in 6.5.x and 6.6 that has been addressed in 6.6.3+. This same issue is referenced here as well: Abnormally large jumping num_mutations_to_index in FTS index while it is 100% ready - #17 by abhinav .
I’d strongly urge moving to the latest release in 6.6.x.
To immediately mitigate this issue - I’d try killing the cbft process running on each of your nodes, which should free up the memory that the cbft process is hoarding but has lost reference to.
Tried bouncing cbft and even bounced the entire couchbase-server process on one node but it still has the same behavior.
The strange thing is that there are no FTS queries running against 8094, nor is there any FTS Indexes created. Is the memory leak also expected with no FTS activity?
Tried bouncing cbft and even bounced the entire couchbase-server process on one node but it still has the same behavior.
Not sure what you mean by bouncing the entire couchbase-server process. The server runs collectively on several processes:
The leak I mention occurs with the search engine in a certain situation - not closing some connections (before releasing their reference) that were opened for indexes, there by the process leaking those resources regardless of the indexes being shut down. Killing the cbft process would certainly release those resources back to the OS. The cluster manager will be responsible for automatically bringing back the service when taken down.
If after restarting the process, you’re still seeing the search service hold on to the memory, then I’ll want to take a look at the logs (run cbcollect_info) and since you are in 6.6.x - I will need you to separately collect the heap profile of the process as well. You can do so by running the following command after logging into the suspect node …
curl -xPOST -u <username>:<password> http://127.0.0.1:8094/api/runtime/profile/memory
This command returns immediately and would’ve created a file called run-memory.pprof somewhere within /opt/couchbase/var/...
I would also urge you to move to a later version of couchbase-server, perhaps 7.1.1 which is available in the community edition.