High Memory Allocation gocbcore.acquireReadBuf

Hello, we are using Couchbase Golang SDK for our Kafka consumer projects.

We observed that as soon as our pods are running, gocb allocates lots of memory therefore we faced serious memory consumption in our projects.

We decided to ask about this problem.

Our resources for these projects like

resources:
requests:
memory: 256M
cpu: 250m
limits:
memory: 1G
cpu: 1500m

When we connect two different bucket, this allocation doubled.

is it possible to configure this behaviour or any recommendation about that?

gocb: v2.4.0 (we also try to upgrade and downgrade this version)
gocbcore/v10 v10.1.0` (we also try to upgrade and downgrade this version)

1 Like

Hi @Abdulsamet_Ileri this is because each connection we create allocates a 20MB buffer (same size buffer as used by KV engine). This was done to reduce CPU usage. If you have a really large number of connections or a low memory environment then this can be an issue.

Note that I’m unsure if this is strictly supported but you can alter this buffer size by using the connection string query string parameter of “kv_buffer_size”, the unit is bytes. e.g. couchbases://myaddress?kv_buffer_size=<somesizeofbytes>. Unfortunately I can’t really offer advice on what to set that value to (but it can’t be 0, 0 will just set it to the default value) so you may have to do some testing there.

I would also advise against running gocb with a version of gocbcore other than it specifies itself. Each release of gocb is only tested (and supported) against the specific version of gocbcore specified in the gocb go.mod file.

3 Likes