Hi there
I’ve just started using official golang sdk for couchbase.
I am inserting documents from threads, by creating them maually. I am doing it for around 10K threads, this is 10K insert operations.
However, each time only 2048 documents gets inserted and rest would result into a queue overflowed error.
Did a lot of google, and look around, but all in vain.
Did someone has similar situation, and found a solution?
Hey @darshanlila,
For safety reasons, the internal queues that the Go SDK uses before operations are dispatched to their respected servers are bounded. This ensures that the SDK does not get overloaded to the point of causing a downwards spiral of request timeouts and allows the application to actually detect these cases and push back on the clients rather than failing less gracefully.
These queue sizes are hard coded in the client at 2048 items per server. The correct way to handle these overload errors is to sleep for a short period of time and attempt to dispatch the operation again.
P.S. I’ll mention here that, while we would strongly suggested against it, it is possible to tune the queue size via the connection string with something like:
cluster, _ := gocb.Connect("couchbase://1.2.3.4?max_queue_size=4096")
Cheers, Brett
Hey Brett,
Thanks for the answer, I tried to go as you suggested (increase max queue size), however still the same issue persists. I also tried sleeping for a little while, but won’t help much.
Any suggestions?