I have a fairly high-volume application that uses Couchbase to store and retrieve real-time score and rankings calculations.
This morning, after virtualizing the the hardware my Golang app runs on, one of my app nodes started throwing this error when trying to Upsert to Couchbase:
Has anyone run into this error before? Is this a Couchbase error (I don’t think so as my other app nodes were operating just fine), or a Golang SDK error?
Any advice on preventing this from happening again?
This indicates that you are submitting operations to gocb faster than we are capable of handling them. This is normal behaviour and you should try to reduce the rate at which you are performing operations to a level which can be handled (rather than sending 10,000 operations at once, spread them out a bit to allow the network IO capabilities to keep up).
Throttling is going to have to be my last resort, I need this stuff as instant as possible.
Right now I’m thinking that spinning up an additional application node could help alleviate this issue, as long as it is not a result of something happening on the Couchbase server side.
Outside of those 2 options, is there anything else I could do to increase that throughput? More Couchbase nodes, network tuning, etc.? I don’t fully understand the internals of gocb, but it sounds like you are telling me that this is a limit of network IO that causes gocb to overflow, is that correct?
@noisewaterphd – assuming you are running on Linux, can you try the networking setting change documented in this issue and let me know if it changes anything for you?
traun, thanks for the suggestion.
I am not sure if our systems team made any changes directly related to your link, but they did find that after virtualizing our nodes we were still using TCP offloading, which was slowing down the network quite a bit.
After turning off the TCP offloading we saw a good increase in speed, but we are still investigating ways to make it even faster without adding additional nodes.