Just to say - since this first post yesterday, I’ve gone through a lot of stuff, none of it has made any difference.
The Lambda function, once started, stays ‘alive’ between invocations - all global variables are maintained, including the couchbase DB cluster connection, and the resulting bucket and collection references.
Each subsequent invocation uses the existing connection, without having to re-open a new connection.
I’m setting context.callbackWaitsForEmptyEventLoop = false which helps the invocation exit properly.
The error 201 I’m getting from couchbase seems to imply that it’s waiting for something to be concluded, but there are no user defined timeouts that I’m aware of?
I’m deploying the code to the lambda function, and it runs perfectly for about 2 hours, roughly 82 invocations based around 1 container.
Then Lambda dumps the container (perfectly normal) and a new container is created, obviously based around the same nodejs code. This time however, right from the first invocation, I’m getting the LCB_ERR_TIMEOUT error.
I’m wracking my brains, but can’t see why this would happen.
Is couchbase running out of connections? Is the old connection somehow being held open with no timeout? Can I just ‘hack’ this and open and close the connection with every invocation? I’m using SDK3.0 and there doesn’t seem to be any command for closing a cluster connection?
It might be AWS related. They introduced some new networking for Lambda functions
There’s new idle connection timeout introduced of 350 seconds (~6 minutes). So if the connection was to be idle for 350 seconds or more, lambda would purge the connection. So any attempt to use that connection on an invocation after 350 seconds or so can trigger connection resets. By default Couchbase SDK heartbeat keep alive is set at 2 hours. You might try reducing TcpKeepAliveTime to a more reasonable timeframe, such as 30 seconds.
" If lcb_get_bootstrap_status is returning with LCB_ETIMEDOUT and you are running on a slow network, modifying this setting may increase the chances of success."
I appended my connection string
couchbase://test.com/bucketName?config_total_timeout=25 and was able to fix it. Looks like its couchbase client failing to get connection config from server in time. You may have a connectivity issue reaching your couchbase server