lcb_create as a Singleton instance

Hey there,

We are using Couchbase cluster for a high-load ad-serving system.

thousands of HTTP requests are coming to our load balancer in a second.
Every HTTP request is handled by one of our application server (which is web server with couchbase C client)

We have written a C code that loaded into our HTTP web server, and every request is handled by the Web server, being filtered with the relevant parameters and headers, and then it will go with the Couchbase C client to the cluster to get a specific documnet and to increase it's counter document.

For every single HTTP request we are using "lcb_create" to create the instance and then "lcb_connect" if the instance was created with no errors.

I'm experiencing an hectic problem with executing this code thousands of times in a second.
every ~400-500 requests we are getting a connection error:

"ERROR: Connection failure (0x18), Could not connect to server within allotted time"

seems like the couchbase cluster can not handle this amount of connections per second.
This is the current code we are using for connecting the cluster:

    err = lcb_create(&instance, &create_options);
    if (err != LCB_SUCCESS) {
        syslog(LOG_DEBUG,"Failed to create libcouchbase instance: %s\n", lcb_strerror(NULL, err));
        return 1;
    }
    (void)lcb_set_error_callback(instance, error_callback);
    /* Initiate the connect sequence in libcouchbase */
    if ((err = lcb_connect(instance)) != LCB_SUCCESS) {
        syslog(LOG_DEBUG,"Failed to initiate connect: %s\n", lcb_strerror(NULL, err));
        lcb_destroy(instance);
        return 1;
    }

As per my understanding, "lcb_create" and "lcb_connect" are two methods that require large amount of resources from the cluster , and therefore the couchbase is having diffucults to respond, and thats why we are getting this connections errors.
Maybe we should somehow gather our connections into one instance and do not execute "lcb_create" and "lcb_connect" for every single request.

Is there anyway to implement it in C code ?
Is it possible to check if there is already an instance that was created, and to use it instead of creating an instance every time ?

Maybe "lcb_create_io_ops" can help here ?

* Just to clarify
the cluster is not cpu-loaded. the servers are fine in terms of IOPS\ CPU RAM and SWAP.

we are really frustrated,
we would like to hear any opinion from you.

Thank you very much.

3 Answers

« Back to question.

Depending on your application you could either use a singleton or a connection pool. One thing to bear in mind is that libcouchbase don't use _ANY_ locking internally, so you as a user have to ensure that you use it in a safe way in your application. The good news is that libcouchbase don't use any global values, so you are safe as long as you don't use the same lcb_t from multiple threads at the same time.

If your application have many threads that once in a while needs to access the Couchbase server a resource pool may be what you want. All you need to do is to create a number of instances and store them in a "pool" somewhere. Now whenever you need to perform an operation you allocate a preconfigured instance from the pool to preform your operation before you return this to the pool.

I wrote an example of a connection pool yesterday that you may use to pick some ideas from:
https://github.com/couchbase/libcouchbase/tree/master/example/instancepool

« Back to question.

Hi

Obviously you shouldn't create new handle each time and connect it during the each HTTP request to your application.

* lcb_create just allocates and initialize memory for structs
* lcb_connect is doing actual connection to the cluster and fetch cluster config and topology using HTTP protocol
* next operation like lcb_increment will also establish new connection to the data node (only first call)

I've done something similar what you are describing. You probably heard about nginx webserver, and I've implemented module for that, you can use it as a distributed cache, or expose couchbase interface as HTTP.

Here is the home page of this experiment
http://labs.couchbase.com/couchbase-nginx-module/

Back to your question, usually it should be enough to create and connect single lcb_t object, and use it for the future operations. If your application is using threads and shared memory, you should implement a queue between IO thread (which will own lcb_t and all callbacks) and the rest application.

« Back to question.

Hey,

This is the very common problem of couchbase . In my company also we are tired of this issue. We seek information from the couchbase team (US) but they suggested to increase the servers node & hardware capacity . We also got the same problem with JAVA & PHP client. Still we are searching for some new things.If your company is using Couchbase then please share the idea so that I can analyze and share some new things.

Sashi
Mumbai