C# couchbase sdk best way to bulk load data into a bucket

.NET SDK
1

I need to load @50,000 items from different unix directories into couchbase. My first attempt was to get an item process it and then write the processed data to couchbase (using a method that receives the data, open a couchbase connect and upsert the data) while this DOES WORK AFTER @ 5000 items the insert slow down and eventually stop/hang.

I then tried a different approach of opening the connection once and then looping through the directory and again building my record and upserting it but now I am getting an error:

Exception = {“A request has been made for a service that is not configured or supported by the cluster. Please check the cluster and enable or add a new node with the requested service: Data.”}

what wierd is that if I use the old program it upserts just fine and the code is identical except in one case the connection is opened per record and in th eother I open once and then try to do the upsert. They use the SAME BUCKET on the same cluseter and system.

below is the c# code

var cluster = new Cluster(new ClientConfiguration
{
Servers = new List { new Uri(“http://system”) }
});

        var authenticator = new PasswordAuthenticator("user", "passwd");
        cluster.Authenticate(authenticator);
        var bucket = cluster.OpenBucket("uvCodeInfo");

        var document = new Document<dynamic>
        {
            Id = id,
            Content = new
            {
                read = readList,
                readu = readuList,
                call = callList,
                execute = executeList,
                clearselect = clearselectList,
                mod = modList,
                write = writeList,
                writeu = writeuList,
                path = path,
                lasteWrite = stamp
            }
        };

        var upsert = bucket.Upsert(document);
        cluster.Dispose();

any advice or suggestions are welcomed

thanks in advance

2

Did you find a solution to this issue ?
I have the same problem.

3

@dougc

First of all, the approach of using a single cluster and bucket object for the lifetime of the bulk load is the correct one. The Cluster and Bucket objects are both designed to work as a singleton for the lifetime of the application.

As to problems at scale, the first thing I’d look for is problems on the server side, rather than the client side. In particular, Couchbase Server is a memory-first system. This means that it accepts your mutations into memory and then returns a success code without ever writing it to disk (unlike typical RDBMS systems). While this makes most operations much faster, it can cause a scenario where available memory is depleted faster than documents can be written to disk, which can then cause failures.

I’d recommend watching the Statistics on your bucket in the Console while running your bulk import. In particular, watch the “disk write queue” and “temp OOM per sec.” metrics.

4

@dougc -

Exception = {“A request has been made for a service that is not configured or supported by the cluster. Please check the cluster and enable or add a new node with the requested service: Data.”}

This is caused by a bug - work around is to use 2.7.6: https://issues.couchbase.com/browse/NCBC-2002

-Jeff

5

thank you switching to version 2.7.6 fixed it, again thanks