Inconsistent SDK response

Java SDK
1

Having a weird behavior on Java SDK 2.2.7,2.2.8,2.3.2,2.3.0:
running this simple test:

Observable.range(1, 100, Schedulers.computation())
                .concatMap(i -> bucketProvider.getBucket().async()
                        .counter("test::" + i, 1, 1, PersistTo.NONE)
                        .map(doc -> i)
                )
                .toBlocking()
                .subscribe(i -> System.out.println("Successful " + i),
                        throwable -> throwable.printStackTrace());

just hangs on one machine and works correctly on another.
Behavior is following:
either it hangs completely (70% chance)
either it shows: com.couchbase.client.java.error.DocumentDoesNotExistException: null on counter which has initialization parameter (10% chance)
either it works normally (20% chance)
if counter already exists it properly increments it.
This behavior begun after updated target machine from 4.1 to 4.5.

What can it be and how to solve this?

Both machines are Ubuntu 14.04 (on is on EC2, another is local) running 4.5.0-2601 Enterprise Edition (build-2601), firewalls disabled. No issues seen on JSON docs or any other functionality, just counters. One difference between machine exists: CPU on EC2 is about 95% used permanently by ‘memcached’ process
Thank you for your time.

2

Simplified test and now it fails on both 4.5 instances:

Observable.range(1, 100, Schedulers.computation())
                .concatMap(i -> bucketProvider.getBucket().async()
                        .counter("test::" + i, 1)
                        .map(doc -> i)
                )
                .toBlocking()
                .subscribe(i -> System.out.println("Successful " + i),
                        throwable -> throwable.printStackTrace());

According to SDK it looks like server is not properly processing Counter requests responding with NOT_FOUND, instead of initializing it.

3

That second test doesn’t use the overload that provides an initial value for the case where the counter doesn’t exist, thus it’ll always throw an exception. But the first test looks correct…

So the exact same test on 4.1 doesn’t hang nor produce DocumentDoesNotExistException?

4

Yes, on 3.0, 4.0 and 4.1 there were no such issues with counters.
As for using inproper method, if you’ll check implementation counter is actually initialized with zero.

I’m asking you to check counters initialization behavior on 4.5 release, because I’m totally lost and reproduced this behaviour on 4 different machines (size and collocation differs).
Now I want to install fresh 4.5 (not rolling update from previous installations) aand see what will happen.

Thank you for your time.

5

For the counter(key, delta) call setting an initial value: no it really doesn’t since version 2.2.0. if you don’t provide the initial value, 0 is used in the code BUT it also sets a special expiry (COUNTER_NOT_EXISTS_EXPIRY) which will let the server know that a NOT_EXIST error should be returned if the counter doesn’t exist.

See here in the code and see issue JCBC-784

6

From the part where it hangs, you might be hitting a bug around counters that hangs during full eviction. https://issues.couchbase.com/browse/MB-17231

It affected 4.1.0 though, was fixed in 4.1.1 but a regression re-introduced it in 4.5.0. 4.5.1 has the fix again…

7

Thanks for investigation, but it seems I’m totally lost.
Checking usage of COUNTER_NOT_EXISTS_EXPIRY constant shows it’s used only to calculate ‘expiry’ at https://github.com/couchbase/couchbase-java-client/blob/2654a7ab26e25e7cbfa19b6eb9bb0055c2877bff/src/main/java/com/couchbase/client/java/CouchbaseAsyncBucket.java#L877

and anyway this code works.

8

OMG, THANKS!!!
That bucket is actually a full evicted, turning it off and will check how it behaves.

9

Tested.
Machine which was initially lagging (turned off full eviction) runs completely fine

Observable.range(1, 100, Schedulers.io())
                .observeOn(Schedulers.io())
                .concatMap(i -> bucketProvider.getBucket().async()
                        **.counter("test::" + i, 10, 1)**
                        .subscribeOn(Schedulers.io())
                        .map(doc -> doc.id())
                )
                .subscribeOn(Schedulers.io())
                .toBlocking()
                .subscribe(i -> System.out.println("Successful " + i),
                        throwable -> throwable.printStackTrace());

but still fails on

Observable.range(1, 100, Schedulers.io())
            .observeOn(Schedulers.io())
            .concatMap(i -> bucketProvider.getBucket().async()
                    **.counter("test::" + i, 10)**
                    .subscribeOn(Schedulers.io())
                    .map(doc -> doc.id())
            )
            .subscribeOn(Schedulers.io())
            .toBlocking()
            .subscribe(i -> System.out.println("Successful " + i),
                    throwable -> throwable.printStackTrace());

BUT, code without initialization parameter not being set works perfectly fine on another 4.5 instance O_o

10

That last part sounds very strange to me. Are you sure the counters don’t exist on this other instance, as a remnant of a previous test run on it for example?
Maybe you can map to both the doc.id() but also the doc.content() to see the value of the successful counter incrementation?

11

You’re correct.
Just missunderstood current test case, so everything works fine:

  1. if initialization parameters set - counters create correctly
  2. if not set - getting NOT_EXISTS exception

Thank you for your time.