YCSB N1QL loading is not working

WenTat · January 8, 2020, 10:11am

Hi guys, I am trying to use YCSB to evaluate Couchbase Enterprise (5.11). In YCSB, you can set couchbase.kv = false to let it run N1QL queries instead of normal get/set. I have encountered problem running workload a using couchbase.kv=false. I have no problem running benchmarks using normal couchbase.kv=true. Following is my stack trace:

com.couchbase.client.core.CouchbaseException: Error while preparing plan
at com.couchbase.client.java.query.core.N1qlQueryExecutor$10$2.call(N1qlQueryExecutor.java:469)
at com.couchbase.client.java.query.core.N1qlQueryExecutor$10$2.call(N1qlQueryExecutor.java:464)
at rx.internal.operators.OperatorMap$MapSubscriber.onNext(OperatorMap.java:66)
at rx.internal.operators.OperatorTakeLastOne$ParentSubscriber.emit(OperatorTakeLastOne.java:168)
at rx.internal.operators.OperatorTakeLastOne$ParentSubscriber.onCompleted(OperatorTakeLastOne.java:141)
at rx.internal.operators.OperatorScan$InitialProducer.checkTerminated(OperatorScan.java:216)
at rx.internal.operators.OperatorScan$InitialProducer.emitLoop(OperatorScan.java:309)
at rx.internal.operators.OperatorScan$InitialProducer.emit(OperatorScan.java:295)
at rx.internal.operators.OperatorScan$InitialProducer.onCompleted(OperatorScan.java:233)
at rx.internal.operators.OperatorScan$3.onCompleted(OperatorScan.java:154)
at rx.internal.operators.OperatorMap$MapSubscriber.onCompleted(OperatorMap.java:94)
at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.checkTerminated(OperatorObserveOn.java:272)
at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.call(OperatorObserveOn.java:207)
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.couchbase.client.core.CouchbaseException: N1qlQuery Error - {"msg":"syntax error - at field17","code":3000}
	at com.couchbase.client.java.query.core.N1qlQueryExecutor$10$4.call(N1qlQueryExecutor.java:447)
	at com.couchbase.client.java.query.core.N1qlQueryExecutor$10$4.call(N1qlQueryExecutor.java:442)
	at rx.internal.operators.OperatorMap$MapSubscriber.onNext(OperatorMap.java:66)
	at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.call(OperatorObserveOn.java:215)
	... 8 more

Sometimes I get errors like this:

com.couchbase.client.core.CouchbaseException: java.lang.IllegalStateException: Error parsing query response (in TRANSITION) at "\u0026K%!$(MISSING)\\}\u002606:", enable trace to see response content
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:239)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:428)
	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:277)
	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:264)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:243)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:962)
	at com.couchbase.client.deps.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:879)
	at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:360)
	at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Error parsing query response (in TRANSITION) at "\u0026K%!$(MISSING)\\}\u002606:", enable trace to see response content
	at com.couchbase.client.core.endpoint.query.QueryHandler.transitionToNextToken(QueryHandler.java:443)
	at com.couchbase.client.core.endpoint.query.QueryHandler.parseQueryError(QueryHandler.java:601)
	at com.couchbase.client.core.endpoint.query.QueryHandler.parseQueryResponse(QueryHandler.java:380)
	at com.couchbase.client.core.endpoint.query.QueryHandler.decodeResponse(QueryHandler.java:227)
	at com.couchbase.client.core.endpoint.query.QueryHandler.decodeResponse(QueryHandler.java:70)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:229)
	... 21 more`

Or this:

com.couchbase.client.core.CouchbaseException: Error while preparing plan
at com.couchbase.client.java.query.core.N1qlQueryExecutor$10$2.call(N1qlQueryExecutor.java:469)
at com.couchbase.client.java.query.core.N1qlQueryExecutor$10$2.call(N1qlQueryExecutor.java:464)
at rx.internal.operators.OperatorMap$MapSubscriber.onNext(OperatorMap.java:66)
at rx.internal.operators.OperatorTakeLastOne$ParentSubscriber.emit(OperatorTakeLastOne.java:168)
at rx.internal.operators.OperatorTakeLastOne$ParentSubscriber.onCompleted(OperatorTakeLastOne.java:141)
at rx.internal.operators.OperatorScan$InitialProducer.checkTerminated(OperatorScan.java:216)
at rx.internal.operators.OperatorScan$InitialProducer.emitLoop(OperatorScan.java:309)
at rx.internal.operators.OperatorScan$InitialProducer.emit(OperatorScan.java:295)
at rx.internal.operators.OperatorScan$InitialProducer.onCompleted(OperatorScan.java:233)
at rx.internal.operators.OperatorScan$3.onCompleted(OperatorScan.java:154)
at rx.internal.operators.OperatorMap$MapSubscriber.onCompleted(OperatorMap.java:94)
at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.checkTerminated(OperatorObserveOn.java:272)
at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.call(OperatorObserveOn.java:207)
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.couchbase.client.core.CouchbaseException: N1qlQuery Error - {"msg":"syntax error: invalid quoted string - at \"#0d\\\"D-%!\\(MISSING)-%!@(MISSING)9*Yy*;b\\\"2r/ *%!\"(MISSING)","code":3000}
	at com.couchbase.client.java.query.core.N1qlQueryExecutor$10$4.call(N1qlQueryExecutor.java:447)
	at com.couchbase.client.java.query.core.N1qlQueryExecutor$10$4.call(N1qlQueryExecutor.java:442)
	at rx.internal.operators.OperatorMap$MapSubscriber.onNext(OperatorMap.java:66)
	at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.call(OperatorObserveOn.java:215)
	... 8 more

I have tried the original YCSB as well Couchbase’s fork. The following are my settings for running YCSB:

YCSB properties:

COUCHBASE_EPOLL=“true”
COUCHBASE_BOOST=“8”
COUCHBASE_KV=“false”

Scale properties: (450 million records)

recordcount=450000000
insertstart=0
insertcount=5625000
operationcount=25000000
threadcount=50
target=12500

Workload properties (workload a):

workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
writeallfields=true
readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0
requestdistribution=zipfian
fieldcount=50
fieldlength=25

System architecture:
27 data nodes
3 index nodes
full primary index on test bucket

Funny thing about the exceptions are that sometimes my fields don’t have what YCSB claim to encounter. For example, N1qlQuery Error - {“msg”:"syntax error: invalid quoted string - at "#0d\“D-%!\(MISSING)-%!@(MISSING)9Yy;b\“2r/ *%!”(MISSING)”,“code”:3000}, I have tried only having 5 records and I could not find this field anywhere. Also YCSB works if I change it to kv mode, so I don’t have any connection issue whatsoever.

Couchbase labs has used YCSB for their own benchmarking purposes, so I think it should work. Any help or suggestions is much appreciated.

vsr1 · January 8, 2020, 6:35pm

Check these links

WenTat · January 9, 2020, 2:01am

Hi @vsr1 thanks for your reply. I have looked into the 2 articles you linked above. The usage of YCSB is heavily modified, I just want to use it out of the box for a general test and I encountered these errors. Do you know if Couchbase lab’s YCSB has any errors using couchbase.kv=false right out of the box?

vsr1 · January 9, 2020, 2:07am

I don’t know about it.

ingenthr · January 9, 2020, 4:50am

@korriganclark or @wayne should know.

korriganclark · January 9, 2020, 5:26am

@WenTat

What branch of couchbaselabs YCSB are you using?
What version of the couchbase java sdk are you building YCSB with?
Can you also supply the full YCSB command you are using to run you tests?
Are you using the couchbase or couchbase2 in YCSSB?
What operating system are you running YCSB from?

Once I have this information I’ll take a look try to see whats going on. Off the top of my head, I have no recollection of seeing this error before.

WenTat · January 9, 2020, 6:00am

Hi, @korriganclark, I think I have found the source of my problem. The YCSB couchbase SDK in the original YCSB was 2.3. I upgraded it to 2.7, flushed all data and rerun the load phase and run phase. Now everything works well. The problem with couchbase lab’s YCSB was that I just did run on the data (450 million item loading and index creation is a long time, so I did not rerun it). So I think overall the problem was the old SDK I was using during loading.

What branch of couchbaselabs YCSB are you using?

Master branch

What version of the couchbase java sdk are you building YCSB with?

2.7 in couchbase labs, 2.3 in original YCSB

Can you also supply the full YCSB command you are using to run you tests?

/ycsb/bin/ycsb $COMMAND couchbase2 -s \

-P $WORKLOAD_FILE \
-P $SCALE_FILE \
-p couchbase.host=$COUCHBASE_HOST \
-p couchbase.bucket=$COUCHBASE_BUCKET \
-p couchbase.password=$COUCHBASE_PW \
-p couchbase.epoll=$COUCHBASE_EPOLL \
-p couchbase.boost=$COUCHBASE_BOOST \
-p couchbase.kv=$COUCHBASE_KV \
>> $OUTPUT_FILE

Are you using the couchbase or couchbase2 in YCSSB?

Couchbase2

What operating system are you running YCSB from?

CentOS 7

WenTat · January 9, 2020, 9:21am

Hi @korriganclark, sorry to come back to you, but apparently my couchbase load test is failing. After changing the original YCSB couchbase t0 2.7 and calling the run command, I see ops on my couchbase console UI which I assumed it is running well. However, in the logs it has been throwing errors. Attached is error log:

com.couchbase.client.core.CouchbaseException: Error while preparing plan
at com.couchbase.client.java.query.core.N1qlQueryExecutor$11$2.call(N1qlQueryExecutor.java:558)
at com.couchbase.client.java.query.core.N1qlQueryExecutor$11$2.call(N1qlQueryExecutor.java:553)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
at rx.internal.operators.DeferredScalarSubscriber.complete(DeferredScalarSubscriber.java:100)
at rx.internal.operators.DeferredScalarSubscriber.onCompleted(DeferredScalarSubscriber.java:73)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onCompleted(OnSubscribeMap.java:97)
at rx.internal.operators.OperatorOnBackpressureBuffer$BufferSubscriber.complete(OperatorOnBackpressureBuffer.java:163)
at rx.internal.util.BackpressureDrainManager.drain(BackpressureDrainManager.java:187)
at rx.internal.util.BackpressureDrainManager.terminateAndDrain(BackpressureDrainManager.java:115)
at rx.internal.operators.OperatorOnBackpressureBuffer$BufferSubscriber.onCompleted(OperatorOnBackpressureBuffer.java:134)
at rx.observers.SafeSubscriber.onCompleted(SafeSubscriber.java:79)
at rx.internal.operators.NotificationLite.accept(NotificationLite.java:125)
at rx.internal.operators.BufferUntilSubscriber$OnSubscribeAction.call(BufferUntilSubscriber.java:108)
at rx.internal.operators.BufferUntilSubscriber$OnSubscribeAction.call(BufferUntilSubscriber.java:80)
at rx.Observable.subscribe(Observable.java:10423)
at rx.Observable.subscribe(Observable.java:10390)
at com.couchbase.client.core.utils.UnicastAutoReleaseSubject$OnSubscribeAction.call(UnicastAutoReleaseSubject.java:228)
at com.couchbase.client.core.utils.UnicastAutoReleaseSubject$OnSubscribeAction.call(UnicastAutoReleaseSubject.java:207)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153)
at rx.internal.operators.OnSubscribeReduceSeed.call(OnSubscribeReduceSeed.java:40)
at rx.internal.operators.OnSubscribeReduceSeed.call(OnSubscribeReduceSeed.java:24)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10327)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:395)
at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:355)
at rx.internal.operators.OperatorMerge$InnerSubscriber.onNext(OperatorMerge.java:846)
at rx.observers.Subscribers$5.onNext(Subscribers.java:235)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onNext(OnSubscribeDoOnEach.java:101)
at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
at rx.Subscriber.setProducer(Subscriber.java:211)
at rx.Subscriber.setProducer(Subscriber.java:205)
at rx.Subscriber.setProducer(Subscriber.java:205)
at rx.subjects.AsyncSubject.onCompleted(AsyncSubject.java:103)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.completeResponse(AbstractGenericHandler.java:508)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.publishResponse(AbstractGenericHandler.java:476)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:358)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
at com.couchbase.client.deps.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:725)
at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:400)
at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:300)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.couchbase.client.core.CouchbaseException: N1qlQuery Error - {"msg":"syntax error: invalid quoted string - at \"*Nm7Z#\\\"\\!02(>Dm3S76<8%!R(MISSING)!8\"","code":3000}
	at com.couchbase.client.java.query.core.N1qlQueryExecutor$11$4.call(N1qlQueryExecutor.java:536)
	at com.couchbase.client.java.query.core.N1qlQueryExecutor$11$4.call(N1qlQueryExecutor.java:531)
	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
	at rx.internal.operators.NotificationLite.accept(NotificationLite.java:135)
	at rx.internal.operators.OperatorOnBackpressureBuffer$BufferSubscriber.accept(OperatorOnBackpressureBuffer.java:156)
	at rx.internal.util.BackpressureDrainManager.drain(BackpressureDrainManager.java:198)
	at rx.internal.operators.OperatorOnBackpressureBuffer$BufferSubscriber.onNext(OperatorOnBackpressureBuffer.java:151)
	at rx.observers.SafeSubscriber.onNext(SafeSubscriber.java:134)
	at rx.internal.operators.NotificationLite.accept(NotificationLite.java:135)
	... 64 more
Jan 09, 2020 6:01:18 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 onSuccess
WARNING: [server:11210][KeyValueEndpoint]: Socket connect took longer than specified timeout: Connect callback did not return, hit safeguarding timeout.
Jan 09, 2020 6:01:18 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 onSuccess
WARNING: [server:11210][KeyValueEndpoint]: Socket connect took longer than specified timeout: Connect callback did not return, hit safeguarding timeout.
WARNING: Received Error during Reconfiguration.
com.couchbase.client.deps.io.netty.channel.ConnectTimeoutException: Connect callback did not return, hit safeguarding timeout.
	at com.couchbase.client.core.endpoint.AbstractEndpoint$3.call(AbstractEndpoint.java:385)
	at com.couchbase.client.core.endpoint.AbstractEndpoint$3.call(AbstractEndpoint.java:378)
	at rx.internal.operators.SingleOperatorOnErrorResumeNext$2.onError(SingleOperatorOnErrorResumeNext.java:69)
	at rx.internal.operators.SingleTimeout$TimeoutSingleSubscriber$OtherSubscriber.onError(SingleTimeout.java:133)
	at rx.Single$1.call(Single.java:477)
	at rx.Single$1.call(Single.java:473)
	at rx.Single.subscribe(Single.java:1979)
	at rx.Single$18.call(Single.java:2518)
	at rx.Single$18.call(Single.java:2505)
	at rx.internal.operators.SingleTimeout$TimeoutSingleSubscriber.call(SingleTimeout.java:110)
	at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189)
	at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

Also, screenshot of YCSB run activity. Previously when I was using Couchbase SDK 2.3 the ops and N1ql query per sec would be 0. Now something is running.

N1QL operations per hour

Btw, I am running YCSB across 4 instances. Each with the same scale properties, but with different insertstart.