Hello,
I’m using Couchbase Community 4.5 server and couchbase spark 2.1 connector to load data from couchbase into spark. However, I always experience timeout for simple query. For example, I have 4 nodes couchbase cluster server, I run my spark code locally to load data. I already created secondary index on the key I need. Also, when I run it locally, meaning connecting to a local couchbase, it worked out without any problem.
Here is my scala code
val spark = SparkSession
.builder()
.appName("myApps")
.master("local[*]")
.config("spark.couchbase.nodes", cbHost)
.config(s"spark.couchbase.bucket.$cbBucket", cbPassword)
.config("com.couchbase.connectTimeout", "30000")
.getOrCreate()
val query = "SELECT COUNT(1) FROM `my_bucket` WHERE `my_key` = 'good'"
//val query = "SELECT * FROM `my_bucket` WHERE `my_key` = 'good'"
spark.sparkContext.couchbaseQuery(N1qlQuery.simple(query)).collect().foreach(println)
I got following error
5318 [cb-io-1-1] INFO com.couchbase.client.core.node.Node - Connected to Node couchbase4-us01-xxxxx.com
5840 [cb-computations-1] INFO com.couchbase.client.core.config.ConfigurationProvider - Opened bucket xxx
6472 [cb-io-1-2] INFO com.couchbase.client.core.node.Node - Connected to Node couchbase4-us02-xxxxx.com
6473 [cb-io-1-3] INFO com.couchbase.client.core.node.Node - Connected to Node couchbase4-us03-xxxxx.com
6473 [cb-io-1-4] INFO com.couchbase.client.core.node.Node - Connected to Node couchbase4-us04-xxxxx.com
6477 [cb-io-1-1] INFO com.couchbase.client.core.node.Node - Connected to Node couchbase4-us05-xxxxx.com
6938 [cb-io-1-2] WARN com.couchbase.client.core.endpoint.Endpoint - [null][QueryEndpoint]: Socket connect took longer than specified timeout.
7948 [cb-io-1-3] WARN com.couchbase.client.core.endpoint.Endpoint - [null][QueryEndpoint]: Socket connect took longer than specified timeout.
7949 [cb-io-1-3] WARN com.couchbase.client.core.endpoint.Endpoint - Error during reconnect:
com.couchbase.client.deps.io.netty.channel.ConnectTimeoutException: connection timed out: couchbase4-us01-xxxxx.com/xx.xx.xx.xx:8093
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:218)
at com.couchbase.client.deps.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at com.couchbase.client.deps.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
7951 [cb-io-1-4] WARN com.couchbase.client.core.endpoint.Endpoint - [null][QueryEndpoint]: Socket connect took longer than specified timeout.
7951 [cb-io-1-3] WARN com.couchbase.client.core.endpoint.Endpoint - [null][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
com.couchbase.client.deps.io.netty.channel.ConnectTimeoutException: connection timed out: couchbase4-us01-xxxxx.com/xx.xx.xx.xx:8093
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:218)
at com.couchbase.client.deps.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at com.couchbase.client.deps.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
8959 [cb-io-1-2] WARN com.couchbase.client.core.endpoint.Endpoint - [null][QueryEndpoint]: Socket connect took longer than specified timeout.
8959 [cb-io-1-1] WARN com.couchbase.client.core.endpoint.Endpoint - [null][QueryEndpoint]: Socket connect took longer than specified timeout.
8960 [cb-io-1-1] WARN com.couchbase.client.core.endpoint.Endpoint - Error during reconnect:
com.couchbase.client.deps.io.netty.channel.ConnectTimeoutException: connection timed out: couchbase4-us02-xxxxx.com/xx.xx.xx.xx:8093
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:218)
at com.couchbase.client.deps.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at com.couchbase.client.deps.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
Looks like the connection to couchbase is alright and the timeout happened when the query is submitted.
I can telnet to all my nodes.
The question is how should I handle this? Thanks a lot.
Update:
- I also tried following settings, still didn’t work
System.setProperty("com.couchbase.socketConnect","30000")
System.setProperty("com.couchbase.connectTimeout","30000")
- I finally solved the problem. It’s my fault that I didn’t open port 8093 since I normally only open 8091-8092.
Xuan