Getting ConnectTimeoutException while reading documents from couchbase bucket in EC2 instance

Couchbase Server
, ,
1

Hi,

I had install couchbase server instance on AWS EC2 and i am trying to read documents from couchbase buckets using Spark, while running spark program its throwing ConnectTimeoutException error.

Check below code and error log file.

Build.sbt

  "org.apache.spark" %% "spark-core" % "2.2.0",
  "org.apache.spark" %% "spark-streaming" % "2.2.0",
  "org.apache.spark" %% "spark-sql" % "2.2.0",
  "com.couchbase.client" %% "spark-connector" % "2.2.0"

SparkRead.scala

import com.couchbase.client.java.document.JsonDocument
import org.apache.spark.sql.SparkSession
import com.couchbase.spark._

object SparkRead {


    def main(args: Array[String]): Unit = {

   
      val spark = SparkSession
        .builder()
        .appName("KeyValueExample")
        .master("local[*]") // use the JVM as the master, great for testing
        .config("spark.couchbase.nodes", "xx.xx.xx.xx:xxxx") // connect to couchbase on hostname
        .config("spark.couchbase.bucket.travel-sample","***")
        .config("com.couchbase.username", "couchdb")
    .config("com.couchbase.password", "couchdb")
        .config("com.couchbase.connectTimeout","300000")
        .config("com.couchbase.socketConnect","100000")
        .getOrCreate()

       spark.sparkContext
        .couchbaseGet[JsonDocument](Seq("airline_10123"))
        .collect() // collect all data from the spark workers
        .foreach(println) // print each document content


  }
}

Error Log:-:

17/12/12 14:46:58 INFO Node: Connected to Node xx.xx.xx.xx:xx
17/12/12 14:47:03 WARN Node: DNS Reverse Lookup of xx.xx.xx.xx:xx is slow, took 4515ms
17/12/12 14:47:07 WARN Node: DNS Reverse Lookup of xx.xx.xx.xx:xx is slow, took 4513ms
17/12/12 14:47:07 INFO ConfigurationProvider: Opened bucket demo
17/12/12 14:47:07 INFO Node: Disconnected from Node xx.xx.xx.xx:xx
17/12/12 14:47:07 INFO Endpoint: [null][KeyValueEndpoint]: Got notified from Channel as inactive, attempting reconnect.

17/12/12 14:47:09 WARN Endpoint: [null][KeyValueEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS: 
com.couchbase.client.deps.io.netty.channel.ConnectTimeoutException: connection timed out: /xx.xx.xx.xx:xx
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:218)
	at com.couchbase.client.deps.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
	at com.couchbase.client.deps.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:745)
17/12/12 14:47:09 WARN Endpoint: [null][KeyValueEndpoint]: Socket connect took longer than specified timeout.
17/12/12 14:47:09 WARN Endpoint: Error during reconnect: 
com.couchbase.client.deps.io.netty.channel.ConnectTimeoutException: connection timed out: /xx.xx.xx.xx:xx
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:218)
	at com.couchbase.client.deps.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
	at com.couchbase.client.deps.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:745)
17/12/12 14:47:09 WARN Endpoint: [null][KeyValueEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS: 
com.couchbase.client.deps.io.netty.channel.ConnectTimeoutException: connection timed out: /xx.xx.xx.xx:xx
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:218)
	at com.couchbase.client.deps.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
	at com.couchbase.client.deps.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:745)
17/12/12 14:47:10 WARN Endpoint: [null][KeyValueEndpoint]: Socket connect took longer than specified timeout.
17/12/12 14:47:10 WARN Endpoint: [null][KeyValueEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS: 
com.couchbase.client.deps.io.netty.channel.ConnectTimeoutException: connection timed out: /xx.xx.xx.xx:xx
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:218)
	at com.couchbase.client.deps.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
	at com.couchbase.client.deps.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:745)
17/12/12 14:47:10 WARN Endpoint: [null][KeyValueEndpoint]: Socket connect took longer than specified timeout.
17/12/12 14:47:10 WARN Endpoint: [null][KeyValueEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS: 
com.couchbase.client.deps.io.netty.channel.ConnectTimeoutException: connection timed out: /xx.xx.xx.xx:xx
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:218)
	at com.couchbase.client.deps.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
	at com.couchbase.client.deps.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:745)
17/12/12 14:47:11 WARN Endpoint: [null][KeyValueEndpoint]: Socket connect took longer than specified timeout.
17/12/12 14:47:11 WARN Endpoint: [null][KeyValueEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS: 
com.couchbase.client.deps.io.netty.channel.ConnectTimeoutException: connection timed out: /xx.xx.xx.xx:xx
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:218)
	at com.couchbase.client.deps.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
	at com.couchbase.client.deps.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:745)
2

Could you make sure you have configured your cluster properly? See details in this guide: https://developer.couchbase.com/documentation/server/5.0/cloud/couchbase-aws-best-practices.html

For instance, it might be that cluster is using private network, while the client deployed in different network, and cannot reach those addresses.