Couchbase connectivity problem in AWS VPC
Hi All,
I am experiencing a connectivity problem which is a bizarre in a private virtual vpc in Amazon environment (AWS). My context is like this i have subnet in in this subnet i have 2 machines a client machine where i used Couchbase java SDK and a machine for Couchbase server. Both are large instances.
Current java version 1.6 .
I have open all the required ports (8091,8092,11210) for Couchbase on the Security Groups. I am testing the connection with a curl command to test the connectivity with a bucket and no problems here.
The strange behavior is when i am trying to access a view with the java client because i receive a timeout error although the java client is authenticated successfully.
This is what my log look like:
(CacheManager.java:102) -B06C9F5CFF85- Cache client checked out [stdout]
INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/10.0.X.XXX:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@57a220c2
INFO net.spy.memcached.auth.AuthThread: Authenticated to 10.0.X.XXX/10.0.X.XXX:11210
Added 10.0.X.XXX to connect queue
INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: /10.0.X.XXX:8092 - Session request successful
ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out: [10.0.X.XXX/10.0.X.XXX:8092]
and then after a while
INFO com.couchbase.client.ViewConnection: Node 10.0.X.XXX has no ops in the queue
INFO com.couchbase.client.ViewNode: I/O reactor terminated for 10.0.X.XXX
SHUTTING DOWN (informed client)) INFO com.couchbase.client.CouchbaseConnection: Shut down Couchbase client
So suddenly the connection times out without any reason and then the client it is shutting down itself.
Other context data:
Couchbase server enterprise 2.0
Java 1.6
Ubuntu 11.10 server x64
On my local machine i have a Couchbase running in a local vm in local lan and i have no problems in using it.
Can everybody help in debugging this. We want to take Couchbase to production but without this we can not advance.
Thank you in advance and any help or information is much appreciated.
Hi Tim,
I have a very detailed talk with Michael and if you would like to join please read the comments in this link.
http://www.couchbase.com/issues/browse/JCBC-151?page=com.atlassian.jira....
What i have until now with Michael help is that the http connection when a view is retrieved is timing out very fast. With curl the query is retrieved fast, in less then half a second.
Certainly Michael is the right person to talk to and is doing a fine job of tracking it down with you on that bug report.
Regards,
Tim
Dragos,
Hi, I have a couple of questions or tests to try to narrow down where the problem is.
How many Couchbase server nodes are in your cluster?
When your Java client is running, can you run "netstat -natp" on the client host, and verify that you're seeing established connections on :8092 to the server nodes?
Are you able to do simple get and set operations with the Java client OK? It is just the query of the view that doesn't work?
Does the view query work when you view it in the Couchbase UI in the Views panel? Clicking on the "Show Results" button displays the results?
If so, does it also display the results when you click the URL next to the view? This will require port 8092 be open to your browser, which may not be the case. You can copy-and-paste that URL and use curl from the command line on a client machine to test querying the actual view.
Can you verify with telnet that you can open a TCP connection from the client machine to both 11210 and 8092 on each server machine?
Thanks,
Tim