Timeout errors

Node.js SDK
1

I’m trying to connect to a remote couchbase server, the connection callback fires ok. But if i try to get a key i get a [Error: Operation timed out] code: 23 error and if i try to query a view i get a [Error: Unknown REST Error]

The web interface works, and i can access views on :8092

I’ve also tried to with port forwarding but get the same result (ssh ec2-user@54.252.223.35 -N -L 8091:localhost:8091 -L 8092:localhost:8092 -L 11210:localhost:11210 -L 11211:localhost:11211)

telnet localhost 8091 etc works fine

2

I see you are using EC2, which might explain where the problem is.

Is the client also inside EC2?
Are you using host names or IP address for the Cluster nodes?

3

No the client is outside EC2 ,i’m using the instances local ip addresses to add servers to the cluster, could that be the problem?

4

EC2 network setup is the problem here.

When the client connects to the node via the public address to get the cluster topology. It will reply with the list of all the nodes in the cluster. If you added the node to the cluster using the local IP then the client will try and access the nodes using those IP addresses. Which will not work as they are in the private range.

The general advise is to use the FQDN when adding nodes to the cluster This means inside EC2 it will resolved to the local private IP address but outside of EC2 it will resolved to a public facing IP address.

The manual covers this in more detail:
http://docs.couchbase.com/couchbase-manual-2.2/#handling-changes-in-ip-addresses

This document explains how the clients get the cluster topology*
http://docs.couchbase.com/couchbase-devguide-2.2/#getting-cluster-topology

The other solution is to move the application inside EC2, from a security point of view this is much better than having a database face the wilds of the internet.

*With Couchbase Server 2.5.X and the newer clients (libcouchbase 2.3.1 and Java 1.4.0) this has changed.

5

For anyone coming across this (as I did) in the future:

In addition to using FDQNs for our server nodes, we also had to open port 11211 to resolve this error.

In hindsight, that is consistent with the docs: http://docs.couchbase.com/admin/admin/Install/install-networkPorts.html. But at the time it seemed counterintuitive that having the REST, WebAdmin, N1QL and Erlang (8081, 8082, 8083, and 4369) ports open wasn’t enough.

6

@vaiek The node.js SDK like all the other Couchbase SDKs does not use port 11211. You can confirm this for yourself by running netstat -an on the client machine and seeing the ports the client uses.

The problem here is how AWS networking works by using a split horizon DNS (split brain DNS as AWS calls it) and as a result the cluster nodes need to use a FQDN.

7

Ok. Even after we switched to FQDN we seemed to have issues without 11211 open, but that may have just been an illusion. Will post back here if those issues arise again. But FQDN is necessary to resolve the issues described above for sure. Thanks!