Issue with 3 node cluster

Hi,

I am running a test application on community edition 3.0.1 and do not have any problem when all nodes and sync gateway is online.

I have shutdown two nodes forcefully like hardware failure but and stopped Sync Gateway on last node and try to start sync gateway and it never starts and always trying to connect through offline node.

btw sync gateway config points to local server. Here are the logs while i am starting sync gateway on running server.

3 node cluster and shutdown two nodes.

[root@couchtest bin]# sudo /opt/couchbase-sync-gateway/bin/sync_gateway -loilePath=/opt/couchbase-sync-gateway/bin/logs/sg_logs.log config.json
2015-09-25T13:29:35.520+10:00 Enabling logging: [CRUD CRUD+ HTTP HTTP+ Access Cache Changes Changes+]
2015/09/25 13:29:35 Non-healthy node; node details:
2015/09/25 13:29:35 Hostname:8091=ec2-54-64-53-131.ap-northeast-1.compute.amazonaws.com, Status=unhealthy, CouchAPIBase=http://ec2-54-64-53-131.ap-northeast-1.compute.amazonaws.com:8092/kodotest2%2Bc011ec7ec0ffe147eaed06e25b31852d, ThisNode=false
2015/09/25 13:29:35 Non-healthy node; node details:
2015/09/25 13:29:35 Hostname:8091=ec2-52-69-23-23.ap-northeast-1.compute.amazonaws.com, Status=unhealthy, CouchAPIBase=http://ec2-52-69-23-23.ap-northeast-1.compute.amazonaws.com:8092/kodotest2%2Bc011ec7ec0ffe147eaed06e25b31852d, ThisNode=false
2015/09/25 13:29:35 Trying with selected node 0
2015/09/25 13:29:35 Non-healthy node; node details:
2015/09/25 13:29:35 Hostname:8091=ec2-54-64-53-131.ap-northeast-1.compute.amazonaws.com, Status=unhealthy, CouchAPIBase=http://ec2-54-64-53-131.ap-northeast-1.compute.amazonaws.com:8092/kodotest2%2Bc011ec7ec0ffe147eaed06e25b31852d, ThisNode=false
2015/09/25 13:29:35 Non-healthy node; node details:
2015/09/25 13:29:35 Hostname:8091=ec2-52-69-23-23.ap-northeast-1.compute.amazonaws.com, Status=unhealthy, CouchAPIBase=http://ec2-52-69-23-23.ap-northeast-1.compute.amazonaws.com:8092/kodotest2%2Bc011ec7ec0ffe147eaed06e25b31852d, ThisNode=false
2015/09/25 13:29:35 Trying with selected node 0
2015/09/25 13:30:43 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.022598018s
2015/09/25 13:31:51 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.022580607s
2015/09/25 13:32:59 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.030650614s
2015/09/25 13:34:07 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.025533909s
2015/09/25 13:35:15 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.025692428s
2015/09/25 13:36:23 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.028676303s
2015/09/25 13:37:31 go-couchbase: call to Do("_sync:syncdata") in github.com/couchbase/go-couchbase.(*Bucket).casNext took 1m8.019288541s
[root@couchtest bin]# sudo /opt/couchbase-sync-gateway/bin/sync_gateway -logFilePath=/opt/couchbase-sync-gateway/bin/logs/sg_logs.log config.json
2015-09-25T13:37:58.415+10:00 Enabling logging: [CRUD CRUD+ HTTP HTTP+ Access Cache Changes Changes+]
2015/09/25 13:37:58 Non-healthy node; node details:
2015/09/25 13:37:58 Hostname:8091=ec2-54-64-53-131.ap-northeast-1.compute.amazonaws.com, Status=unhealthy, CouchAPIBase=http://ec2-54-64-53-131.ap-northeast-1.compute.amazonaws.com:8092/kodotest2%2Bc011ec7ec0ffe147eaed06e25b31852d, ThisNode=false
2015/09/25 13:37:58 Non-healthy node; node details:
2015/09/25 13:37:58 Hostname:8091=ec2-52-69-23-23.ap-northeast-1.compute.amazonaws.com, Status=unhealthy, CouchAPIBase=http://ec2-52-69-23-23.ap-northeast-1.compute.amazonaws.com:8092/kodotest2%2Bc011ec7ec0ffe147eaed06e25b31852d, ThisNode=false
2015/09/25 13:37:58 Trying with selected node 0
2015/09/25 13:37:58 Non-healthy node; node details:
2015/09/25 13:37:58 Hostname:8091=ec2-54-64-53-131.ap-northeast-1.compute.amazonaws.com, Status=unhealthy, CouchAPIBase=http://ec2-54-64-53-131.ap-northeast-1.compute.amazonaws.com:8092/kodotest2%2Bc011ec7ec0ffe147eaed06e25b31852d, ThisNode=false
2015/09/25 13:37:58 Non-healthy node; node details:
2015/09/25 13:37:58 Hostname:8091=ec2-52-69-23-23.ap-northeast-1.compute.amazonaws.com, Status=unhealthy, CouchAPIBase=http://ec2-52-69-23-23.ap-northeast-1.compute.amazonaws.com:8092/kodotest2%2Bc011ec7ec0ffe147eaed06e25b31852d, ThisNode=false
2015/09/25 13:37:58 Trying with selected node 0
2015/09/25 13:39:06 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.025675799s
2015/09/25 13:40:14 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.023763751s
2015/09/25 13:41:22 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.026473893s
2015/09/25 13:42:30 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.024542466s
2015/09/25 13:43:39 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.025715802s
2015/09/25 13:44:47 go-couchbase: call to Do("_sync:seq") in github.com/couchbase/go-couchbase.(*Bucket).Incr took 1m8.022674847s
2015/09/25 13:45:55 go-couchbase: Error connecting to tap feed of ec2-52-69-23-23.ap-northeast-1.compute.amazonaws.com:11210: dial tcp 52.69.23.23:11210: connection timed out
2015/09/25 13:45:55 go-couchbase: TAP connection lost; reconnecting to bucket “kodotest2” in 1s
2015/09/25 13:45:55 go-couchbase: call to Do("_sync:syncdata") in github.com/couchbase/go-couchbase.(*Bucket).casNext took 1m8.228776945s
[root@couchtest bin]#

Thanks,
Karunakar

Anyone has any idea on this issue?