Couchbase Server

modify ec2 security group and open traffic to/from port 11209(Replication doesn't work on EC2.)

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: documentation
  • Security Level: Public
  • Environment:
    EC2, Ubuntu 12.04

    ec2-174-129-82-112.compute-1.amazonaws.com (Administrator:password, root:couchbase)

Description

Build:
-- 2.0.0-1351-rel

Security group:
-- couchbase

Steps:
-- create 4 nodes cluster (nodes use private ip addresses)
-- create default bucket with 1 replica enabled
-- insert arbitrary number of items (2M in my case)

Actual result:
-- no doc is replicated

Notice:
-- It's not about XDCR.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Chiyoung Seo added a comment -
Mike, please do the initial investigation. Thanks.
Show
Chiyoung Seo added a comment - Mike, please do the initial investigation. Thanks.
Chiyoung Seo made changes -
Field Original Value New Value
Assignee Chiyoung Seo [ chiyoung ] Mike Wiederhold [ mikew ]
Hide
Permalink
Mike Wiederhold added a comment -
Ebucketmigrator is timing out while connecting to memcached. It looks like memcached might have crashed or become unresponsive after the initial vbucket creation mechanism. This appears to be the case since no longer see stats updates from ns_server in the logs. There are no crash reports from memcached though.

=========================CRASH REPORT=========================
  crasher:
    initial call: ebucketmigrator_srv:init/1
    pid: <0.3854.0>
    registered_name: []
    exception error: no match of right hand side value {error,timeout}
      in function ebucketmigrator_srv:connect/4
      in call from ebucketmigrator_srv:init/1
    ancestors: ['ns_vbm_new_sup-default','ns_memcached_sup-default',
                  'single_bucket_sup-default',<0.1187.0>]
    messages: []
    links: [#Port<0.8470>,<0.1200.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 610
    stack_size: 24
    reductions: 7063
  neighbours:

[error_logger:error] [2012-06-22 16:46:33] [ns_1@10.121.13.111:error_logger:ale_error_logger_handler:log_report:72]
Show
Mike Wiederhold added a comment - Ebucketmigrator is timing out while connecting to memcached. It looks like memcached might have crashed or become unresponsive after the initial vbucket creation mechanism. This appears to be the case since no longer see stats updates from ns_server in the logs. There are no crash reports from memcached though. =========================CRASH REPORT=========================   crasher:     initial call: ebucketmigrator_srv:init/1     pid: <0.3854.0>     registered_name: []     exception error: no match of right hand side value {error,timeout}       in function ebucketmigrator_srv:connect/4       in call from ebucketmigrator_srv:init/1     ancestors: ['ns_vbm_new_sup-default','ns_memcached_sup-default',                   'single_bucket_sup-default',<0.1187.0>]     messages: []     links: [#Port<0.8470>,<0.1200.0>]     dictionary: []     trap_exit: false     status: running     heap_size: 610     stack_size: 24     reductions: 7063   neighbours: [error_logger:error] [2012-06-22 16:46:33] [ ns_1@10.121.13.111 :error_logger:ale_error_logger_handler:log_report:72]
Mike Wiederhold made changes -
Priority Blocker [ 1 ] Major [ 3 ]
Hide
Permalink
Mike Wiederhold added a comment -
Lowering priority since I don't have much information to address further.
Show
Mike Wiederhold added a comment - Lowering priority since I don't have much information to address further.
Hide
Permalink
Mike Wiederhold added a comment -
Assigning back to Pavel. Is this reproducible or do you have a cluster I can look at?
Show
Mike Wiederhold added a comment - Assigning back to Pavel. Is this reproducible or do you have a cluster I can look at?
Mike Wiederhold made changes -
Assignee Mike Wiederhold [ mikew ] Pavel Paulau [ pavelpaulau ]
Hide
Permalink
Karan Kumar added a comment -
@Pavel:

Please put the build number on the ticket.
Show
Karan Kumar added a comment - @Pavel: Please put the build number on the ticket.
Hide
Permalink
Pavel Paulau added a comment -
try this cluster:

ec2-174-129-98-20.compute-1.amazonaws.com
ec2-23-20-111-39.compute-1.amazonaws.com
ec2-23-22-235-210.compute-1.amazonaws.com
ec2-107-20-81-224.compute-1.amazonaws.com
Show
Pavel Paulau added a comment - try this cluster: ec2-174-129-98-20.compute-1.amazonaws.com ec2-23-20-111-39.compute-1.amazonaws.com ec2-23-22-235-210.compute-1.amazonaws.com ec2-107-20-81-224.compute-1.amazonaws.com
Pavel Paulau made changes -
Assignee Pavel Paulau [ pavelpaulau ] Mike Wiederhold [ mikew ]
Pavel Paulau made changes -
Description Steps:
-- create 4 nodes cluster (nodes use private ip addresses)
-- create default bucket with 1 replica enabled
-- insert arbitrary number of items (2M in my case)

Actual result:
-- no doc is replicated

Notice:
-- It's not about XDCR. 		Build:
2.0.0-1351-rel

Steps:
-- create 4 nodes cluster (nodes use private ip addresses)
-- create default bucket with 1 replica enabled
-- insert arbitrary number of items (2M in my case)

Actual result:
-- no doc is replicated

Notice:
-- It's not about XDCR.
Pavel Paulau made changes -
Description Build:
2.0.0-1351-rel

Steps:
-- create 4 nodes cluster (nodes use private ip addresses)
-- create default bucket with 1 replica enabled
-- insert arbitrary number of items (2M in my case)

Actual result:
-- no doc is replicated

Notice:
-- It's not about XDCR. 		Build:
-- 2.0.0-1351-rel

Security group:
-- couchbase

Steps:
-- create 4 nodes cluster (nodes use private ip addresses)
-- create default bucket with 1 replica enabled
-- insert arbitrary number of items (2M in my case)

Actual result:
-- no doc is replicated

Notice:
-- It's not about XDCR.
Mike Wiederhold made changes -
Component/s couchbase-bucket [ 10173 ]
Hide
Permalink
Mike Wiederhold added a comment -
The security group doesn't have port 11209 open and this port is used by ns_server for replication. Karan has just edited the security group to enable this port.
Show
Mike Wiederhold added a comment - The security group doesn't have port 11209 open and this port is used by ns_server for replication. Karan has just edited the security group to enable this port.
Mike Wiederhold made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Won't Fix [ 2 ]
Farshid Ghods made changes -
Summary Replication doesn't work on EC2. modify ec2 security group and open traffic to/from port 11209(Replication doesn't work on EC2.)
Labels 1.8.1-release-notes
Hide
Permalink
Pavel Paulau added a comment -
I think we also should fix this:

http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-network-ports.html
Show
Pavel Paulau added a comment - I think we also should fix this: http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-network-ports.html
Hide
Permalink
Farshid Ghods added a comment -
add port 11209 to 2.0 and 1.8 manual
http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-network-ports.html
Show
Farshid Ghods added a comment - add port 11209 to 2.0 and 1.8 manual http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-network-ports.html
Farshid Ghods made changes -
Resolution Won't Fix [ 2 ]
Status Resolved [ 5 ] Reopened [ 4 ]
Assignee Mike Wiederhold [ mikew ] MC Brown [ mccouch ]
Farshid Ghods made changes -
Component/s documentation [ 10012 ]
Mike Wiederhold made changes -
Component/s couchbase-bucket [ 10173 ]
Hide
Permalink
MC Brown added a comment -
Documentation has been updated with port 11209
Show
MC Brown added a comment - Documentation has been updated with port 11209
MC Brown made changes -
Status Reopened [ 4 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]

People

  • Assignee:
    MC Brown
    Reporter:
    Pavel Paulau
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: