[windows] nodes became disconnected during load, source cluster is unstable

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 2.0
Fix Version/s: 2.0.2
Component/s: ns_server
Security Level: Public
Labels:
- windows
Environment:

Hide
windows server r2 2008
xdcr unidir
3 nodes clusters:
2 nodes on each has 2.0.0-1976-rel and 1 node on each 2.0.1-155-rel

<manifest><remote name="couchbase" fetch="git://10.1.1.210/"/><remote name="membase" fetch="git://10.1.1.210/"/><remote name="apache" fetch="git://github.com/apache/"/><remote name="erlang" fetch="git://github.com/erlang/"/><default remote="couchbase" revision="master"/><project name="tlm" path="tlm" revision="12abea946eafd7411273d18a10ae1f84390db3d4"><copyfile src="Makefile.top" dest="Makefile"/></project><project name="bucket_engine" path="bucket_engine" revision="1495eb770b9735dd791c483eee7e69641f82f09c"/><project name="ep-engine" path="ep-engine" revision="eee2e9564ef844cf8cc435911d9e80af0dece244"/><project name="libconflate" path="libconflate" revision="2cc8eff8e77d497d9f03a30fafaecb85280535d6"/><project name="libmemcached" path="libmemcached" revision="ca739a890349ac36dc79447e37da7caa9ae819f5" remote="membase"/><project name="libvbucket" path="libvbucket" revision="026c79ae424a6daed4bb9345e86cc8fc21759b28"/><project name="membase-cli" path="membase-cli" revision="8a716cc15a04a1ed2a67eb10ff8fa061f14d6a58" remote="membase"/><project name="memcached" path="memcached" revision="e6f892cf7bd61e91a79be0d8d2c6f075d3c4ae1b" remote="membase"/><project name="moxi" path="moxi" revision="52a5fa887bfff0bf719c4ee5f29634dd8707500e"/><project name="ns_server" path="ns_server" revision="d8aa12706c2edac309743a502606594a00dc9929"/><project name="portsigar" path="portsigar" revision="1bc865e1622fb93a3fe0d1a4cdf18eb97ed9d600"/><project name="sigar" path="sigar" revision="63a3cd1b316d2d4aa6dd31ce8fc66101b983e0b0"/><project name="couchbase-examples" path="couchbase-examples" revision="cd9c8600589a1996c1ba6dbea9ac171b937d3379"/><project name="couchbase-python-client" path="couchbase-python-client" revision="006c1aa8b76f6bce11109af8a309133b57079c4c"/><project name="couchdb" path="couchdb" revision="c6e715704e9fd7f2ae2e7899dc027727b50c5b8a"/><project name="couchdbx-app" path="couchdbx-app" revision="52d20848312254148c889de517b39199765d8e22"/><project name="couchstore" path="couchstore" revision="146c965a5aa5da21901209122bde60a8f3a52171"/><project name="geocouch" path="geocouch" revision="5f5706d8a0db214aaa942ca496661938eaf2385e"/><project name="testrunner" path="testrunner" revision="f3ed4fdc95236086f54b4de4bee90cab8b5e3b06"/><project name="otp" path="otp" revision="b6dc1a844eab061d0a7153d46e7e68296f15a504" remote="erlang"/><project name="icu4c" path="icu4c" revision="26359393672c378f41f2103a8699c4357c894be7" remote="couchbase"/><project name="snappy" path="snappy" revision="5681dde156e9d07adbeeab79666c9a9d7a10ec95" remote="couchbase"/><project name="v8" path="v8" revision="447decb75060a106131ab4de934bcc374648e7f2" remote="couchbase"/><project name="gperftools" path="gperftools" revision="8f60ba949fb8576c530ef4be148bff97106ddc59" remote="couchbase"/><project name="pysqlite" path="pysqlite" revision="0ff6e32ea05037fddef1eb41a648f2a2141009ea" remote="couchbase"/></manifest>

Show
windows server r2 2008 xdcr unidir 3 nodes clusters: 2 nodes on each has 2.0.0-1976-rel and 1 node on each 2.0.1-155-rel <manifest><remote name="couchbase" fetch=" git://10.1.1.210/ "/><remote name="membase" fetch=" git://10.1.1.210/ "/><remote name="apache" fetch=" git://github.com/apache/ "/><remote name="erlang" fetch=" git://github.com/erlang/ "/><default remote="couchbase" revision="master"/><project name="tlm" path="tlm" revision="12abea946eafd7411273d18a10ae1f84390db3d4"><copyfile src="Makefile.top" dest="Makefile"/></project><project name="bucket_engine" path="bucket_engine" revision="1495eb770b9735dd791c483eee7e69641f82f09c"/><project name="ep-engine" path="ep-engine" revision="eee2e9564ef844cf8cc435911d9e80af0dece244"/><project name="libconflate" path="libconflate" revision="2cc8eff8e77d497d9f03a30fafaecb85280535d6"/><project name="libmemcached" path="libmemcached" revision="ca739a890349ac36dc79447e37da7caa9ae819f5" remote="membase"/><project name="libvbucket" path="libvbucket" revision="026c79ae424a6daed4bb9345e86cc8fc21759b28"/><project name="membase-cli" path="membase-cli" revision="8a716cc15a04a1ed2a67eb10ff8fa061f14d6a58" remote="membase"/><project name="memcached" path="memcached" revision="e6f892cf7bd61e91a79be0d8d2c6f075d3c4ae1b" remote="membase"/><project name="moxi" path="moxi" revision="52a5fa887bfff0bf719c4ee5f29634dd8707500e"/><project name="ns_server" path="ns_server" revision="d8aa12706c2edac309743a502606594a00dc9929"/><project name="portsigar" path="portsigar" revision="1bc865e1622fb93a3fe0d1a4cdf18eb97ed9d600"/><project name="sigar" path="sigar" revision="63a3cd1b316d2d4aa6dd31ce8fc66101b983e0b0"/><project name="couchbase-examples" path="couchbase-examples" revision="cd9c8600589a1996c1ba6dbea9ac171b937d3379"/><project name="couchbase-python-client" path="couchbase-python-client" revision="006c1aa8b76f6bce11109af8a309133b57079c4c"/><project name="couchdb" path="couchdb" revision="c6e715704e9fd7f2ae2e7899dc027727b50c5b8a"/><project name="couchdbx-app" path="couchdbx-app" revision="52d20848312254148c889de517b39199765d8e22"/><project name="couchstore" path="couchstore" revision="146c965a5aa5da21901209122bde60a8f3a52171"/><project name="geocouch" path="geocouch" revision="5f5706d8a0db214aaa942ca496661938eaf2385e"/><project name="testrunner" path="testrunner" revision="f3ed4fdc95236086f54b4de4bee90cab8b5e3b06"/><project name="otp" path="otp" revision="b6dc1a844eab061d0a7153d46e7e68296f15a504" remote="erlang"/><project name="icu4c" path="icu4c" revision="26359393672c378f41f2103a8699c4357c894be7" remote="couchbase"/><project name="snappy" path="snappy" revision="5681dde156e9d07adbeeab79666c9a9d7a10ec95" remote="couchbase"/><project name="v8" path="v8" revision="447decb75060a106131ab4de934bcc374648e7f2" remote="couchbase"/><project name="gperftools" path="gperftools" revision="8f60ba949fb8576c530ef4be148bff97106ddc59" remote="couchbase"/><project name="pysqlite" path="pysqlite" revision="0ff6e32ea05037fddef1eb41a648f2a2141009ea" remote="couchbase"/></manifest>

Description

2 clusters (unidirectional replication)
2 buckets: default and sasl
loading data to source default bucket

some nodes in source cluster lose connection to memcached. (Different nodes, one become down, then when is up other become down)
error like :
Control connection to memcached on 'ns_1@10.3.121.83' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
cmd_binary_vocal_recv,
5},
{mc_client_binary,
select_bucket,
2},
{ns_memcached,
ensure_bucket,
2},
{ns_memcached,
handle_info,
2},
{gen_server,
handle_msg,
5},
{proc_lib,
init_p_do_apply,
3}]}
appear
attaching collect_info

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Iryna Mironava added a comment - 13/Feb/13 6:16 AM - edited

source cluster:
https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.3.187-source.zip
https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.121.74-source.zip
https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.121.83-source.zip

destination cluster:
https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.121.80-dest.zip
https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.2.78-dest.zip
https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.2.79-dest.zip

Show

Iryna Mironava added a comment - 13/Feb/13 6:16 AM - edited source cluster: https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.3.187-source.zip https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.121.74-source.zip https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.121.83-source.zip destination cluster: https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.121.80-dest.zip https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.2.78-dest.zip https://s3.amazonaws.com/bugdb/jira/MB-7733/a0dedc45/10.3.2.79-dest.zip

Hide

Permalink

Farshid Ghods added a comment - 13/Feb/13 7:41 PM

the node which shows up as down is indeed a 2.0.1 node

Siri could you please take a look at the 10.3.121.83-source.zip ?

Show

Farshid Ghods added a comment - 13/Feb/13 7:41 PM the node which shows up as down is indeed a 2.0.1 node Siri could you please take a look at the 10.3.121.83-source.zip ?

Hide

Permalink

Sriram Melkote added a comment - 22/Feb/13 3:45 AM

In server log:
  2013-02-13T0:51:55.675 - Control connection to memcached on 'ns_1@10.3.121.83' disconnected

In windows event log on 10.3.121.83:
  Date: 2/13/2013 12:48:37 AM
  Event ID: 10001
  Computer: WIN-5IC1JB7LHUB
  Description: Network Disconnected, Name: Network 2, State: Disconnected

So it appears that there is indeed a network disconnection. We need to investigate why. But as we've narrowed down the source of the problem to outside of the product, I would recommend moving this out of 2.0.1

Show

Sriram Melkote added a comment - 22/Feb/13 3:45 AM In server log: 2013-02-13T0:51:55.675 - Control connection to memcached on ' ns_1@10.3.121.83 ' disconnected In windows event log on 10.3.121.83: Date: 2/13/2013 12:48:37 AM Event ID: 10001 Computer: WIN-5IC1JB7LHUB Description: Network Disconnected, Name: Network 2, State: Disconnected So it appears that there is indeed a network disconnection. We need to investigate why. But as we've narrowed down the source of the problem to outside of the product, I would recommend moving this out of 2.0.1

Hide

Permalink

Farshid Ghods added a comment - 22/Feb/13 3:59 PM

please ask helpdesk to investigate this issue

Show

Farshid Ghods added a comment - 22/Feb/13 3:59 PM please ask helpdesk to investigate this issue

People

Assignee:

Iryna Mironava

Reporter:

Iryna Mironava

Vote (0)

Watch (3)

Dates

Created:

13/Feb/13 5:10 AM

Updated:

02/Apr/13 9:22 AM

Resolved:

22/Feb/13 4:00 PM

Agile

View on Board