Connection between ep-engine and erlang mccouch is constantly being timeout and reset on windows build 1908

Chiyoung Seo added a comment - 29/Oct/12 12:36 PM Can you grabe cbcollect_info from all the nodes? Thanks!

Chiyoung Seo added a comment - 29/Oct/12 12:48 PM The replication building for vbucket 256 on 10.2.2.230 was being stuck: Chiyoung-MacBook:ep-engine chiyoung$ ./management/cbstats 10.2.2.230:11210 checkpoint | grep vb_256  vb_256:checkpoint_extension: false  vb_256:last_closed_checkpoint_id: 9  vb_256:num_checkpoint_items: 3096  vb_256:num_checkpoints: 2  vb_256:num_items_for_persistence: 3096  vb_256:num_open_checkpoint_items: 0  vb_256:num_tap_cursors: 0  vb_256:open_checkpoint_id: 10  vb_256:persisted_checkpoint_id: 2  vb_256:state: replica We can see that the persisted checkpoint id for vbucket 256 is still 2 even if the open checkpoint id is 10. The flusher on 10.2.2.230 was being stuck and has been running for 822 minutes in a single cycle: dispatcher      runtime: 822m      state: dispatcher_running      status: running      task: Running a flusher loop.

Chiyoung Seo added a comment - 29/Oct/12 12:57 PM Ronnie, Can you attach the memcached logs from 10.2.2.230?

Chiyoung Seo added a comment - 29/Oct/12 1:41 PM The socket connection between ep-engine and erlang mccouch in 10.2.2.230 is constantly being reset now: Sun Oct 28 22:17:20.900843 Pacific Daylight Time 3: TAP (Producer) eq_tapq: replication_ns_1@10.2.2.233 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0 Sun Oct 28 22:17:20.900843 Pacific Daylight Time 3: TAP (Producer) eq_tapq:replication_building_170_' ns_1@10.2.2.233 ' - Clear the tap queues by force Sun Oct 28 22:18:20.588343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:18:21.572718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:21:21.588343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:21:22.619593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:22:20.994593 Pacific Daylight Time 3: Schedule cleanup of "eq_tapq: replication_ns_1@10.2.2.232 " Sun Oct 28 22:22:20.994593 Pacific Daylight Time 3: TAP (Producer) eq_tapq: replication_ns_1@10.2.2.232 - Clear the tap queues by force Sun Oct 28 22:24:22.619593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:24:23.666468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:27:23.682093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:27:24.760218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:30:24.760218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:30:25.760218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:33:25.760218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:33:26.760218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:36:26.760218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:36:27.838343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:39:27.838343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:39:28.885218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:42:28.885218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:42:29.916468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:45:29.916468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:45:30.932093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:48:30.932093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:48:31.947718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:51:31.947718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:51:32.978968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:54:32.978968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:54:33.978968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 22:57:33.978968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 22:57:35.025843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:00:35.025843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:00:36.103968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:03:36.103968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:03:37.166468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:06:37.166468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:06:38.197718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:09:38.197718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:09:39.228968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:12:39.228968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:12:40.244593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:15:40.244593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:15:41.275843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:18:41.275843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:18:42.307093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:21:42.307093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:21:43.322718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:24:43.322718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:24:44.353968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:27:44.353968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:27:45.353968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:30:45.353968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:30:46.385218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:33:46.385218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:33:47.416468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:36:47.416468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:36:48.432093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:39:48.432093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:39:49.447718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:42:49.447718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:42:50.463343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:45:50.463343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:45:51.463343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:48:51.463343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:48:52.463343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:51:52.463343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:51:53.463343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:54:53.463343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:54:54.463343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Sun Oct 28 23:57:54.463343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Sun Oct 28 23:57:55.478968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:00:55.478968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:00:56.525843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:03:56.525843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:03:57.557093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:06:57.557093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:06:58.588343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:09:58.588343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:09:59.603968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:12:59.603968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:13:00.635218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:16:00.635218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:16:01.635218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:19:01.635218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:19:02.650843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:22:02.650843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:22:03.666468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:25:03.666468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:25:04.697718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:28:04.697718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:28:05.713343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:31:05.713343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:31:06.728968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:34:06.728968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:34:07.744593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:37:07.744593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:37:08.744593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:40:08.744593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:40:09.744593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:43:09.744593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:43:10.775843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:46:10.775843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:46:11.807093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:49:11.807093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:49:12.822718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:52:12.822718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:52:13.822718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:55:13.822718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:55:14.822718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 00:58:14.822718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 00:58:15.838343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:01:15.838343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:01:16.853968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:04:16.853968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:04:17.869593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:07:17.869593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:07:18.885218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:10:18.885218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:10:19.885218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:13:19.885218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:13:20.885218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:16:20.885218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:16:21.885218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:19:21.885218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:19:22.916468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:22:22.916468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:22:23.947718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:25:23.947718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:25:24.963343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:28:24.963343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:28:25.994593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:31:25.994593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:31:27.025843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:34:27.025843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:34:28.041468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:37:28.041468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:37:29.057093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:40:29.057093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:40:30.088343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:43:30.088343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:43:31.119593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:46:31.119593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:46:32.135218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:49:32.135218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:49:33.135218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:52:33.135218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:52:34.166468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:55:34.166468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:55:35.182093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 01:58:35.182093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 01:58:36.197718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:01:36.197718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:01:37.228968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:04:37.228968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:04:38.244593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:07:38.244593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:07:39.275843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:10:39.275843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:10:40.307093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:13:40.307093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:13:41.322718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:16:41.322718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:16:42.353968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:19:42.353968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:19:43.385218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:22:43.385218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:22:44.385218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:25:44.385218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:25:45.385218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:28:45.385218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:28:46.385218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:31:46.385218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:31:47.400843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:34:47.400843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:34:48.416468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:37:48.416468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:37:49.432093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:40:49.432093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:40:50.463343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:43:50.463343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:43:51.494593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:46:51.494593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:46:52.494593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:49:52.494593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:49:53.525843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:52:53.525843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:52:54.525843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:55:54.525843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:55:55.525843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 02:58:55.525843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 02:58:56.525843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:01:56.525843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:01:57.525843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:04:57.525843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:04:58.557093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:07:58.557093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:07:59.572718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:10:59.572718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:11:00.572718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:14:00.572718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:14:01.603968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:17:01.603968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:17:02.603968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:20:02.603968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:20:03.603968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:23:03.603968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:23:04.603968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:26:04.603968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:26:05.635218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:29:05.635218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:29:06.666468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:32:06.666468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:32:07.666468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:35:07.666468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:35:08.666468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:38:08.666468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:38:09.697718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:41:09.697718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:41:10.713343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:44:10.713343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:44:11.713343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:47:11.713343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:47:12.713343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:50:12.713343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:50:13.713343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:53:13.713343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:53:14.744593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:56:14.744593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:56:15.775843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 03:59:15.775843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 03:59:16.775843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:02:16.775843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:02:17.775843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:05:17.775843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:05:18.775843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:08:18.775843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:08:19.807093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:11:19.807093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:11:20.838343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:14:20.838343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:14:21.853968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:17:21.853968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:17:22.853968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:20:22.853968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:20:23.885218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:23:23.885218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:23:24.885218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:26:24.885218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:26:25.916468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:29:25.916468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:29:26.916468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:32:26.916468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:32:27.932093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:35:27.932093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:35:28.932093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:38:28.932093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:38:29.947718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:41:29.947718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:41:30.978968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:44:30.978968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:44:31.994593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:47:31.994593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:47:32.994593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:50:32.994593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:50:33.994593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:53:33.994593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:53:35.025843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:56:35.025843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:56:36.025843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 04:59:36.025843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 04:59:37.025843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:02:37.025843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:02:38.041468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:05:38.041468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:05:39.072718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:08:39.072718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:08:40.088343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:11:40.088343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:11:41.103968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:14:41.103968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:14:42.103968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:17:42.103968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:17:43.135218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:20:43.135218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:20:44.135218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:23:44.135218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:23:45.135218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:26:45.135218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:26:46.135218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:29:46.135218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:29:47.135218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:32:47.135218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:32:48.135218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:35:48.135218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:35:49.135218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:38:49.135218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:38:50.135218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:41:50.135218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:41:51.166468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:44:51.166468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:44:52.166468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:47:52.166468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:47:53.166468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:50:53.166468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:50:54.166468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:53:54.166468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:53:55.182093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:56:55.182093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:56:56.197718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 05:59:56.197718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 05:59:57.213343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:02:57.213343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:02:58.213343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:05:58.213343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:05:59.244593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:08:59.244593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:09:00.260218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:12:00.260218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:12:01.291468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:15:01.291468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:15:02.307093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:18:02.307093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:18:03.307093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:21:03.307093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:21:04.307093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:24:04.307093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:24:05.338343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:27:05.338343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:27:06.353968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:30:06.353968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:30:07.385218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:33:07.385218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:33:08.416468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:36:08.416468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:36:09.416468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:39:09.416468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:39:10.432093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:42:10.432093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:42:11.447718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:45:11.447718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:45:12.478968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:48:12.478968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:48:13.510218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:51:13.510218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:51:14.525843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:54:14.525843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:54:15.525843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 06:57:15.525843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 06:57:16.557093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:00:16.557093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:00:17.557093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:03:17.557093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:03:18.588343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:06:18.588343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:06:19.619593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:09:19.619593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:09:20.635218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:12:20.635218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:12:21.635218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:15:21.635218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:15:22.666468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:18:22.666468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:18:23.666468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:21:23.666468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:21:24.697718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:24:24.697718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:24:25.697718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:27:25.697718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:27:26.728968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:30:26.728968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:30:27.744593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:33:27.744593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:33:28.744593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:36:28.744593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:36:29.775843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:39:29.775843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:39:30.807093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:42:30.807093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:42:31.838343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:45:31.838343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:45:32.869593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:48:32.869593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:48:33.900843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:51:33.900843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:51:34.932093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:54:34.932093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:54:35.947718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 07:57:35.947718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 07:57:36.963343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:00:36.963343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:00:37.978968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:03:37.994593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:03:39.010218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:06:39.010218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:06:40.041468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:09:40.041468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:09:41.057093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:12:41.057093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:12:42.072718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:15:42.072718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:15:43.088343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:18:43.088343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:18:44.119593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:21:44.119593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:21:45.150843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:24:45.150843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:24:46.166468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:27:46.166468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:27:47.197718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:30:47.197718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:30:48.213343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:33:48.213343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:33:49.228968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:36:49.228968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:36:50.244593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:39:50.244593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:39:51.244593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:42:51.244593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:42:52.244593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:45:52.244593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:45:53.275843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:48:53.275843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:48:54.307093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:51:54.307093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:51:55.338343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:54:55.338343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:54:56.353968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 08:57:56.353968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 08:57:57.369593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:00:57.369593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:00:58.385218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:03:58.385218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:03:59.385218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:06:59.385218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:07:00.385218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:10:00.385218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:10:01.416468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:13:01.416468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:13:02.447718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:16:02.447718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:16:03.447718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:19:03.447718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:19:04.447718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:22:04.447718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:22:05.447718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:25:05.447718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:25:06.447718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:28:06.447718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:28:07.463343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:31:07.463343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:31:08.463343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:34:08.463343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:34:09.478968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:37:09.478968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:37:10.494593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:40:10.494593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:40:11.510218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:43:11.510218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:43:12.525843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:46:12.525843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:46:13.557093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:49:13.557093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:49:14.557093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:52:14.557093 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:52:15.557093 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:55:15.572718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:55:16.572718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 09:58:16.572718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 09:58:17.588343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:01:17.588343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:01:18.619593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:04:18.619593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:04:19.650843 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:07:19.650843 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:07:20.666468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:10:20.666468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:10:21.666468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:13:21.666468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:13:22.666468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:16:22.666468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:16:23.697718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:19:23.697718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:19:24.713343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:22:24.713343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:22:25.728968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:25:25.728968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:25:26.728968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:28:26.728968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:28:27.728968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:31:27.728968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:31:28.728968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:34:28.728968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:34:29.728968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:37:29.728968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:37:30.760218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:40:30.760218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:40:31.791468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:43:31.791468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:43:32.822718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:46:32.838343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:46:33.853968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:49:33.853968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:49:34.853968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:52:34.853968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:52:35.869593 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:55:35.869593 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:55:36.885218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 10:58:36.885218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 10:58:37.885218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 11:01:37.885218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 11:01:38.885218 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 11:04:38.885218 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 11:04:39.916468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 11:07:39.916468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 11:07:40.916468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 11:10:40.916468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 11:10:41.916468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 11:13:41.916468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 11:13:42.916468 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 11:16:42.916468 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 11:16:43.947718 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 11:19:43.947718 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 11:19:44.963343 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 11:22:44.963343 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 11:22:45.978968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" Mon Oct 29 11:25:45.978968 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" Mon Oct 29 11:25:46.978968 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" This is the reason why the flusher on 10.2.2.230 is being stuck and consequently rebalance got stuck.

Chiyoung Seo added a comment - 29/Oct/12 4:12 PM Mike, I wasn't able to find the culprit to the connection reset issue between ep-engine and mccouch. Can you add more warning logs to couch-notifier and make couch-notifier stats available as part of CouchKVStore stats? Now, there is no way of getting couch-notifier stats.

Chiyoung Seo added a comment - 30/Oct/12 12:43 PM Degraded it to critical as we see this issue only one time in windows cluster. In other windows tests, we haven't seen this yet.

Chiyoung Seo added a comment - 30/Oct/12 2:17 PM Mike, please update the bug with the info from Tony's cluster and assign it to Jin.

Thuan Nguyen added a comment - 30/Oct/12 3:32 PM Hit this bug in orange cluster in system test. Rebalance failed due to timeout in node 43 (perhaps at the end of rebalance first bucket). Node 43 becomes unstable. Kill all loads point to orange cluster. Node 43 is still not up. Look at memcached log on node 43, see at lot of connection reset memcached.log.2.txt:Tue Oct 30 13:17:20.526066 PDT 3: Trying to connect to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:17:20.526699 PDT 3: Connected to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:17:20.750043 PDT 3: Trying to connect to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:17:20.750558 PDT 3: Connected to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:20:20.706066 PDT 3: Trying to connect to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:20:20.706704 PDT 3: Connected to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:20:20.930052 PDT 3: Trying to connect to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:20:20.930623 PDT 3: Connected to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:23:20.886060 PDT 3: Trying to connect to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:23:20.886608 PDT 3: Connected to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:23:21.135127 PDT 3: Trying to connect to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:23:21.135694 PDT 3: Connected to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:26:21.066074 PDT 3: Trying to connect to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:26:21.066709 PDT 3: Connected to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:26:21.356276 PDT 3: Trying to connect to mccouch: "localhost:11213" memcached.log.2.txt:Tue Oct 30 13:26:21.356836 PDT 3: Connected to mccouch: "localhost:11213" Cluster information: - 8 centos 6.2 64bit server with 4 cores CPU - Each server has 32 GB RAM and 400 GB SSD disk. - 24.8 GB RAM for couchbase server at each node - SSD disk format ext4 on /data - Each server has its own SSD drive, no disk sharing with other server. - Create cluster with 6 nodes installed couchbase server 2.0.0-1908 - Cluster has 2 buckets, default (12GB with 2 replica) and saslbucket (12GB with 1 replica). - Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11) 10.6.2.37 10.6.2.38 10.6.2.44 10.6.2.45 10.6.2.42 10.6.2.43 * Load 16 million items to default bucket and 20 million items to saslbuckett. Each key has size from 512 bytes to 1024 bytes * After done loading, wait until initial index. Disable view compaction. * After initial indexing done, mutate all items with size from 1024 to 1512 bytes. * Queries all 4 views from 2 docs * Do swap rebalance, remove node 39, 40 and add node 44, 45. Link to collect info of all nodes https://s3.amazonaws.com/bugdb/jira/MB-7046/orange-ci-1908-node43-unstable-trying-to-connect-to-mccouch-20121030.tgz

Chiyoung Seo added a comment - 31/Oct/12 7:00 PM We see the connection issue between ep-engine and mccouch more often in windows cluster. Chisheng just saw this issue while testing in the windows large cluster. I think this is a blocker for windows release. I asked Chisheng to test it with the latest build that would have more logs in ep-engine's couch-notifier.

Thuan Nguyen added a comment - 01/Nov/12 2:49 AM Integrated in github-ep-engine-2-0 #456 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/456/ ])      MB-7046 instrument more diagnostic logging in CouchNotifier (Revision 4fb28863f96247585f857589cc5b5f6f0722a8e2)      Result = SUCCESS Jin : Files : * src/couch-kvstore/couch-notifier.cc * tests/ep_testsuite.cc * src/couch-kvstore/couch-kvstore.cc * src/couch-kvstore/couch-notifier.hh

Farshid Ghods added a comment - 01/Nov/12 11:20 AM Siri/Alk, on windows at least this looks like a regression since we changed erlang setting from +A16 to +S128:128 and maybe this is caused by having erlang run too many schedulers internallly. chisheng is going to restart windows cluster with S32:32 and confirm if he sees same behavior after a day

Mike Wiederhold added a comment - 01/Nov/12 12:57 PM Also, see MB-7070 for another report of this issue.

Steve Yen added a comment - 01/Nov/12 4:51 PM see also MB-7057

Jin Lim added a comment - 01/Nov/12 6:19 PM - edited 11/1/12 progress up to date: * Currently running 4-2 nodes rebalance test adopted from testrunner's perf/reb-1-out.conf * No issue so far * System configurations, 6 Windows R2 nodes cluster with 4G memory quota * 30 concurrent clients data access over 7m items * 10.3.121.142 / 10.3.2.123 / 10.3.2.124 / 10.3.2.125 / 10.3.2.126 / 10.3.2.127

Steve Yen added a comment - 02/Nov/12 1:32 PM reviewed in bug-scrub mtg... If we don't see this issue over next few days in latest builds, then can deprioritize / cannot-repo this issue.

Jin Lim added a comment - 03/Nov/12 12:45 AM - edited Ran into the same issue twice today. Please see below: <6-nodes-Windows-cluster> build:1930 test: 6m items + rebalance in/out (3 <-> 3) + 30 clients concurrent data access while rebalance failure: lost connection to mccouch -> flusher stuck for incomplete notification to mccouch -> rebalance timeout root cause: mccouch crash (while processing do_notify_update) snippet from ns log from 10.5.2.123:   [ns_server:info,2012-11-02T16:07:54.492, ns_1@10.3.2.123 :<0.8858.9>:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `default` [ns_server:info,2012-11-02T16:07:54.492, ns_1@10.3.2.123 :<0.8858.9>:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket default with config: [{database_fragmentation_threshold,{30,undefined}},  {view_fragmentation_threshold,{30,undefined}}] [ns_server:info,2012-11-02T16:07:59.383, ns_1@10.3.2.123 :ns_config_rep<0.19286.7>:ns_config_rep:do_pull:341]Pulling config from: ' ns_1@10.3.2.127 ' [error_logger:error,2012-11-02T16:08:06.758, ns_1@10.3.2.123 :error_logger<0.5.0>:ale_error_logger_handler:log_report:72] =========================CRASH REPORT=========================   crasher:     initial call: mc_connection:init/1     pid: <0.19932.7>     registered_name: []     exception error: no case clause matching                      {'EXIT',{{badmatch,eof},                               [{couch_file,load_header,2},                                {couch_file,handle_reader_message,4},                                {proc_lib,init_p_do_apply,3}]}}       in function mc_connection:do_notify_vbucket_update/3       in call from mc_connection:handle_message/9       in call from mc_connection:read_full_message/2       in call from mc_connection:run_loop/2     ancestors: [mc_conn_sup,mc_sup,ns_server_sup,ns_server_cluster_sup,                   <0.66.0>]     messages: []     links: [<0.19355.7>,#Port<0.68903>]     dictionary: []     trap_exit: false     status: running     heap_size: 2584     stack_size: 24     reductions: 569368   neighbours: [error_logger:error,2012-11-02T16:08:06.758, ns_1@10.3.2.123 :error_logger<0.5.0>:ale_error_logger_handler:log_report:72] =========================SUPERVISOR REPORT=========================      Supervisor: {local,mc_conn_sup}      Context: child_terminated      Reason: {case_clause,{'EXIT',{{badmatch,eof},                                        [{couch_file,load_header,2},                                         {couch_file,handle_reader_message,4},                                         {proc_lib,init_p_do_apply,3}]}}}      Offender: [{pid,<0.19932.7>},                   {name,mc_connection},                   {mfargs,{mc_connection,start_link,undefined}},                   {restart_type,temporary},                   {shutdown,brutal_kill},                   {child_type,worker}] *** After the above crash, ep engine eventually caught in the infinite retry of resetting connection to failed mccouch [ns_server:info,2012-11-02T16:08:08.992, ns_1@10.3.2.123 :ns_port_memcached<0.19362.7>:ns_port_server:log:171]memcached<0.19362.7>: Fri Nov 02 16:08:08.796687 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" memcached<0.19362.7>: Fri Nov 02 16:08:08.796687 Pacific Daylight Time 3: Connection closed by mccouch memcached<0.19362.7>: Fri Nov 02 16:08:08.796687 Pacific Daylight Time 3: Resetting connection to mccouch, lastReceivedCommand = select_bucket lastSentCommand = notify_vbucket_update currentCommand =unknown memcached<0.19362.7>: Fri Nov 02 16:08:08.796687 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" [error_logger:error,2012-11-02T16:08:09.805, ns_1@10.3.2.123 :error_logger<0.5.0>:ale_error_logger_handler:log_report:72] =========================CRASH REPORT=========================   crasher:     initial call: mc_connection:init/1     pid: <0.8963.9>     registered_name: []     exception error: no case clause matching                      {'EXIT',{{badmatch,eof},                               [{couch_file,load_header,2},                                {couch_file,handle_reader_message,4},                                {proc_lib,init_p_do_apply,3}]}}       in function mc_connection:do_notify_vbucket_update/3       in call from mc_connection:handle_message/9       in call from mc_connection:read_full_message/2       in call from mc_connection:run_loop/2     ancestors: [mc_conn_sup,mc_sup,ns_server_sup,ns_server_cluster_sup,                   <0.66.0>]     messages: []     links: [<0.19355.7>,#Port<0.81279>]     dictionary: []     trap_exit: false     status: running     heap_size: 987     stack_size: 24     reductions: 230   neighbours: [error_logger:error,2012-11-02T16:08:09.805, ns_1@10.3.2.123 :error_logger<0.5.0>:ale_error_logger_handler:log_report:72] =========================SUPERVISOR REPORT=========================      Supervisor: {local,mc_conn_sup}      Context: child_terminated      Reason: {case_clause,{'EXIT',{{badmatch,eof},                                        [{couch_file,load_header,2},                                         {couch_file,handle_reader_message,4},                                         {proc_lib,init_p_do_apply,3}]}}}      Offender: [{pid,<0.8963.9>},                   {name,mc_connection},                   {mfargs,{mc_connection,start_link,undefined}},                   {restart_type,temporary},                   {shutdown,brutal_kill},                   {child_type,worker}] [ns_server:info,2012-11-02T16:08:10.008, ns_1@10.3.2.123 :ns_port_memcached<0.19362.7>:ns_port_server:log:171]memcached<0.19362.7>: Fri Nov 02 16:08:09.812312 Pacific Daylight Time 3: Connected to mccouch: "localhost:11213" memcached<0.19362.7>: Fri Nov 02 16:08:09.812312 Pacific Daylight Time 3: Connection closed by mccouch memcached<0.19362.7>: Fri Nov 02 16:08:09.812312 Pacific Daylight Time 3: Resetting connection to mccouch, lastReceivedCommand = select_bucket lastSentCommand = notify_vbucket_update currentCommand =unknown memcached<0.19362.7>: Fri Nov 02 16:08:09.812312 Pacific Daylight Time 3: Trying to connect to mccouch: "localhost:11213" [error_logger:error,2012-11-02T16:08:10.820, ns_1@10.3.2.123 :error_logger<0.5.0>:ale_error_logger_handler:log_report:72] =========================CRASH REPORT=========================   crasher:     initial call: mc_connection:init/1     pid: <0.8964.9>     registered_name: []     exception error: no case clause matching                      {'EXIT',{{badmatch,eof},                               [{couch_file,load_header,2},                                {couch_file,handle_reader_message,4},                                {proc_lib,init_p_do_apply,3}]}}       in function mc_connection:do_notify_vbucket_update/3       in call from mc_connection:handle_message/9       in call from mc_connection:read_full_message/2       in call from mc_connection:run_loop/2     ancestors: [mc_conn_sup,mc_sup,ns_server_sup,ns_server_cluster_sup,                   <0.66.0>]     messages: []     links: [<0.19355.7>,#Port<0.81281>]     dictionary: []     trap_exit: false     status: running     heap_size: 987     stack_size: 24     reductions: 231   neighbours: <5-nodes-Centos-cluster> build: 1930 test: same as above failure & log snippets: pretty much the same manifestation of the above failure, except that ns_server timed out its select_bucket call to mccouch first and decalred the rebalance failure. error.13:[error_logger:error,2012-11-02T10:32:27.371, ns_1@10.5.2.30 :error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]Error in process <0.22250.25> on node ' ns_1@10.5.2.30 ' with exit value: {{wait_checkpoint_persisted_failed,"default",615,10,[{' ns_1@10.5.2.30 ',{'EXIT',{{{{badmatch,{error,timeout}},[{mc_client_binary,cmd_binary_vocal_recv,5},{mc_client_binary,select_bucket,2},{ns_memcached,ensure_bucket,2},... error.13:[error_logger:error,2012-11-02T10:32:27.371, ns_1@10.5.2.30 :error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]Error in process <0.22263.25> on node ' ns_1@10.5.2.30 ' with exit value: {{wait_checkpoint_persisted_failed,"default",410,8,[{' ns_1@10.5.2.30 ',{'EXIT',{{{{badmatch,{error,timeout}},[{mc_client_binary,cmd_binary_vocal_recv,5},{mc_client_binary,select_bucket,2},{ns_memcached,ensure_bucket,2},... error.13:[error_logger:error,2012-11-02T10:32:27.372, ns_1@10.5.2.30 :error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]Error in process <0.22272.25> on node ' ns_1@10.5.2.30 ' with exit value: {{wait_checkpoint_persisted_failed,"default",820,10,[{' ns_1@10.5.2.30 ',{'EXIT',{{{{badmatch,{error,timeout}},[{mc_client_binary,cmd_binary_vocal_recv,5},{mc_client_binary,select_bucket,2},{ns_memcached,ensure_bucket,2},... error.13:[error_logger:error,2012-11-02T10:32:27.372, ns_1@10.5.2.30 :error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]Error in process <0.22283.25> on node ' ns_1@10.5.2.30 ' with exit value: {{wait_checkpoint_persisted_failed,"default",257,8,[{' ns_1@10.5.2.30 ',{'EXIT',{{{{badmatch,{error,timeout}},[{mc_client_binary,cmd_binary_vocal_recv,5},{mc_client_binary,select_bucket,2},{ns_memcached,ensure_bucket,2},... Confirmed with the ns server team (Aliaksey A) that the first timeout above would have resulted in a rebalance failure.

Aleksey Kondratenko added a comment - 05/Nov/12 1:00 PM Everything is stuck in couchdb_updater. This couchdb_updater:          {<0.2393.0>,           [{registered_name,[]},            {status,waiting},            {initial_call,{proc_lib,init_p,5}},            {backtrace,                [<<"Program counter: 0x039342a4 (proc_lib:sync_wait/2 + 28)">>,                 <<"CP: 0x00000000 (invalid)">>,<<"arity = 0">>,<<>>,                 <<"0x15789bc0 Return addr 0x101e6488 (couch_db_updater:handle_call/3 + 392)">>,                 <<"y(0) infinity">>,<<"y(1) <0.19271.0>">>,<<>>,                 <<"0x15789bcc Return addr 0x03dba67c (gen_server:handle_msg/5 + 148)">>,                 <<"y(0) []">>,<<"y(1) []">>,<<"y(2) []">>,                 <<"(3) {db,<0.2392.0>,<0.2393.0>,<0.19266.0>,<<16 bytes>>,<0.2389.0>,<0.2394.0>,{db_heade">>,                 <<"y(4) \"e:/data/default/258.couch.1.compact\"">>,                 <<"y(5) []">>,                 <<"y(6) \"e:/data/default/258.couch.1\"">>,                 <<"y(7) <<11 bytes>>">>,<<"y(8) 7976">>,                 <<"(9) {btree,<0.2389.0>,{20009051,<<0 bytes>>,94},#Fun<couch_btree.3.59827385>,#Fun<couc">>,                 <<"y(10) <0.2389.0>">>,<<"y(11) <<16 bytes>>">>,                 <<"y(12) <0.2392.0>">>,<<>>,                 <<"0x15789c04 Return addr 0x03933e70 (proc_lib:init_p_do_apply/3 + 28)">>,                 <<"y(0) couch_db_updater">>,                 <<"(1) {db,<0.2392.0>,<0.2393.0>,<0.19266.0>,<<16 bytes>>,<0.2389.0>,<0.2394.0>,{db_heade">>,                 <<"y(2) <0.2393.0>">>,<<"y(3) <0.2392.0>">>,                 <<"y(4) {compact_done,\"e:/data/default/258.couch.1.compact\"}">>,                 <<"y(5) {<0.19266.0>,#Ref<0.0.13.148061>}">>,                 <<"y(6) Catch 0x03dba67c (gen_server:handle_msg/5 + 148)">>,                 <<>>,                 <<"0x15789c24 Return addr 0x00b009b4 (<terminate process normally>)">>,                 <<"y(0) Catch 0x03933e80 (proc_lib:init_p_do_apply/3 + 44)">>,                 <<>>]},            {error_handler,error_handler},            {garbage_collection,                [{min_bin_vheap_size,46368},                 {min_heap_size,233},                 {fullsweep_after,512},                 {minor_gcs,184}]},            {heap_size,4181},            {total_heap_size,4791},            {links,[<0.19266.0>,<0.19271.0>,<0.2392.0>]},            {memory,34380},            {message_queue_len,261},            {reductions,74898},            {trap_exit,true}]}, We see that a bunch of mccouch connections are stuck on gen_server:do_call-s to this folks. And you can see lots of messages in it's message queue.

Chiyoung Seo added a comment - 05/Nov/12 1:04 PM We asked the QE and perf team to reduce the erlang S paramter from 128 to 68, and didn't see this issue since last Friday night.

Aleksey Kondratenko added a comment - 05/Nov/12 1:11 PM Here's what happens: [couchdb:info,2012-10-28T21:02:53.384, ns_1@10.2.2.230 :<0.19272.0>:couch_log:info:39]eacces error opening file "e:/data/default/258.couch.1.compact" waiting infinityms to retry Somehow fd_close_after parameter is passed as CloseTimeout to spawn_reader which passes it to try_open_fd as Timewait. Causing everything to stuck

Aleksey Kondratenko added a comment - 05/Nov/12 1:11 PM CC-ing Filipe

Aleksey Kondratenko added a comment - 05/Nov/12 1:11 PM Assigning to back to Damien

Steve Yen added a comment - 05/Nov/12 1:26 PM related to the recent windows-specific (?) eaccess changes / infinity (ask Alk).

Steve Yen added a comment - 06/Nov/12 2:00 PM fix now on gerrit

Thuan Nguyen added a comment - 06/Nov/12 11:37 PM Integrated in github-couchdb-preview #534 (See [ http://qa.hq.northscale.net/job/github-couchdb-preview/534/ ])      MB-7046 : Fix file open retry interval to never be infinity (Revision dd6de6a0525ba4b8c3d5768bc4eefd9673e688ea)      Result = SUCCESS steve : Files : * src/couchdb/couch_file.erl