Details
-
Type:
Bug
-
Status:
Closed
-
Priority:
Critical
-
Resolution: Won't Fix
-
Affects Version/s: 1.8.1-release-candidate
-
Fix Version/s: 2.0
-
Component/s: couchbase-bucket
-
Security Level: Public
-
Labels:None
-
Environment:1.8.1-815-rel
2 buckets
Centos 64 bit
Description
Test case failing:-
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1
Basic swaprebalance of 1 node
Test cases failing
swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=2,num-swap=3,swap-orchestrator=True
1) Failover 3 nodes, including the orchestrator
2) rebalance them out
3) Add back the 3 node
4) Rebalance in
5) Results in data loss
Cluster @ 10.1.3.74
Seems to be regression from http://www.couchbase.com/issues/browse/MB-5193
couchbase:testrunner karan$ python scripts/ssh.py -i swap.ini "/opt/couchbase/bin/cbstats localhost:11210 all bucket-0 password |grep items"
10.1.3.74
curr_items: 15552
curr_items_tot: 46638
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 47228
ep_total_del_items: 0
ep_total_new_items: 81495
ep_uncommitted_items: 0
vb_active_curr_items: 15552
vb_pending_curr_items: 0
vb_replica_curr_items: 31086
10.1.3.79
curr_items: 13930
curr_items_tot: 41763
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 42282
ep_total_del_items: 2438
ep_total_new_items: 227195
ep_uncommitted_items: 0
vb_active_curr_items: 13930
vb_pending_curr_items: 0
vb_replica_curr_items: 27833
10.1.3.80
curr_items: 13933
curr_items_tot: 41584
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 42096
ep_total_del_items: 2488
ep_total_new_items: 224318
ep_uncommitted_items: 0
vb_active_curr_items: 13933
vb_pending_curr_items: 0
vb_replica_curr_items: 27651
10.1.3.76
curr_items: 13975
curr_items_tot: 41826
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 42377
ep_total_del_items: 2576
ep_total_new_items: 197397
ep_uncommitted_items: 0
vb_active_curr_items: 13975
vb_pending_curr_items: 0
vb_replica_curr_items: 27851
10.1.3.75
curr_items: 13702
curr_items_tot: 43174
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 43704
ep_total_del_items: 2477
ep_total_new_items: 232395
ep_uncommitted_items: 0
vb_active_curr_items: 13702
vb_pending_curr_items: 0
vb_replica_curr_items: 29472
10.1.3.77
curr_items: 13894
curr_items_tot: 38447
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 38896
ep_total_del_items: 0
ep_total_new_items: 77506
ep_uncommitted_items: 0
vb_active_curr_items: 13894
vb_pending_curr_items: 0
vb_replica_curr_items: 24553
10.1.3.78
curr_items: 13852
curr_items_tot: 38553
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 39027
ep_total_del_items: 0
ep_total_new_items: 70145
ep_uncommitted_items: 0
vb_active_curr_items: 13852
vb_pending_curr_items: 0
vb_replica_curr_items: 24701
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1
Basic swaprebalance of 1 node
Test cases failing
swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=2,num-swap=3,swap-orchestrator=True
1) Failover 3 nodes, including the orchestrator
2) rebalance them out
3) Add back the 3 node
4) Rebalance in
5) Results in data loss
Cluster @ 10.1.3.74
Seems to be regression from http://www.couchbase.com/issues/browse/MB-5193
couchbase:testrunner karan$ python scripts/ssh.py -i swap.ini "/opt/couchbase/bin/cbstats localhost:11210 all bucket-0 password |grep items"
10.1.3.74
curr_items: 15552
curr_items_tot: 46638
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 47228
ep_total_del_items: 0
ep_total_new_items: 81495
ep_uncommitted_items: 0
vb_active_curr_items: 15552
vb_pending_curr_items: 0
vb_replica_curr_items: 31086
10.1.3.79
curr_items: 13930
curr_items_tot: 41763
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 42282
ep_total_del_items: 2438
ep_total_new_items: 227195
ep_uncommitted_items: 0
vb_active_curr_items: 13930
vb_pending_curr_items: 0
vb_replica_curr_items: 27833
10.1.3.80
curr_items: 13933
curr_items_tot: 41584
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 42096
ep_total_del_items: 2488
ep_total_new_items: 224318
ep_uncommitted_items: 0
vb_active_curr_items: 13933
vb_pending_curr_items: 0
vb_replica_curr_items: 27651
10.1.3.76
curr_items: 13975
curr_items_tot: 41826
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 42377
ep_total_del_items: 2576
ep_total_new_items: 197397
ep_uncommitted_items: 0
vb_active_curr_items: 13975
vb_pending_curr_items: 0
vb_replica_curr_items: 27851
10.1.3.75
curr_items: 13702
curr_items_tot: 43174
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 43704
ep_total_del_items: 2477
ep_total_new_items: 232395
ep_uncommitted_items: 0
vb_active_curr_items: 13702
vb_pending_curr_items: 0
vb_replica_curr_items: 29472
10.1.3.77
curr_items: 13894
curr_items_tot: 38447
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 38896
ep_total_del_items: 0
ep_total_new_items: 77506
ep_uncommitted_items: 0
vb_active_curr_items: 13894
vb_pending_curr_items: 0
vb_replica_curr_items: 24553
10.1.3.78
curr_items: 13852
curr_items_tot: 38553
ep_diskqueue_items: 0
ep_items_rm_from_checkpoints: 39027
ep_total_del_items: 0
ep_total_new_items: 70145
ep_uncommitted_items: 0
vb_active_curr_items: 13852
vb_pending_curr_items: 0
vb_replica_curr_items: 24701
-
- 10.1.3.74-8091-diag.txt.gz
- 14/May/12 2:47 PM
- 496 kB
- Karan Kumar
-
- 10.1.3.75-8091-diag.txt.gz
- 14/May/12 2:47 PM
- 554 kB
- Karan Kumar
-
- 10.1.3.76-8091-diag.txt.gz
- 14/May/12 2:47 PM
- 550 kB
- Karan Kumar
-
- 10.1.3.77-8091-diag.txt.gz
- 14/May/12 2:47 PM
- 474 kB
- Karan Kumar
-
- 10.1.3.78-8091-diag.txt.gz
- 14/May/12 2:47 PM
- 470 kB
- Karan Kumar
-
- 10.1.3.79-8091-diag.txt.gz
- 14/May/12 2:47 PM
- 554 kB
- Karan Kumar
-
- 10.1.3.80-8091-diag.txt.gz
- 14/May/12 2:47 PM
- 545 kB
- Karan Kumar
-
- 12d7883a-fb19-4a34-a9c5-4ae67daacf30-10.1.3.74-diag.gz
- 15/May/12 2:03 PM
- 276 kB
- Karan Kumar
-
- 12d7883a-fb19-4a34-a9c5-4ae67daacf30-10.1.3.75-diag.gz
- 15/May/12 2:03 PM
- 356 kB
- Karan Kumar
-
- 12d7883a-fb19-4a34-a9c5-4ae67daacf30-10.1.3.76-diag.gz
- 15/May/12 2:03 PM
- 325 kB
- Karan Kumar
-
- 12d7883a-fb19-4a34-a9c5-4ae67daacf30-10.1.3.77-diag.gz
- 15/May/12 2:03 PM
- 257 kB
- Karan Kumar
-
- 12d7883a-fb19-4a34-a9c5-4ae67daacf30-10.1.3.78-diag.gz
- 15/May/12 2:03 PM
- 241 kB
- Karan Kumar
-
- 12d7883a-fb19-4a34-a9c5-4ae67daacf30-10.1.3.79-diag.gz
- 15/May/12 2:03 PM
- 370 kB
- Karan Kumar
-
- 12d7883a-fb19-4a34-a9c5-4ae67daacf30-10.1.3.80-diag.gz
- 15/May/12 2:03 PM
- 334 kB
- Karan Kumar
Activity
- All
- Comments
- Work Log
- History
- Activity
- Gerrit Reviews
Hide
Karan Kumar
added a comment -
The following replica vbuckets have 0 items.
10.1.3.78
vb_50:counted: 0
10.1.3.77
vb_42:counted: 0
vb_50:counted: 0
10.1.3.78
vb_50:counted: 0
10.1.3.77
vb_42:counted: 0
vb_50:counted: 0
Show
Karan Kumar
added a comment - The following replica vbuckets have 0 items.
10.1.3.78
vb_50:counted: 0
10.1.3.77
vb_42:counted: 0
vb_50:counted: 0
Show
Karan Kumar
added a comment - Attachings diags from the latest run with the data loss issue.
Hide
Thuan Nguyen
added a comment -
Integrated in github-ep-engine-2-0 #282 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/282/])
MB-5297 Fix to the bug in removing vbucket from backfill vb list (Revision 726d29ead13859b7a2237408e7e8483b9c4d590b)
MB-5297 Don't check backfill completion from backfill tasks. (Revision b66b983e5e05c445d6bfe4cb59d996ff883b2f20)
MB-5297 Run TAP backfill task after sending all OPAQUE messages. (Revision 10e986221564e60939343d8079ef0ae3832f8ce7)
Result = SUCCESS
Chiyoung Seo :
Files :
* tapconnection.cc
Chiyoung Seo :
Files :
* tapconnmap.cc
* backfill.cc
* tapconnmap.hh
Chiyoung Seo :
Files :
* ep_engine.cc
Result = SUCCESS
Chiyoung Seo :
Files :
* tapconnection.cc
Chiyoung Seo :
Files :
* tapconnmap.cc
* backfill.cc
* tapconnmap.hh
Chiyoung Seo :
Files :
* ep_engine.cc
Show
Thuan Nguyen
added a comment - Integrated in github-ep-engine-2-0 #282 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/282/ ])
MB-5297 Fix to the bug in removing vbucket from backfill vb list (Revision 726d29ead13859b7a2237408e7e8483b9c4d590b)
MB-5297 Don't check backfill completion from backfill tasks. (Revision b66b983e5e05c445d6bfe4cb59d996ff883b2f20)
MB-5297 Run TAP backfill task after sending all OPAQUE messages. (Revision 10e986221564e60939343d8079ef0ae3832f8ce7)
Result = SUCCESS
Chiyoung Seo :
Files :
* tapconnection.cc
Chiyoung Seo :
Files :
* tapconnmap.cc
* backfill.cc
* tapconnmap.hh
Chiyoung Seo :
Files :
* ep_engine.cc
Show
Karan Kumar
added a comment - Can still repro on 181-825 build.
Show
Chiyoung Seo
added a comment - http://review.couchbase.org/#change,16201
Hide
Thuan Nguyen
added a comment -
Integrated in github-ep-engine-2-0 #289 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/289/])
MB-5297 Send INITIAL_VBUCKET_STREAM message for new vbuckets only (Revision 7618bc065447587f9068b2899ad7451f4ec09f0e)
MB-5297 doRunBackfill flag should be set in backfill functions (Revision 0a18607125b88a4adaefbd38124ad91761234f2d)
MB-5297 Initialize TAP producer within TapConnMap::newProducer() (Revision c781787ac3b9d0262b5ba44e4fa2b37c9bbee3f2)
MB-5297 Check vbucket filter before adding items to TAP queue (Revision 940802dcfaf6c6228ea5dd65bcaf886a1bd32896)
Result = SUCCESS
Chiyoung Seo :
Files :
* tapconnection.cc
Chiyoung Seo :
Files :
* tapconnection.cc
Chiyoung Seo :
Files :
* tapconnmap.cc
* tapconnection.hh
* tapconnection.cc
* ep_engine.cc
* tapconnmap.hh
Chiyoung Seo :
Files :
* tapconnection.cc
Result = SUCCESS
Chiyoung Seo :
Files :
* tapconnection.cc
Chiyoung Seo :
Files :
* tapconnection.cc
Chiyoung Seo :
Files :
* tapconnmap.cc
* tapconnection.hh
* tapconnection.cc
* ep_engine.cc
* tapconnmap.hh
Chiyoung Seo :
Files :
* tapconnection.cc
Show
Thuan Nguyen
added a comment - Integrated in github-ep-engine-2-0 #289 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/289/ ])
MB-5297 Send INITIAL_VBUCKET_STREAM message for new vbuckets only (Revision 7618bc065447587f9068b2899ad7451f4ec09f0e)
MB-5297 doRunBackfill flag should be set in backfill functions (Revision 0a18607125b88a4adaefbd38124ad91761234f2d)
MB-5297 Initialize TAP producer within TapConnMap::newProducer() (Revision c781787ac3b9d0262b5ba44e4fa2b37c9bbee3f2)
MB-5297 Check vbucket filter before adding items to TAP queue (Revision 940802dcfaf6c6228ea5dd65bcaf886a1bd32896)
Result = SUCCESS
Chiyoung Seo :
Files :
* tapconnection.cc
Chiyoung Seo :
Files :
* tapconnection.cc
Chiyoung Seo :
Files :
* tapconnmap.cc
* tapconnection.hh
* tapconnection.cc
* ep_engine.cc
* tapconnmap.hh
Chiyoung Seo :
Files :
* tapconnection.cc
Hide
Karan Kumar
added a comment -
Build: 181-851-rel
Test case failing:-
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1
It does expiry/deletes. Swap one node at the same time.
For bucket-1
Total active items 96021
Total expected replica items 192042
But we have 192041 (1 items missing) . Maybe with more items loaded we would see more loss.
Test case failing:-
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1
It does expiry/deletes. Swap one node at the same time.
For bucket-1
Total active items 96021
Total expected replica items 192042
But we have 192041 (1 items missing) . Maybe with more items loaded we would see more loss.
Show
Karan Kumar
added a comment - Build: 181-851-rel
Test case failing:-
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1
It does expiry/deletes. Swap one node at the same time.
For bucket-1
Total active items 96021
Total expected replica items 192042
But we have 192041 (1 items missing) . Maybe with more items loaded we would see more loss.
Hide
Karan Kumar
added a comment -
Tap stats from all the nodes.
10.1.3.75
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 4
ep_tap_deletes: 6027
ep_tap_fg_fetched: 423533
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 155148
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 507585
ep_tap_total_queue: 0
eq_tapq:anon_2068:connected: true
eq_tapq:anon_2068:created: 806
eq_tapq:anon_2068:num_checkpoint_end: 319
eq_tapq:anon_2068:num_checkpoint_end_failed: 0
eq_tapq:anon_2068:num_checkpoint_start: 325
eq_tapq:anon_2068:num_checkpoint_start_failed: 0
eq_tapq:anon_2068:num_delete: 13
eq_tapq:anon_2068:num_delete_failed: 0
eq_tapq:anon_2068:num_flush: 0
eq_tapq:anon_2068:num_flush_failed: 0
eq_tapq:anon_2068:num_mutation: 182
eq_tapq:anon_2068:num_mutation_failed: 0
eq_tapq:anon_2068:num_opaque: 12
eq_tapq:anon_2068:num_opaque_failed: 0
eq_tapq:anon_2068:num_unknown: 0
eq_tapq:anon_2068:num_vbucket_set: 0
eq_tapq:anon_2068:num_vbucket_set_failed: 0
eq_tapq:anon_2068:pending_disconnect: false
eq_tapq:anon_2068:reserved: 0
eq_tapq:anon_2068:supports_ack: true
eq_tapq:anon_2068:type: consumer
eq_tapq:anon_4104:connected: true
eq_tapq:anon_4104:created: 1215
eq_tapq:anon_4104:num_checkpoint_end: 341
eq_tapq:anon_4104:num_checkpoint_end_failed: 0
eq_tapq:anon_4104:num_checkpoint_start: 353
eq_tapq:anon_4104:num_checkpoint_start_failed: 0
eq_tapq:anon_4104:num_delete: 11
eq_tapq:anon_4104:num_delete_failed: 0
eq_tapq:anon_4104:num_flush: 0
eq_tapq:anon_4104:num_flush_failed: 0
eq_tapq:anon_4104:num_mutation: 110
eq_tapq:anon_4104:num_mutation_failed: 0
eq_tapq:anon_4104:num_opaque: 24
eq_tapq:anon_4104:num_opaque_failed: 0
eq_tapq:anon_4104:num_unknown: 0
eq_tapq:anon_4104:num_vbucket_set: 0
eq_tapq:anon_4104:num_vbucket_set_failed: 0
eq_tapq:anon_4104:pending_disconnect: false
eq_tapq:anon_4104:reserved: 0
eq_tapq:anon_4104:supports_ack: true
eq_tapq:anon_4104:type: consumer
eq_tapq:replication_ns_1@10.1.3.74:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_seqno: 229803
eq_tapq:replication_ns_1@10.1.3.74:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.74:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.74:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.74:complete: false
eq_tapq:replication_ns_1@10.1.3.74:connected: true
eq_tapq:replication_ns_1@10.1.3.74:created: 52
eq_tapq:replication_ns_1@10.1.3.74:disconnects: 682
eq_tapq:replication_ns_1@10.1.3.74:empty: true
eq_tapq:replication_ns_1@10.1.3.74:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.74:has_item: false
eq_tapq:replication_ns_1@10.1.3.74:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.74:idle: true
eq_tapq:replication_ns_1@10.1.3.74:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.74:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.74:paused: 1
eq_tapq:replication_ns_1@10.1.3.74:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:qlen: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_drain: 94493
eq_tapq:replication_ns_1@10.1.3.74:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.74:rec_fetched: 163176
eq_tapq:replication_ns_1@10.1.3.74:reconnects: 682
eq_tapq:replication_ns_1@10.1.3.74:recv_ack_seqno: 229802
eq_tapq:replication_ns_1@10.1.3.74:reserved: 1
eq_tapq:replication_ns_1@10.1.3.74:seqno_ack_requested: 229802
eq_tapq:replication_ns_1@10.1.3.74:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.74:suspended: false
eq_tapq:replication_ns_1@10.1.3.74:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:total_noops: 156
eq_tapq:replication_ns_1@10.1.3.74:type: producer
eq_tapq:replication_ns_1@10.1.3.74:vb_filter: { [342,512], [854,1023] }
eq_tapq:replication_ns_1@10.1.3.74:vb_filters: 341
eq_tapq:replication_ns_1@10.1.3.78:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_seqno: 64524
eq_tapq:replication_ns_1@10.1.3.78:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.78:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.78:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.78:complete: false
eq_tapq:replication_ns_1@10.1.3.78:connected: true
eq_tapq:replication_ns_1@10.1.3.78:created: 100
eq_tapq:replication_ns_1@10.1.3.78:disconnects: 340
eq_tapq:replication_ns_1@10.1.3.78:empty: true
eq_tapq:replication_ns_1@10.1.3.78:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.78:has_item: false
eq_tapq:replication_ns_1@10.1.3.78:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.78:idle: true
eq_tapq:replication_ns_1@10.1.3.78:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.78:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.78:paused: 1
eq_tapq:replication_ns_1@10.1.3.78:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:qlen: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_drain: 60655
eq_tapq:replication_ns_1@10.1.3.78:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.78:rec_fetched: 63313
eq_tapq:replication_ns_1@10.1.3.78:reconnects: 340
eq_tapq:replication_ns_1@10.1.3.78:recv_ack_seqno: 64523
eq_tapq:replication_ns_1@10.1.3.78:reserved: 1
eq_tapq:replication_ns_1@10.1.3.78:seqno_ack_requested: 64523
eq_tapq:replication_ns_1@10.1.3.78:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.78:suspended: false
eq_tapq:replication_ns_1@10.1.3.78:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:total_noops: 165
eq_tapq:replication_ns_1@10.1.3.78:type: producer
eq_tapq:replication_ns_1@10.1.3.78:vb_filter: { [0,170], [513,682] }
eq_tapq:replication_ns_1@10.1.3.78:vb_filters: 341
10.1.3.80
Traceback (most recent call last):
File "/opt/couchbase/lib/python/cbstats", line 320, in ?
main()
File "/opt/couchbase/lib/python/cbstats", line 317, in main
c.execute()
File "/opt/couchbase/lib/python/clitool.py", line 43, in execute
f[0](mc, *args[2:], **opts.__dict__)
File "/opt/couchbase/lib/python/cbstats", line 27, in g
mc.sasl_auth_plain(username, password)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 191, in sasl_auth_plain
return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password]))
File "/opt/couchbase/lib/python/mc_bin_client.py", line 187, in sasl_auth_start
return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 104, in _doCmd
return self._handleSingleResponse(opaque)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 97, in _handleSingleResponse
cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 93, in _handleKeyedResponse
raise MemcachedError(errcode, rv)
mc_bin_client.MemcachedError: Memcached error #32: Auth failure
10.1.3.78
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 4
ep_tap_deletes: 1180
ep_tap_fg_fetched: 38134
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 39314
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 43391
ep_tap_total_queue: 0
eq_tapq:anon_2716:connected: true
eq_tapq:anon_2716:created: 1120
eq_tapq:anon_2716:num_checkpoint_end: 341
eq_tapq:anon_2716:num_checkpoint_end_failed: 0
eq_tapq:anon_2716:num_checkpoint_start: 342
eq_tapq:anon_2716:num_checkpoint_start_failed: 0
eq_tapq:anon_2716:num_delete: 2
eq_tapq:anon_2716:num_delete_failed: 0
eq_tapq:anon_2716:num_flush: 0
eq_tapq:anon_2716:num_flush_failed: 0
eq_tapq:anon_2716:num_mutation: 8
eq_tapq:anon_2716:num_mutation_failed: 0
eq_tapq:anon_2716:num_opaque: 2
eq_tapq:anon_2716:num_opaque_failed: 0
eq_tapq:anon_2716:num_unknown: 0
eq_tapq:anon_2716:num_vbucket_set: 0
eq_tapq:anon_2716:num_vbucket_set_failed: 0
eq_tapq:anon_2716:pending_disconnect: false
eq_tapq:anon_2716:reserved: 0
eq_tapq:anon_2716:supports_ack: true
eq_tapq:anon_2716:type: consumer
eq_tapq:anon_2727:connected: true
eq_tapq:anon_2727:created: 1124
eq_tapq:anon_2727:num_checkpoint_end: 342
eq_tapq:anon_2727:num_checkpoint_end_failed: 0
eq_tapq:anon_2727:num_checkpoint_start: 343
eq_tapq:anon_2727:num_checkpoint_start_failed: 0
eq_tapq:anon_2727:num_delete: 0
eq_tapq:anon_2727:num_delete_failed: 0
eq_tapq:anon_2727:num_flush: 0
eq_tapq:anon_2727:num_flush_failed: 0
eq_tapq:anon_2727:num_mutation: 11
eq_tapq:anon_2727:num_mutation_failed: 0
eq_tapq:anon_2727:num_opaque: 2
eq_tapq:anon_2727:num_opaque_failed: 0
eq_tapq:anon_2727:num_unknown: 0
eq_tapq:anon_2727:num_vbucket_set: 0
eq_tapq:anon_2727:num_vbucket_set_failed: 0
eq_tapq:anon_2727:pending_disconnect: false
eq_tapq:anon_2727:reserved: 0
eq_tapq:anon_2727:supports_ack: true
eq_tapq:anon_2727:type: consumer
eq_tapq:replication_ns_1@10.1.3.74:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_seqno: 37333
eq_tapq:replication_ns_1@10.1.3.74:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.74:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.74:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.74:complete: false
eq_tapq:replication_ns_1@10.1.3.74:connected: true
eq_tapq:replication_ns_1@10.1.3.74:created: 7
eq_tapq:replication_ns_1@10.1.3.74:disconnects: 340
eq_tapq:replication_ns_1@10.1.3.74:empty: true
eq_tapq:replication_ns_1@10.1.3.74:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.74:has_item: false
eq_tapq:replication_ns_1@10.1.3.74:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.74:idle: true
eq_tapq:replication_ns_1@10.1.3.74:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.74:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.74:paused: 1
eq_tapq:replication_ns_1@10.1.3.74:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:qlen: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_drain: 34333
eq_tapq:replication_ns_1@10.1.3.74:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.74:rec_fetched: 36402
eq_tapq:replication_ns_1@10.1.3.74:reconnects: 340
eq_tapq:replication_ns_1@10.1.3.74:recv_ack_seqno: 37332
eq_tapq:replication_ns_1@10.1.3.74:reserved: 1
eq_tapq:replication_ns_1@10.1.3.74:seqno_ack_requested: 37332
eq_tapq:replication_ns_1@10.1.3.74:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.74:suspended: false
eq_tapq:replication_ns_1@10.1.3.74:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:total_noops: 155
eq_tapq:replication_ns_1@10.1.3.74:type: producer
eq_tapq:replication_ns_1@10.1.3.74:vb_filter: { [513,853] }
eq_tapq:replication_ns_1@10.1.3.74:vb_filters: 341
eq_tapq:replication_ns_1@10.1.3.75:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_seqno: 7830
eq_tapq:replication_ns_1@10.1.3.75:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.75:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.75:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.75:complete: false
eq_tapq:replication_ns_1@10.1.3.75:connected: true
eq_tapq:replication_ns_1@10.1.3.75:created: 715
eq_tapq:replication_ns_1@10.1.3.75:disconnects: 340
eq_tapq:replication_ns_1@10.1.3.75:empty: true
eq_tapq:replication_ns_1@10.1.3.75:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.75:has_item: false
eq_tapq:replication_ns_1@10.1.3.75:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.75:idle: true
eq_tapq:replication_ns_1@10.1.3.75:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.75:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.75:paused: 1
eq_tapq:replication_ns_1@10.1.3.75:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:qlen: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_drain: 4981
eq_tapq:replication_ns_1@10.1.3.75:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.75:rec_fetched: 6989
eq_tapq:replication_ns_1@10.1.3.75:reconnects: 340
eq_tapq:replication_ns_1@10.1.3.75:recv_ack_seqno: 7829
eq_tapq:replication_ns_1@10.1.3.75:reserved: 1
eq_tapq:replication_ns_1@10.1.3.75:seqno_ack_requested: 7829
eq_tapq:replication_ns_1@10.1.3.75:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.75:suspended: false
eq_tapq:replication_ns_1@10.1.3.75:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:total_noops: 130
eq_tapq:replication_ns_1@10.1.3.75:type: producer
eq_tapq:replication_ns_1@10.1.3.75:vb_filter: { [171,341], [854,1023] }
eq_tapq:replication_ns_1@10.1.3.75:vb_filters: 341
10.1.3.74
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 4
ep_tap_deletes: 6531
ep_tap_fg_fetched: 436209
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 283655
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 459312
ep_tap_total_queue: 0
eq_tapq:anon_8195:connected: true
eq_tapq:anon_8195:created: 2560
eq_tapq:anon_8195:num_checkpoint_end: 220
eq_tapq:anon_8195:num_checkpoint_end_failed: 0
eq_tapq:anon_8195:num_checkpoint_start: 222
eq_tapq:anon_8195:num_checkpoint_start_failed: 0
eq_tapq:anon_8195:num_delete: 4
eq_tapq:anon_8195:num_delete_failed: 0
eq_tapq:anon_8195:num_flush: 0
eq_tapq:anon_8195:num_flush_failed: 0
eq_tapq:anon_8195:num_mutation: 16
eq_tapq:anon_8195:num_mutation_failed: 0
eq_tapq:anon_8195:num_opaque: 2
eq_tapq:anon_8195:num_opaque_failed: 0
eq_tapq:anon_8195:num_unknown: 0
eq_tapq:anon_8195:num_vbucket_set: 0
eq_tapq:anon_8195:num_vbucket_set_failed: 0
eq_tapq:anon_8195:pending_disconnect: false
eq_tapq:anon_8195:reserved: 0
eq_tapq:anon_8195:supports_ack: true
eq_tapq:anon_8195:type: consumer
eq_tapq:anon_8204:connected: true
eq_tapq:anon_8204:created: 2563
eq_tapq:anon_8204:num_checkpoint_end: 341
eq_tapq:anon_8204:num_checkpoint_end_failed: 0
eq_tapq:anon_8204:num_checkpoint_start: 349
eq_tapq:anon_8204:num_checkpoint_start_failed: 0
eq_tapq:anon_8204:num_delete: 104
eq_tapq:anon_8204:num_delete_failed: 0
eq_tapq:anon_8204:num_flush: 0
eq_tapq:anon_8204:num_flush_failed: 0
eq_tapq:anon_8204:num_mutation: 1287
eq_tapq:anon_8204:num_mutation_failed: 0
eq_tapq:anon_8204:num_opaque: 10
eq_tapq:anon_8204:num_opaque_failed: 0
eq_tapq:anon_8204:num_unknown: 0
eq_tapq:anon_8204:num_vbucket_set: 0
eq_tapq:anon_8204:num_vbucket_set_failed: 0
eq_tapq:anon_8204:pending_disconnect: false
eq_tapq:anon_8204:reserved: 0
eq_tapq:anon_8204:supports_ack: true
eq_tapq:anon_8204:type: consumer
eq_tapq:replication_ns_1@10.1.3.75:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_seqno: 238912
eq_tapq:replication_ns_1@10.1.3.75:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.75:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.75:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.75:complete: false
eq_tapq:replication_ns_1@10.1.3.75:connected: true
eq_tapq:replication_ns_1@10.1.3.75:created: 1403
eq_tapq:replication_ns_1@10.1.3.75:disconnects: 684
eq_tapq:replication_ns_1@10.1.3.75:empty: true
eq_tapq:replication_ns_1@10.1.3.75:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.75:has_item: false
eq_tapq:replication_ns_1@10.1.3.75:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.75:idle: true
eq_tapq:replication_ns_1@10.1.3.75:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.75:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.75:paused: 1
eq_tapq:replication_ns_1@10.1.3.75:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:qlen: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_drain: 225828
eq_tapq:replication_ns_1@10.1.3.75:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.75:rec_fetched: 233535
eq_tapq:replication_ns_1@10.1.3.75:reconnects: 684
eq_tapq:replication_ns_1@10.1.3.75:recv_ack_seqno: 238911
eq_tapq:replication_ns_1@10.1.3.75:reserved: 1
eq_tapq:replication_ns_1@10.1.3.75:seqno_ack_requested: 238911
eq_tapq:replication_ns_1@10.1.3.75:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.75:suspended: false
eq_tapq:replication_ns_1@10.1.3.75:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:total_noops: 155
eq_tapq:replication_ns_1@10.1.3.75:type: producer
eq_tapq:replication_ns_1@10.1.3.75:vb_filter: { [0,170], [683,853] }
eq_tapq:replication_ns_1@10.1.3.75:vb_filters: 342
eq_tapq:replication_ns_1@10.1.3.78:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_seqno: 61641
eq_tapq:replication_ns_1@10.1.3.78:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.78:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.78:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.78:complete: false
eq_tapq:replication_ns_1@10.1.3.78:connected: true
eq_tapq:replication_ns_1@10.1.3.78:created: 1447
eq_tapq:replication_ns_1@10.1.3.78:disconnects: 341
eq_tapq:replication_ns_1@10.1.3.78:empty: true
eq_tapq:replication_ns_1@10.1.3.78:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.78:has_item: false
eq_tapq:replication_ns_1@10.1.3.78:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.78:idle: true
eq_tapq:replication_ns_1@10.1.3.78:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.78:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.78:paused: 1
eq_tapq:replication_ns_1@10.1.3.78:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:qlen: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_drain: 57827
eq_tapq:replication_ns_1@10.1.3.78:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.78:rec_fetched: 60456
eq_tapq:replication_ns_1@10.1.3.78:reconnects: 341
eq_tapq:replication_ns_1@10.1.3.78:recv_ack_seqno: 61640
eq_tapq:replication_ns_1@10.1.3.78:reserved: 1
eq_tapq:replication_ns_1@10.1.3.78:seqno_ack_requested: 61640
eq_tapq:replication_ns_1@10.1.3.78:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.78:suspended: false
eq_tapq:replication_ns_1@10.1.3.78:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:total_noops: 159
eq_tapq:replication_ns_1@10.1.3.78:type: producer
eq_tapq:replication_ns_1@10.1.3.78:vb_filter: { [171,512] }
eq_tapq:replication_ns_1@10.1.3.78:vb_filters: 342
10.1.3.75
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 4
ep_tap_deletes: 6027
ep_tap_fg_fetched: 423533
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 155148
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 507585
ep_tap_total_queue: 0
eq_tapq:anon_2068:connected: true
eq_tapq:anon_2068:created: 806
eq_tapq:anon_2068:num_checkpoint_end: 319
eq_tapq:anon_2068:num_checkpoint_end_failed: 0
eq_tapq:anon_2068:num_checkpoint_start: 325
eq_tapq:anon_2068:num_checkpoint_start_failed: 0
eq_tapq:anon_2068:num_delete: 13
eq_tapq:anon_2068:num_delete_failed: 0
eq_tapq:anon_2068:num_flush: 0
eq_tapq:anon_2068:num_flush_failed: 0
eq_tapq:anon_2068:num_mutation: 182
eq_tapq:anon_2068:num_mutation_failed: 0
eq_tapq:anon_2068:num_opaque: 12
eq_tapq:anon_2068:num_opaque_failed: 0
eq_tapq:anon_2068:num_unknown: 0
eq_tapq:anon_2068:num_vbucket_set: 0
eq_tapq:anon_2068:num_vbucket_set_failed: 0
eq_tapq:anon_2068:pending_disconnect: false
eq_tapq:anon_2068:reserved: 0
eq_tapq:anon_2068:supports_ack: true
eq_tapq:anon_2068:type: consumer
eq_tapq:anon_4104:connected: true
eq_tapq:anon_4104:created: 1215
eq_tapq:anon_4104:num_checkpoint_end: 341
eq_tapq:anon_4104:num_checkpoint_end_failed: 0
eq_tapq:anon_4104:num_checkpoint_start: 353
eq_tapq:anon_4104:num_checkpoint_start_failed: 0
eq_tapq:anon_4104:num_delete: 11
eq_tapq:anon_4104:num_delete_failed: 0
eq_tapq:anon_4104:num_flush: 0
eq_tapq:anon_4104:num_flush_failed: 0
eq_tapq:anon_4104:num_mutation: 110
eq_tapq:anon_4104:num_mutation_failed: 0
eq_tapq:anon_4104:num_opaque: 24
eq_tapq:anon_4104:num_opaque_failed: 0
eq_tapq:anon_4104:num_unknown: 0
eq_tapq:anon_4104:num_vbucket_set: 0
eq_tapq:anon_4104:num_vbucket_set_failed: 0
eq_tapq:anon_4104:pending_disconnect: false
eq_tapq:anon_4104:reserved: 0
eq_tapq:anon_4104:supports_ack: true
eq_tapq:anon_4104:type: consumer
eq_tapq:replication_ns_1@10.1.3.74:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_seqno: 229803
eq_tapq:replication_ns_1@10.1.3.74:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.74:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.74:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.74:complete: false
eq_tapq:replication_ns_1@10.1.3.74:connected: true
eq_tapq:replication_ns_1@10.1.3.74:created: 52
eq_tapq:replication_ns_1@10.1.3.74:disconnects: 682
eq_tapq:replication_ns_1@10.1.3.74:empty: true
eq_tapq:replication_ns_1@10.1.3.74:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.74:has_item: false
eq_tapq:replication_ns_1@10.1.3.74:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.74:idle: true
eq_tapq:replication_ns_1@10.1.3.74:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.74:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.74:paused: 1
eq_tapq:replication_ns_1@10.1.3.74:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:qlen: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_drain: 94493
eq_tapq:replication_ns_1@10.1.3.74:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.74:rec_fetched: 163176
eq_tapq:replication_ns_1@10.1.3.74:reconnects: 682
eq_tapq:replication_ns_1@10.1.3.74:recv_ack_seqno: 229802
eq_tapq:replication_ns_1@10.1.3.74:reserved: 1
eq_tapq:replication_ns_1@10.1.3.74:seqno_ack_requested: 229802
eq_tapq:replication_ns_1@10.1.3.74:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.74:suspended: false
eq_tapq:replication_ns_1@10.1.3.74:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:total_noops: 156
eq_tapq:replication_ns_1@10.1.3.74:type: producer
eq_tapq:replication_ns_1@10.1.3.74:vb_filter: { [342,512], [854,1023] }
eq_tapq:replication_ns_1@10.1.3.74:vb_filters: 341
eq_tapq:replication_ns_1@10.1.3.78:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_seqno: 64524
eq_tapq:replication_ns_1@10.1.3.78:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.78:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.78:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.78:complete: false
eq_tapq:replication_ns_1@10.1.3.78:connected: true
eq_tapq:replication_ns_1@10.1.3.78:created: 100
eq_tapq:replication_ns_1@10.1.3.78:disconnects: 340
eq_tapq:replication_ns_1@10.1.3.78:empty: true
eq_tapq:replication_ns_1@10.1.3.78:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.78:has_item: false
eq_tapq:replication_ns_1@10.1.3.78:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.78:idle: true
eq_tapq:replication_ns_1@10.1.3.78:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.78:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.78:paused: 1
eq_tapq:replication_ns_1@10.1.3.78:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:qlen: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_drain: 60655
eq_tapq:replication_ns_1@10.1.3.78:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.78:rec_fetched: 63313
eq_tapq:replication_ns_1@10.1.3.78:reconnects: 340
eq_tapq:replication_ns_1@10.1.3.78:recv_ack_seqno: 64523
eq_tapq:replication_ns_1@10.1.3.78:reserved: 1
eq_tapq:replication_ns_1@10.1.3.78:seqno_ack_requested: 64523
eq_tapq:replication_ns_1@10.1.3.78:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.78:suspended: false
eq_tapq:replication_ns_1@10.1.3.78:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:total_noops: 165
eq_tapq:replication_ns_1@10.1.3.78:type: producer
eq_tapq:replication_ns_1@10.1.3.78:vb_filter: { [0,170], [513,682] }
eq_tapq:replication_ns_1@10.1.3.78:vb_filters: 341
10.1.3.80
Traceback (most recent call last):
File "/opt/couchbase/lib/python/cbstats", line 320, in ?
main()
File "/opt/couchbase/lib/python/cbstats", line 317, in main
c.execute()
File "/opt/couchbase/lib/python/clitool.py", line 43, in execute
f[0](mc, *args[2:], **opts.__dict__)
File "/opt/couchbase/lib/python/cbstats", line 27, in g
mc.sasl_auth_plain(username, password)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 191, in sasl_auth_plain
return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password]))
File "/opt/couchbase/lib/python/mc_bin_client.py", line 187, in sasl_auth_start
return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 104, in _doCmd
return self._handleSingleResponse(opaque)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 97, in _handleSingleResponse
cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 93, in _handleKeyedResponse
raise MemcachedError(errcode, rv)
mc_bin_client.MemcachedError: Memcached error #32: Auth failure
10.1.3.78
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 4
ep_tap_deletes: 1180
ep_tap_fg_fetched: 38134
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 39314
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 43391
ep_tap_total_queue: 0
eq_tapq:anon_2716:connected: true
eq_tapq:anon_2716:created: 1120
eq_tapq:anon_2716:num_checkpoint_end: 341
eq_tapq:anon_2716:num_checkpoint_end_failed: 0
eq_tapq:anon_2716:num_checkpoint_start: 342
eq_tapq:anon_2716:num_checkpoint_start_failed: 0
eq_tapq:anon_2716:num_delete: 2
eq_tapq:anon_2716:num_delete_failed: 0
eq_tapq:anon_2716:num_flush: 0
eq_tapq:anon_2716:num_flush_failed: 0
eq_tapq:anon_2716:num_mutation: 8
eq_tapq:anon_2716:num_mutation_failed: 0
eq_tapq:anon_2716:num_opaque: 2
eq_tapq:anon_2716:num_opaque_failed: 0
eq_tapq:anon_2716:num_unknown: 0
eq_tapq:anon_2716:num_vbucket_set: 0
eq_tapq:anon_2716:num_vbucket_set_failed: 0
eq_tapq:anon_2716:pending_disconnect: false
eq_tapq:anon_2716:reserved: 0
eq_tapq:anon_2716:supports_ack: true
eq_tapq:anon_2716:type: consumer
eq_tapq:anon_2727:connected: true
eq_tapq:anon_2727:created: 1124
eq_tapq:anon_2727:num_checkpoint_end: 342
eq_tapq:anon_2727:num_checkpoint_end_failed: 0
eq_tapq:anon_2727:num_checkpoint_start: 343
eq_tapq:anon_2727:num_checkpoint_start_failed: 0
eq_tapq:anon_2727:num_delete: 0
eq_tapq:anon_2727:num_delete_failed: 0
eq_tapq:anon_2727:num_flush: 0
eq_tapq:anon_2727:num_flush_failed: 0
eq_tapq:anon_2727:num_mutation: 11
eq_tapq:anon_2727:num_mutation_failed: 0
eq_tapq:anon_2727:num_opaque: 2
eq_tapq:anon_2727:num_opaque_failed: 0
eq_tapq:anon_2727:num_unknown: 0
eq_tapq:anon_2727:num_vbucket_set: 0
eq_tapq:anon_2727:num_vbucket_set_failed: 0
eq_tapq:anon_2727:pending_disconnect: false
eq_tapq:anon_2727:reserved: 0
eq_tapq:anon_2727:supports_ack: true
eq_tapq:anon_2727:type: consumer
eq_tapq:replication_ns_1@10.1.3.74:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_seqno: 37333
eq_tapq:replication_ns_1@10.1.3.74:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.74:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.74:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.74:complete: false
eq_tapq:replication_ns_1@10.1.3.74:connected: true
eq_tapq:replication_ns_1@10.1.3.74:created: 7
eq_tapq:replication_ns_1@10.1.3.74:disconnects: 340
eq_tapq:replication_ns_1@10.1.3.74:empty: true
eq_tapq:replication_ns_1@10.1.3.74:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.74:has_item: false
eq_tapq:replication_ns_1@10.1.3.74:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.74:idle: true
eq_tapq:replication_ns_1@10.1.3.74:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.74:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.74:paused: 1
eq_tapq:replication_ns_1@10.1.3.74:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:qlen: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_drain: 34333
eq_tapq:replication_ns_1@10.1.3.74:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.74:rec_fetched: 36402
eq_tapq:replication_ns_1@10.1.3.74:reconnects: 340
eq_tapq:replication_ns_1@10.1.3.74:recv_ack_seqno: 37332
eq_tapq:replication_ns_1@10.1.3.74:reserved: 1
eq_tapq:replication_ns_1@10.1.3.74:seqno_ack_requested: 37332
eq_tapq:replication_ns_1@10.1.3.74:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.74:suspended: false
eq_tapq:replication_ns_1@10.1.3.74:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:total_noops: 155
eq_tapq:replication_ns_1@10.1.3.74:type: producer
eq_tapq:replication_ns_1@10.1.3.74:vb_filter: { [513,853] }
eq_tapq:replication_ns_1@10.1.3.74:vb_filters: 341
eq_tapq:replication_ns_1@10.1.3.75:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_seqno: 7830
eq_tapq:replication_ns_1@10.1.3.75:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.75:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.75:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.75:complete: false
eq_tapq:replication_ns_1@10.1.3.75:connected: true
eq_tapq:replication_ns_1@10.1.3.75:created: 715
eq_tapq:replication_ns_1@10.1.3.75:disconnects: 340
eq_tapq:replication_ns_1@10.1.3.75:empty: true
eq_tapq:replication_ns_1@10.1.3.75:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.75:has_item: false
eq_tapq:replication_ns_1@10.1.3.75:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.75:idle: true
eq_tapq:replication_ns_1@10.1.3.75:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.75:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.75:paused: 1
eq_tapq:replication_ns_1@10.1.3.75:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:qlen: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_drain: 4981
eq_tapq:replication_ns_1@10.1.3.75:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.75:rec_fetched: 6989
eq_tapq:replication_ns_1@10.1.3.75:reconnects: 340
eq_tapq:replication_ns_1@10.1.3.75:recv_ack_seqno: 7829
eq_tapq:replication_ns_1@10.1.3.75:reserved: 1
eq_tapq:replication_ns_1@10.1.3.75:seqno_ack_requested: 7829
eq_tapq:replication_ns_1@10.1.3.75:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.75:suspended: false
eq_tapq:replication_ns_1@10.1.3.75:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:total_noops: 130
eq_tapq:replication_ns_1@10.1.3.75:type: producer
eq_tapq:replication_ns_1@10.1.3.75:vb_filter: { [171,341], [854,1023] }
eq_tapq:replication_ns_1@10.1.3.75:vb_filters: 341
10.1.3.74
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 4
ep_tap_deletes: 6531
ep_tap_fg_fetched: 436209
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 283655
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 459312
ep_tap_total_queue: 0
eq_tapq:anon_8195:connected: true
eq_tapq:anon_8195:created: 2560
eq_tapq:anon_8195:num_checkpoint_end: 220
eq_tapq:anon_8195:num_checkpoint_end_failed: 0
eq_tapq:anon_8195:num_checkpoint_start: 222
eq_tapq:anon_8195:num_checkpoint_start_failed: 0
eq_tapq:anon_8195:num_delete: 4
eq_tapq:anon_8195:num_delete_failed: 0
eq_tapq:anon_8195:num_flush: 0
eq_tapq:anon_8195:num_flush_failed: 0
eq_tapq:anon_8195:num_mutation: 16
eq_tapq:anon_8195:num_mutation_failed: 0
eq_tapq:anon_8195:num_opaque: 2
eq_tapq:anon_8195:num_opaque_failed: 0
eq_tapq:anon_8195:num_unknown: 0
eq_tapq:anon_8195:num_vbucket_set: 0
eq_tapq:anon_8195:num_vbucket_set_failed: 0
eq_tapq:anon_8195:pending_disconnect: false
eq_tapq:anon_8195:reserved: 0
eq_tapq:anon_8195:supports_ack: true
eq_tapq:anon_8195:type: consumer
eq_tapq:anon_8204:connected: true
eq_tapq:anon_8204:created: 2563
eq_tapq:anon_8204:num_checkpoint_end: 341
eq_tapq:anon_8204:num_checkpoint_end_failed: 0
eq_tapq:anon_8204:num_checkpoint_start: 349
eq_tapq:anon_8204:num_checkpoint_start_failed: 0
eq_tapq:anon_8204:num_delete: 104
eq_tapq:anon_8204:num_delete_failed: 0
eq_tapq:anon_8204:num_flush: 0
eq_tapq:anon_8204:num_flush_failed: 0
eq_tapq:anon_8204:num_mutation: 1287
eq_tapq:anon_8204:num_mutation_failed: 0
eq_tapq:anon_8204:num_opaque: 10
eq_tapq:anon_8204:num_opaque_failed: 0
eq_tapq:anon_8204:num_unknown: 0
eq_tapq:anon_8204:num_vbucket_set: 0
eq_tapq:anon_8204:num_vbucket_set_failed: 0
eq_tapq:anon_8204:pending_disconnect: false
eq_tapq:anon_8204:reserved: 0
eq_tapq:anon_8204:supports_ack: true
eq_tapq:anon_8204:type: consumer
eq_tapq:replication_ns_1@10.1.3.75:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_seqno: 238912
eq_tapq:replication_ns_1@10.1.3.75:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.75:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.75:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.75:complete: false
eq_tapq:replication_ns_1@10.1.3.75:connected: true
eq_tapq:replication_ns_1@10.1.3.75:created: 1403
eq_tapq:replication_ns_1@10.1.3.75:disconnects: 684
eq_tapq:replication_ns_1@10.1.3.75:empty: true
eq_tapq:replication_ns_1@10.1.3.75:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.75:has_item: false
eq_tapq:replication_ns_1@10.1.3.75:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.75:idle: true
eq_tapq:replication_ns_1@10.1.3.75:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.75:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.75:paused: 1
eq_tapq:replication_ns_1@10.1.3.75:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:qlen: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_drain: 225828
eq_tapq:replication_ns_1@10.1.3.75:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.75:rec_fetched: 233535
eq_tapq:replication_ns_1@10.1.3.75:reconnects: 684
eq_tapq:replication_ns_1@10.1.3.75:recv_ack_seqno: 238911
eq_tapq:replication_ns_1@10.1.3.75:reserved: 1
eq_tapq:replication_ns_1@10.1.3.75:seqno_ack_requested: 238911
eq_tapq:replication_ns_1@10.1.3.75:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.75:suspended: false
eq_tapq:replication_ns_1@10.1.3.75:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:total_noops: 155
eq_tapq:replication_ns_1@10.1.3.75:type: producer
eq_tapq:replication_ns_1@10.1.3.75:vb_filter: { [0,170], [683,853] }
eq_tapq:replication_ns_1@10.1.3.75:vb_filters: 342
eq_tapq:replication_ns_1@10.1.3.78:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_seqno: 61641
eq_tapq:replication_ns_1@10.1.3.78:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.78:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.78:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.78:complete: false
eq_tapq:replication_ns_1@10.1.3.78:connected: true
eq_tapq:replication_ns_1@10.1.3.78:created: 1447
eq_tapq:replication_ns_1@10.1.3.78:disconnects: 341
eq_tapq:replication_ns_1@10.1.3.78:empty: true
eq_tapq:replication_ns_1@10.1.3.78:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.78:has_item: false
eq_tapq:replication_ns_1@10.1.3.78:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.78:idle: true
eq_tapq:replication_ns_1@10.1.3.78:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.78:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.78:paused: 1
eq_tapq:replication_ns_1@10.1.3.78:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:qlen: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_drain: 57827
eq_tapq:replication_ns_1@10.1.3.78:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.78:rec_fetched: 60456
eq_tapq:replication_ns_1@10.1.3.78:reconnects: 341
eq_tapq:replication_ns_1@10.1.3.78:recv_ack_seqno: 61640
eq_tapq:replication_ns_1@10.1.3.78:reserved: 1
eq_tapq:replication_ns_1@10.1.3.78:seqno_ack_requested: 61640
eq_tapq:replication_ns_1@10.1.3.78:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.78:suspended: false
eq_tapq:replication_ns_1@10.1.3.78:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:total_noops: 159
eq_tapq:replication_ns_1@10.1.3.78:type: producer
eq_tapq:replication_ns_1@10.1.3.78:vb_filter: { [171,512] }
eq_tapq:replication_ns_1@10.1.3.78:vb_filters: 342
Show
Karan Kumar
added a comment - Tap stats from all the nodes.
10.1.3.75
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 4
ep_tap_deletes: 6027
ep_tap_fg_fetched: 423533
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 155148
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 507585
ep_tap_total_queue: 0
eq_tapq:anon_2068:connected: true
eq_tapq:anon_2068:created: 806
eq_tapq:anon_2068:num_checkpoint_end: 319
eq_tapq:anon_2068:num_checkpoint_end_failed: 0
eq_tapq:anon_2068:num_checkpoint_start: 325
eq_tapq:anon_2068:num_checkpoint_start_failed: 0
eq_tapq:anon_2068:num_delete: 13
eq_tapq:anon_2068:num_delete_failed: 0
eq_tapq:anon_2068:num_flush: 0
eq_tapq:anon_2068:num_flush_failed: 0
eq_tapq:anon_2068:num_mutation: 182
eq_tapq:anon_2068:num_mutation_failed: 0
eq_tapq:anon_2068:num_opaque: 12
eq_tapq:anon_2068:num_opaque_failed: 0
eq_tapq:anon_2068:num_unknown: 0
eq_tapq:anon_2068:num_vbucket_set: 0
eq_tapq:anon_2068:num_vbucket_set_failed: 0
eq_tapq:anon_2068:pending_disconnect: false
eq_tapq:anon_2068:reserved: 0
eq_tapq:anon_2068:supports_ack: true
eq_tapq:anon_2068:type: consumer
eq_tapq:anon_4104:connected: true
eq_tapq:anon_4104:created: 1215
eq_tapq:anon_4104:num_checkpoint_end: 341
eq_tapq:anon_4104:num_checkpoint_end_failed: 0
eq_tapq:anon_4104:num_checkpoint_start: 353
eq_tapq:anon_4104:num_checkpoint_start_failed: 0
eq_tapq:anon_4104:num_delete: 11
eq_tapq:anon_4104:num_delete_failed: 0
eq_tapq:anon_4104:num_flush: 0
eq_tapq:anon_4104:num_flush_failed: 0
eq_tapq:anon_4104:num_mutation: 110
eq_tapq:anon_4104:num_mutation_failed: 0
eq_tapq:anon_4104:num_opaque: 24
eq_tapq:anon_4104:num_opaque_failed: 0
eq_tapq:anon_4104:num_unknown: 0
eq_tapq:anon_4104:num_vbucket_set: 0
eq_tapq:anon_4104:num_vbucket_set_failed: 0
eq_tapq:anon_4104:pending_disconnect: false
eq_tapq:anon_4104:reserved: 0
eq_tapq:anon_4104:supports_ack: true
eq_tapq:anon_4104:type: consumer
eq_tapq: replication_ns_1@10.1.3.74 :ack_log_size: 0
eq_tapq: replication_ns_1@10.1.3.74 :ack_playback_size: 0
eq_tapq: replication_ns_1@10.1.3.74 :ack_seqno: 229803
eq_tapq: replication_ns_1@10.1.3.74 :ack_window_full: false
eq_tapq: replication_ns_1@10.1.3.74 :backfill_completed: true
eq_tapq: replication_ns_1@10.1.3.74 :bg_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_jobs_completed: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_jobs_issued: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_queued: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_result_size: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_results: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_wait_for_results: false
eq_tapq: replication_ns_1@10.1.3.74 :complete: false
eq_tapq: replication_ns_1@10.1.3.74 :connected: true
eq_tapq: replication_ns_1@10.1.3.74 :created: 52
eq_tapq: replication_ns_1@10.1.3.74 :disconnects: 682
eq_tapq: replication_ns_1@10.1.3.74 :empty: true
eq_tapq: replication_ns_1@10.1.3.74 :flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq: replication_ns_1@10.1.3.74 :has_item: false
eq_tapq: replication_ns_1@10.1.3.74 :has_queued_item: false
eq_tapq: replication_ns_1@10.1.3.74 :idle: true
eq_tapq: replication_ns_1@10.1.3.74 :num_tap_nack: 0
eq_tapq: replication_ns_1@10.1.3.74 :num_tap_tmpfail_survivors: 0
eq_tapq: replication_ns_1@10.1.3.74 :paused: 1
eq_tapq: replication_ns_1@10.1.3.74 :pending_backfill: false
eq_tapq: replication_ns_1@10.1.3.74 :pending_disconnect: false
eq_tapq: replication_ns_1@10.1.3.74 :pending_disk_backfill: false
eq_tapq: replication_ns_1@10.1.3.74 :qlen: 0
eq_tapq: replication_ns_1@10.1.3.74 :qlen_high_pri: 0
eq_tapq: replication_ns_1@10.1.3.74 :qlen_low_pri: 0
eq_tapq: replication_ns_1@10.1.3.74 :queue_backfillremaining: 0
eq_tapq: replication_ns_1@10.1.3.74 :queue_backoff: 0
eq_tapq: replication_ns_1@10.1.3.74 :queue_drain: 94493
eq_tapq: replication_ns_1@10.1.3.74 :queue_fill: 0
eq_tapq: replication_ns_1@10.1.3.74 :queue_itemondisk: 0
eq_tapq: replication_ns_1@10.1.3.74 :queue_memory: 0
eq_tapq: replication_ns_1@10.1.3.74 :rec_fetched: 163176
eq_tapq: replication_ns_1@10.1.3.74 :reconnects: 682
eq_tapq: replication_ns_1@10.1.3.74 :recv_ack_seqno: 229802
eq_tapq: replication_ns_1@10.1.3.74 :reserved: 1
eq_tapq: replication_ns_1@10.1.3.74 :seqno_ack_requested: 229802
eq_tapq: replication_ns_1@10.1.3.74 :supports_ack: true
eq_tapq: replication_ns_1@10.1.3.74 :suspended: false
eq_tapq: replication_ns_1@10.1.3.74 :total_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.74 :total_noops: 156
eq_tapq: replication_ns_1@10.1.3.74 :type: producer
eq_tapq: replication_ns_1@10.1.3.74 :vb_filter: { [342,512], [854,1023] }
eq_tapq: replication_ns_1@10.1.3.74 :vb_filters: 341
eq_tapq: replication_ns_1@10.1.3.78 :ack_log_size: 0
eq_tapq: replication_ns_1@10.1.3.78 :ack_playback_size: 0
eq_tapq: replication_ns_1@10.1.3.78 :ack_seqno: 64524
eq_tapq: replication_ns_1@10.1.3.78 :ack_window_full: false
eq_tapq: replication_ns_1@10.1.3.78 :backfill_completed: true
eq_tapq: replication_ns_1@10.1.3.78 :bg_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_jobs_completed: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_jobs_issued: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_queued: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_result_size: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_results: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_wait_for_results: false
eq_tapq: replication_ns_1@10.1.3.78 :complete: false
eq_tapq: replication_ns_1@10.1.3.78 :connected: true
eq_tapq: replication_ns_1@10.1.3.78 :created: 100
eq_tapq: replication_ns_1@10.1.3.78 :disconnects: 340
eq_tapq: replication_ns_1@10.1.3.78 :empty: true
eq_tapq: replication_ns_1@10.1.3.78 :flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq: replication_ns_1@10.1.3.78 :has_item: false
eq_tapq: replication_ns_1@10.1.3.78 :has_queued_item: false
eq_tapq: replication_ns_1@10.1.3.78 :idle: true
eq_tapq: replication_ns_1@10.1.3.78 :num_tap_nack: 0
eq_tapq: replication_ns_1@10.1.3.78 :num_tap_tmpfail_survivors: 0
eq_tapq: replication_ns_1@10.1.3.78 :paused: 1
eq_tapq: replication_ns_1@10.1.3.78 :pending_backfill: false
eq_tapq: replication_ns_1@10.1.3.78 :pending_disconnect: false
eq_tapq: replication_ns_1@10.1.3.78 :pending_disk_backfill: false
eq_tapq: replication_ns_1@10.1.3.78 :qlen: 0
eq_tapq: replication_ns_1@10.1.3.78 :qlen_high_pri: 0
eq_tapq: replication_ns_1@10.1.3.78 :qlen_low_pri: 0
eq_tapq: replication_ns_1@10.1.3.78 :queue_backfillremaining: 0
eq_tapq: replication_ns_1@10.1.3.78 :queue_backoff: 0
eq_tapq: replication_ns_1@10.1.3.78 :queue_drain: 60655
eq_tapq: replication_ns_1@10.1.3.78 :queue_fill: 0
eq_tapq: replication_ns_1@10.1.3.78 :queue_itemondisk: 0
eq_tapq: replication_ns_1@10.1.3.78 :queue_memory: 0
eq_tapq: replication_ns_1@10.1.3.78 :rec_fetched: 63313
eq_tapq: replication_ns_1@10.1.3.78 :reconnects: 340
eq_tapq: replication_ns_1@10.1.3.78 :recv_ack_seqno: 64523
eq_tapq: replication_ns_1@10.1.3.78 :reserved: 1
eq_tapq: replication_ns_1@10.1.3.78 :seqno_ack_requested: 64523
eq_tapq: replication_ns_1@10.1.3.78 :supports_ack: true
eq_tapq: replication_ns_1@10.1.3.78 :suspended: false
eq_tapq: replication_ns_1@10.1.3.78 :total_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.78 :total_noops: 165
eq_tapq: replication_ns_1@10.1.3.78 :type: producer
eq_tapq: replication_ns_1@10.1.3.78 :vb_filter: { [0,170], [513,682] }
eq_tapq: replication_ns_1@10.1.3.78 :vb_filters: 341
10.1.3.80
Traceback (most recent call last):
File "/opt/couchbase/lib/python/cbstats", line 320, in ?
main()
File "/opt/couchbase/lib/python/cbstats", line 317, in main
c.execute()
File "/opt/couchbase/lib/python/clitool.py", line 43, in execute
f[0](mc, *args[2:], **opts.__dict__)
File "/opt/couchbase/lib/python/cbstats", line 27, in g
mc.sasl_auth_plain(username, password)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 191, in sasl_auth_plain
return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password]))
File "/opt/couchbase/lib/python/mc_bin_client.py", line 187, in sasl_auth_start
return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 104, in _doCmd
return self._handleSingleResponse(opaque)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 97, in _handleSingleResponse
cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque)
File "/opt/couchbase/lib/python/mc_bin_client.py", line 93, in _handleKeyedResponse
raise MemcachedError(errcode, rv)
mc_bin_client.MemcachedError: Memcached error #32: Auth failure
10.1.3.78
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 4
ep_tap_deletes: 1180
ep_tap_fg_fetched: 38134
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 39314
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 43391
ep_tap_total_queue: 0
eq_tapq:anon_2716:connected: true
eq_tapq:anon_2716:created: 1120
eq_tapq:anon_2716:num_checkpoint_end: 341
eq_tapq:anon_2716:num_checkpoint_end_failed: 0
eq_tapq:anon_2716:num_checkpoint_start: 342
eq_tapq:anon_2716:num_checkpoint_start_failed: 0
eq_tapq:anon_2716:num_delete: 2
eq_tapq:anon_2716:num_delete_failed: 0
eq_tapq:anon_2716:num_flush: 0
eq_tapq:anon_2716:num_flush_failed: 0
eq_tapq:anon_2716:num_mutation: 8
eq_tapq:anon_2716:num_mutation_failed: 0
eq_tapq:anon_2716:num_opaque: 2
eq_tapq:anon_2716:num_opaque_failed: 0
eq_tapq:anon_2716:num_unknown: 0
eq_tapq:anon_2716:num_vbucket_set: 0
eq_tapq:anon_2716:num_vbucket_set_failed: 0
eq_tapq:anon_2716:pending_disconnect: false
eq_tapq:anon_2716:reserved: 0
eq_tapq:anon_2716:supports_ack: true
eq_tapq:anon_2716:type: consumer
eq_tapq:anon_2727:connected: true
eq_tapq:anon_2727:created: 1124
eq_tapq:anon_2727:num_checkpoint_end: 342
eq_tapq:anon_2727:num_checkpoint_end_failed: 0
eq_tapq:anon_2727:num_checkpoint_start: 343
eq_tapq:anon_2727:num_checkpoint_start_failed: 0
eq_tapq:anon_2727:num_delete: 0
eq_tapq:anon_2727:num_delete_failed: 0
eq_tapq:anon_2727:num_flush: 0
eq_tapq:anon_2727:num_flush_failed: 0
eq_tapq:anon_2727:num_mutation: 11
eq_tapq:anon_2727:num_mutation_failed: 0
eq_tapq:anon_2727:num_opaque: 2
eq_tapq:anon_2727:num_opaque_failed: 0
eq_tapq:anon_2727:num_unknown: 0
eq_tapq:anon_2727:num_vbucket_set: 0
eq_tapq:anon_2727:num_vbucket_set_failed: 0
eq_tapq:anon_2727:pending_disconnect: false
eq_tapq:anon_2727:reserved: 0
eq_tapq:anon_2727:supports_ack: true
eq_tapq:anon_2727:type: consumer
eq_tapq: replication_ns_1@10.1.3.74 :ack_log_size: 0
eq_tapq: replication_ns_1@10.1.3.74 :ack_playback_size: 0
eq_tapq: replication_ns_1@10.1.3.74 :ack_seqno: 37333
eq_tapq: replication_ns_1@10.1.3.74 :ack_window_full: false
eq_tapq: replication_ns_1@10.1.3.74 :backfill_completed: true
eq_tapq: replication_ns_1@10.1.3.74 :bg_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_jobs_completed: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_jobs_issued: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_queued: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_result_size: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_results: 0
eq_tapq: replication_ns_1@10.1.3.74 :bg_wait_for_results: false
eq_tapq: replication_ns_1@10.1.3.74 :complete: false
eq_tapq: replication_ns_1@10.1.3.74 :connected: true
eq_tapq: replication_ns_1@10.1.3.74 :created: 7
eq_tapq: replication_ns_1@10.1.3.74 :disconnects: 340
eq_tapq: replication_ns_1@10.1.3.74 :empty: true
eq_tapq: replication_ns_1@10.1.3.74 :flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq: replication_ns_1@10.1.3.74 :has_item: false
eq_tapq: replication_ns_1@10.1.3.74 :has_queued_item: false
eq_tapq: replication_ns_1@10.1.3.74 :idle: true
eq_tapq: replication_ns_1@10.1.3.74 :num_tap_nack: 0
eq_tapq: replication_ns_1@10.1.3.74 :num_tap_tmpfail_survivors: 0
eq_tapq: replication_ns_1@10.1.3.74 :paused: 1
eq_tapq: replication_ns_1@10.1.3.74 :pending_backfill: false
eq_tapq: replication_ns_1@10.1.3.74 :pending_disconnect: false
eq_tapq: replication_ns_1@10.1.3.74 :pending_disk_backfill: false
eq_tapq: replication_ns_1@10.1.3.74 :qlen: 0
eq_tapq: replication_ns_1@10.1.3.74 :qlen_high_pri: 0
eq_tapq: replication_ns_1@10.1.3.74 :qlen_low_pri: 0
eq_tapq: replication_ns_1@10.1.3.74 :queue_backfillremaining: 0
eq_tapq: replication_ns_1@10.1.3.74 :queue_backoff: 0
eq_tapq: replication_ns_1@10.1.3.74 :queue_drain: 34333
eq_tapq: replication_ns_1@10.1.3.74 :queue_fill: 0
eq_tapq: replication_ns_1@10.1.3.74 :queue_itemondisk: 0
eq_tapq: replication_ns_1@10.1.3.74 :queue_memory: 0
eq_tapq: replication_ns_1@10.1.3.74 :rec_fetched: 36402
eq_tapq: replication_ns_1@10.1.3.74 :reconnects: 340
eq_tapq: replication_ns_1@10.1.3.74 :recv_ack_seqno: 37332
eq_tapq: replication_ns_1@10.1.3.74 :reserved: 1
eq_tapq: replication_ns_1@10.1.3.74 :seqno_ack_requested: 37332
eq_tapq: replication_ns_1@10.1.3.74 :supports_ack: true
eq_tapq: replication_ns_1@10.1.3.74 :suspended: false
eq_tapq: replication_ns_1@10.1.3.74 :total_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.74 :total_noops: 155
eq_tapq: replication_ns_1@10.1.3.74 :type: producer
eq_tapq: replication_ns_1@10.1.3.74 :vb_filter: { [513,853] }
eq_tapq: replication_ns_1@10.1.3.74 :vb_filters: 341
eq_tapq: replication_ns_1@10.1.3.75 :ack_log_size: 0
eq_tapq: replication_ns_1@10.1.3.75 :ack_playback_size: 0
eq_tapq: replication_ns_1@10.1.3.75 :ack_seqno: 7830
eq_tapq: replication_ns_1@10.1.3.75 :ack_window_full: false
eq_tapq: replication_ns_1@10.1.3.75 :backfill_completed: true
eq_tapq: replication_ns_1@10.1.3.75 :bg_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_jobs_completed: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_jobs_issued: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_queued: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_result_size: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_results: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_wait_for_results: false
eq_tapq: replication_ns_1@10.1.3.75 :complete: false
eq_tapq: replication_ns_1@10.1.3.75 :connected: true
eq_tapq: replication_ns_1@10.1.3.75 :created: 715
eq_tapq: replication_ns_1@10.1.3.75 :disconnects: 340
eq_tapq: replication_ns_1@10.1.3.75 :empty: true
eq_tapq: replication_ns_1@10.1.3.75 :flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq: replication_ns_1@10.1.3.75 :has_item: false
eq_tapq: replication_ns_1@10.1.3.75 :has_queued_item: false
eq_tapq: replication_ns_1@10.1.3.75 :idle: true
eq_tapq: replication_ns_1@10.1.3.75 :num_tap_nack: 0
eq_tapq: replication_ns_1@10.1.3.75 :num_tap_tmpfail_survivors: 0
eq_tapq: replication_ns_1@10.1.3.75 :paused: 1
eq_tapq: replication_ns_1@10.1.3.75 :pending_backfill: false
eq_tapq: replication_ns_1@10.1.3.75 :pending_disconnect: false
eq_tapq: replication_ns_1@10.1.3.75 :pending_disk_backfill: false
eq_tapq: replication_ns_1@10.1.3.75 :qlen: 0
eq_tapq: replication_ns_1@10.1.3.75 :qlen_high_pri: 0
eq_tapq: replication_ns_1@10.1.3.75 :qlen_low_pri: 0
eq_tapq: replication_ns_1@10.1.3.75 :queue_backfillremaining: 0
eq_tapq: replication_ns_1@10.1.3.75 :queue_backoff: 0
eq_tapq: replication_ns_1@10.1.3.75 :queue_drain: 4981
eq_tapq: replication_ns_1@10.1.3.75 :queue_fill: 0
eq_tapq: replication_ns_1@10.1.3.75 :queue_itemondisk: 0
eq_tapq: replication_ns_1@10.1.3.75 :queue_memory: 0
eq_tapq: replication_ns_1@10.1.3.75 :rec_fetched: 6989
eq_tapq: replication_ns_1@10.1.3.75 :reconnects: 340
eq_tapq: replication_ns_1@10.1.3.75 :recv_ack_seqno: 7829
eq_tapq: replication_ns_1@10.1.3.75 :reserved: 1
eq_tapq: replication_ns_1@10.1.3.75 :seqno_ack_requested: 7829
eq_tapq: replication_ns_1@10.1.3.75 :supports_ack: true
eq_tapq: replication_ns_1@10.1.3.75 :suspended: false
eq_tapq: replication_ns_1@10.1.3.75 :total_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.75 :total_noops: 130
eq_tapq: replication_ns_1@10.1.3.75 :type: producer
eq_tapq: replication_ns_1@10.1.3.75 :vb_filter: { [171,341], [854,1023] }
eq_tapq: replication_ns_1@10.1.3.75 :vb_filters: 341
10.1.3.74
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 4
ep_tap_deletes: 6531
ep_tap_fg_fetched: 436209
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 283655
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 459312
ep_tap_total_queue: 0
eq_tapq:anon_8195:connected: true
eq_tapq:anon_8195:created: 2560
eq_tapq:anon_8195:num_checkpoint_end: 220
eq_tapq:anon_8195:num_checkpoint_end_failed: 0
eq_tapq:anon_8195:num_checkpoint_start: 222
eq_tapq:anon_8195:num_checkpoint_start_failed: 0
eq_tapq:anon_8195:num_delete: 4
eq_tapq:anon_8195:num_delete_failed: 0
eq_tapq:anon_8195:num_flush: 0
eq_tapq:anon_8195:num_flush_failed: 0
eq_tapq:anon_8195:num_mutation: 16
eq_tapq:anon_8195:num_mutation_failed: 0
eq_tapq:anon_8195:num_opaque: 2
eq_tapq:anon_8195:num_opaque_failed: 0
eq_tapq:anon_8195:num_unknown: 0
eq_tapq:anon_8195:num_vbucket_set: 0
eq_tapq:anon_8195:num_vbucket_set_failed: 0
eq_tapq:anon_8195:pending_disconnect: false
eq_tapq:anon_8195:reserved: 0
eq_tapq:anon_8195:supports_ack: true
eq_tapq:anon_8195:type: consumer
eq_tapq:anon_8204:connected: true
eq_tapq:anon_8204:created: 2563
eq_tapq:anon_8204:num_checkpoint_end: 341
eq_tapq:anon_8204:num_checkpoint_end_failed: 0
eq_tapq:anon_8204:num_checkpoint_start: 349
eq_tapq:anon_8204:num_checkpoint_start_failed: 0
eq_tapq:anon_8204:num_delete: 104
eq_tapq:anon_8204:num_delete_failed: 0
eq_tapq:anon_8204:num_flush: 0
eq_tapq:anon_8204:num_flush_failed: 0
eq_tapq:anon_8204:num_mutation: 1287
eq_tapq:anon_8204:num_mutation_failed: 0
eq_tapq:anon_8204:num_opaque: 10
eq_tapq:anon_8204:num_opaque_failed: 0
eq_tapq:anon_8204:num_unknown: 0
eq_tapq:anon_8204:num_vbucket_set: 0
eq_tapq:anon_8204:num_vbucket_set_failed: 0
eq_tapq:anon_8204:pending_disconnect: false
eq_tapq:anon_8204:reserved: 0
eq_tapq:anon_8204:supports_ack: true
eq_tapq:anon_8204:type: consumer
eq_tapq: replication_ns_1@10.1.3.75 :ack_log_size: 0
eq_tapq: replication_ns_1@10.1.3.75 :ack_playback_size: 0
eq_tapq: replication_ns_1@10.1.3.75 :ack_seqno: 238912
eq_tapq: replication_ns_1@10.1.3.75 :ack_window_full: false
eq_tapq: replication_ns_1@10.1.3.75 :backfill_completed: true
eq_tapq: replication_ns_1@10.1.3.75 :bg_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_jobs_completed: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_jobs_issued: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_queued: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_result_size: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_results: 0
eq_tapq: replication_ns_1@10.1.3.75 :bg_wait_for_results: false
eq_tapq: replication_ns_1@10.1.3.75 :complete: false
eq_tapq: replication_ns_1@10.1.3.75 :connected: true
eq_tapq: replication_ns_1@10.1.3.75 :created: 1403
eq_tapq: replication_ns_1@10.1.3.75 :disconnects: 684
eq_tapq: replication_ns_1@10.1.3.75 :empty: true
eq_tapq: replication_ns_1@10.1.3.75 :flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq: replication_ns_1@10.1.3.75 :has_item: false
eq_tapq: replication_ns_1@10.1.3.75 :has_queued_item: false
eq_tapq: replication_ns_1@10.1.3.75 :idle: true
eq_tapq: replication_ns_1@10.1.3.75 :num_tap_nack: 0
eq_tapq: replication_ns_1@10.1.3.75 :num_tap_tmpfail_survivors: 0
eq_tapq: replication_ns_1@10.1.3.75 :paused: 1
eq_tapq: replication_ns_1@10.1.3.75 :pending_backfill: false
eq_tapq: replication_ns_1@10.1.3.75 :pending_disconnect: false
eq_tapq: replication_ns_1@10.1.3.75 :pending_disk_backfill: false
eq_tapq: replication_ns_1@10.1.3.75 :qlen: 0
eq_tapq: replication_ns_1@10.1.3.75 :qlen_high_pri: 0
eq_tapq: replication_ns_1@10.1.3.75 :qlen_low_pri: 0
eq_tapq: replication_ns_1@10.1.3.75 :queue_backfillremaining: 0
eq_tapq: replication_ns_1@10.1.3.75 :queue_backoff: 0
eq_tapq: replication_ns_1@10.1.3.75 :queue_drain: 225828
eq_tapq: replication_ns_1@10.1.3.75 :queue_fill: 0
eq_tapq: replication_ns_1@10.1.3.75 :queue_itemondisk: 0
eq_tapq: replication_ns_1@10.1.3.75 :queue_memory: 0
eq_tapq: replication_ns_1@10.1.3.75 :rec_fetched: 233535
eq_tapq: replication_ns_1@10.1.3.75 :reconnects: 684
eq_tapq: replication_ns_1@10.1.3.75 :recv_ack_seqno: 238911
eq_tapq: replication_ns_1@10.1.3.75 :reserved: 1
eq_tapq: replication_ns_1@10.1.3.75 :seqno_ack_requested: 238911
eq_tapq: replication_ns_1@10.1.3.75 :supports_ack: true
eq_tapq: replication_ns_1@10.1.3.75 :suspended: false
eq_tapq: replication_ns_1@10.1.3.75 :total_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.75 :total_noops: 155
eq_tapq: replication_ns_1@10.1.3.75 :type: producer
eq_tapq: replication_ns_1@10.1.3.75 :vb_filter: { [0,170], [683,853] }
eq_tapq: replication_ns_1@10.1.3.75 :vb_filters: 342
eq_tapq: replication_ns_1@10.1.3.78 :ack_log_size: 0
eq_tapq: replication_ns_1@10.1.3.78 :ack_playback_size: 0
eq_tapq: replication_ns_1@10.1.3.78 :ack_seqno: 61641
eq_tapq: replication_ns_1@10.1.3.78 :ack_window_full: false
eq_tapq: replication_ns_1@10.1.3.78 :backfill_completed: true
eq_tapq: replication_ns_1@10.1.3.78 :bg_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_jobs_completed: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_jobs_issued: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_queued: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_result_size: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_results: 0
eq_tapq: replication_ns_1@10.1.3.78 :bg_wait_for_results: false
eq_tapq: replication_ns_1@10.1.3.78 :complete: false
eq_tapq: replication_ns_1@10.1.3.78 :connected: true
eq_tapq: replication_ns_1@10.1.3.78 :created: 1447
eq_tapq: replication_ns_1@10.1.3.78 :disconnects: 341
eq_tapq: replication_ns_1@10.1.3.78 :empty: true
eq_tapq: replication_ns_1@10.1.3.78 :flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq: replication_ns_1@10.1.3.78 :has_item: false
eq_tapq: replication_ns_1@10.1.3.78 :has_queued_item: false
eq_tapq: replication_ns_1@10.1.3.78 :idle: true
eq_tapq: replication_ns_1@10.1.3.78 :num_tap_nack: 0
eq_tapq: replication_ns_1@10.1.3.78 :num_tap_tmpfail_survivors: 0
eq_tapq: replication_ns_1@10.1.3.78 :paused: 1
eq_tapq: replication_ns_1@10.1.3.78 :pending_backfill: false
eq_tapq: replication_ns_1@10.1.3.78 :pending_disconnect: false
eq_tapq: replication_ns_1@10.1.3.78 :pending_disk_backfill: false
eq_tapq: replication_ns_1@10.1.3.78 :qlen: 0
eq_tapq: replication_ns_1@10.1.3.78 :qlen_high_pri: 0
eq_tapq: replication_ns_1@10.1.3.78 :qlen_low_pri: 0
eq_tapq: replication_ns_1@10.1.3.78 :queue_backfillremaining: 0
eq_tapq: replication_ns_1@10.1.3.78 :queue_backoff: 0
eq_tapq: replication_ns_1@10.1.3.78 :queue_drain: 57827
eq_tapq: replication_ns_1@10.1.3.78 :queue_fill: 0
eq_tapq: replication_ns_1@10.1.3.78 :queue_itemondisk: 0
eq_tapq: replication_ns_1@10.1.3.78 :queue_memory: 0
eq_tapq: replication_ns_1@10.1.3.78 :rec_fetched: 60456
eq_tapq: replication_ns_1@10.1.3.78 :reconnects: 341
eq_tapq: replication_ns_1@10.1.3.78 :recv_ack_seqno: 61640
eq_tapq: replication_ns_1@10.1.3.78 :reserved: 1
eq_tapq: replication_ns_1@10.1.3.78 :seqno_ack_requested: 61640
eq_tapq: replication_ns_1@10.1.3.78 :supports_ack: true
eq_tapq: replication_ns_1@10.1.3.78 :suspended: false
eq_tapq: replication_ns_1@10.1.3.78 :total_backlog_size: 0
eq_tapq: replication_ns_1@10.1.3.78 :total_noops: 159
eq_tapq: replication_ns_1@10.1.3.78 :type: producer
eq_tapq: replication_ns_1@10.1.3.78 :vb_filter: { [171,512] }
eq_tapq: replication_ns_1@10.1.3.78 :vb_filters: 342
Hide
Farshid Ghods
added a comment -
Karan,
can you please update the ticket with your findings about the data loss tests you ran ?
can you please update the ticket with your findings about the data loss tests you ran ?
Show
Farshid Ghods
added a comment - Karan,
can you please update the ticket with your findings about the data loss tests you ran ?
Hide
Karan Kumar
added a comment -
Show
Karan Kumar
added a comment - Still running the tests.
Hitting MB-5457 . Will try with older build.
Hide
Karan Kumar
added a comment -
Observed that there is no data loss:
1) When load it not under during rebalance
2) Parallel load (with no expiry and deletes) during rebalance
Observed data loss observed not always:
1) Parallel load (with expiry and deletes) during rebalance.
2 keys lost with 3.5 M items.
Running more tests for case when we have just deletes and just expiries.
1) When load it not under during rebalance
2) Parallel load (with no expiry and deletes) during rebalance
Observed data loss observed not always:
1) Parallel load (with expiry and deletes) during rebalance.
2 keys lost with 3.5 M items.
Running more tests for case when we have just deletes and just expiries.
Show
Karan Kumar
added a comment - Observed that there is no data loss:
1) When load it not under during rebalance
2) Parallel load (with no expiry and deletes) during rebalance
Observed data loss observed not always:
1) Parallel load (with expiry and deletes) during rebalance.
2 keys lost with 3.5 M items.
Running more tests for case when we have just deletes and just expiries.
Hide
Chiyoung Seo
added a comment -
>Test case failing
>
>swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=2,num-swap=3,swap-orchestrator=True
>
>1) Failover 3 nodes, including the orchestrator
>2) rebalance them out
>3) Add back the 3 node
>4) Rebalance in
>5) Results in data loss
I just noticed that there was a failover on three nodes. The failover and rebalance out mostly causes data loss.
>
>swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=2,num-swap=3,swap-orchestrator=True
>
>1) Failover 3 nodes, including the orchestrator
>2) rebalance them out
>3) Add back the 3 node
>4) Rebalance in
>5) Results in data loss
I just noticed that there was a failover on three nodes. The failover and rebalance out mostly causes data loss.
Show
Chiyoung Seo
added a comment - >Test case failing
>
>swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=2,num-swap=3,swap-orchestrator=True
>
>1) Failover 3 nodes, including the orchestrator
>2) rebalance them out
>3) Add back the 3 node
>4) Rebalance in
>5) Results in data loss
I just noticed that there was a failover on three nodes. The failover and rebalance out mostly causes data loss.
Hide
Karan Kumar
added a comment -
Chiyoung:
That testcase you mentioned is for the original ticket.
There is a comment with the testcase that causes failure. Which is simply swap rebalance without anything fancy
added a comment - 31/May/12 2:37 PM
Build: 181-851-rel
Test case failing:-
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1
It does expiry/deletes. Swap one node at the same time.
For bucket-1
Total active items 96021
Total expected replica items 192042
But we have 192041 (1 items missing) . Maybe with more items loaded we would see more loss.
That testcase you mentioned is for the original ticket.
There is a comment with the testcase that causes failure. Which is simply swap rebalance without anything fancy
added a comment - 31/May/12 2:37 PM
Build: 181-851-rel
Test case failing:-
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1
It does expiry/deletes. Swap one node at the same time.
For bucket-1
Total active items 96021
Total expected replica items 192042
But we have 192041 (1 items missing) . Maybe with more items loaded we would see more loss.
Show
Karan Kumar
added a comment - Chiyoung:
That testcase you mentioned is for the original ticket.
There is a comment with the testcase that causes failure. Which is simply swap rebalance without anything fancy
added a comment - 31/May/12 2:37 PM
Build: 181-851-rel
Test case failing:-
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1
It does expiry/deletes. Swap one node at the same time.
For bucket-1
Total active items 96021
Total expected replica items 192042
But we have 192041 (1 items missing) . Maybe with more items loaded we would see more loss.
Hide
Chiyoung Seo
added a comment -
I ran the test "swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=2,num-swap=3,swap-orchestrator=True" several times on my 7VM cluster with 1024 vbuckets, but didn't see any loss yet.
Show
Chiyoung Seo
added a comment - I ran the test "swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=2,num-swap=3,swap-orchestrator=True" several times on my 7VM cluster with 1024 vbuckets, but didn't see any loss yet.
Show
Karan Kumar
added a comment - I can give you a cluster with the issue, if that helps?
Hide
Farshid Ghods
added a comment -
Chiyoung,
as Karan mentioned the test case that is failing now is swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1 which does not have any failover operation.
the confusion with the test case was because jira does not show when the original ticket was re-opened and lead to confusion on what tests was failing.
Karan,
how many items does this test insert by default?
as Karan mentioned the test case that is failing now is swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1 which does not have any failover operation.
the confusion with the test case was because jira does not show when the original ticket was re-opened and lead to confusion on what tests was failing.
Karan,
how many items does this test insert by default?
Show
Farshid Ghods
added a comment - Chiyoung,
as Karan mentioned the test case that is failing now is swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1 which does not have any failover operation.
the confusion with the test case was because jira does not show when the original ticket was re-opened and lead to confusion on what tests was failing.
Karan,
how many items does this test insert by default?
Hide
Karan Kumar
added a comment -
Use the following tests, 2M items
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1,keys-count=2000000
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1,keys-count=2000000
Show
Karan Kumar
added a comment - Use the following tests, 2M items
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1,keys-count=2000000
Hide
Farshid Ghods
added a comment -
do we have results from the test that runs the load for 30 mins before rebalancing a new node ?
if the item mismatch is less than 0.008 percent we can defer this to 2.0 dp5
if the item mismatch is less than 0.008 percent we can defer this to 2.0 dp5
Show
Farshid Ghods
added a comment - do we have results from the test that runs the load for 30 mins before rebalancing a new node ?
if the item mismatch is less than 0.008 percent we can defer this to 2.0 dp5
Hide
Karan Kumar
added a comment -
Yes. There is no data loss in that case.
Seen data loss (upto 2 keys/3.5 M at the most) when we have load running with rebalance.
Seen data loss (upto 2 keys/3.5 M at the most) when we have load running with rebalance.
Show
Karan Kumar
added a comment - Yes. There is no data loss in that case.
Seen data loss (upto 2 keys/3.5 M at the most) when we have load running with rebalance.
Show
Karan Kumar
added a comment - I think we should defer this to dp5?
Hide
Chiyoung Seo
added a comment -
Just a quick update, I ran the following test case lots of times (more than 20 times), which caused data loss in the QE regressions:
python testrunner -i my_vm.ini -t swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1,keys-count=2000000 -p skip-cleanup=True
But, I didn't see any data loss in my tests. Please let me know if you have any other test cases that I can try.
python testrunner -i my_vm.ini -t swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1,keys-count=2000000 -p skip-cleanup=True
But, I didn't see any data loss in my tests. Please let me know if you have any other test cases that I can try.
Show
Chiyoung Seo
added a comment - Just a quick update, I ran the following test case lots of times (more than 20 times), which caused data loss in the QE regressions:
python testrunner -i my_vm.ini -t swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=2,num-swap=1,keys-count=2000000 -p skip-cleanup=True
But, I didn't see any data loss in my tests. Please let me know if you have any other test cases that I can try.
Show
Peter Wansch
added a comment - Can you keep track if we see this on 2.0 regressions?
Hide
Karan Kumar
added a comment -
Yes. We are seeing intermittent failures on 2.0 as well.
Failing test against 7 node cluster.
/testrunner -i /tmp/swaprebalance.ini -t swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=2,num-swap=3,swap-orchestrator=True
http://qa.hq.northscale.net/job/centos-64-2.0-swap-rebalance/lastCompletedBuild/consoleFull
Let us know what is needed to further get to this issue. Logs/cluster.
Replica had 85 more items than active. This test does deletes/expiry in parallel with rebalance.
Failing test against 7 node cluster.
/testrunner -i /tmp/swaprebalance.ini -t swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=2,num-swap=3,swap-orchestrator=True
http://qa.hq.northscale.net/job/centos-64-2.0-swap-rebalance/lastCompletedBuild/consoleFull
Let us know what is needed to further get to this issue. Logs/cluster.
Replica had 85 more items than active. This test does deletes/expiry in parallel with rebalance.
Show
Karan Kumar
added a comment - Yes. We are seeing intermittent failures on 2.0 as well.
Failing test against 7 node cluster.
/testrunner -i /tmp/swaprebalance.ini -t swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=2,num-swap=3,swap-orchestrator=True
http://qa.hq.northscale.net/job/centos-64-2.0-swap-rebalance/lastCompletedBuild/consoleFull
Let us know what is needed to further get to this issue. Logs/cluster.
Replica had 85 more items than active. This test does deletes/expiry in parallel with rebalance.
Show
Karan Kumar
added a comment - This is build 2.0.0-1464-rel
Show
Farshid Ghods
added a comment - closing this as won't fix.
QE team will open a new issue
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 12
ep_tap_deletes: 319
ep_tap_fg_fetched: 61141
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 61460
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 1
ep_tap_total_fetched: 67170
ep_tap_total_queue: 0
eq_tapq:anon_101:connected: true
eq_tapq:anon_101:created: 55
eq_tapq:anon_101:num_checkpoint_end: 4
eq_tapq:anon_101:num_checkpoint_end_failed: 0
eq_tapq:anon_101:num_checkpoint_start: 6
eq_tapq:anon_101:num_checkpoint_start_failed: 0
eq_tapq:anon_101:num_delete: 1377
eq_tapq:anon_101:num_delete_failed: 0
eq_tapq:anon_101:num_flush: 0
eq_tapq:anon_101:num_flush_failed: 0
eq_tapq:anon_101:num_mutation: 6532
eq_tapq:anon_101:num_mutation_failed: 0
eq_tapq:anon_101:num_opaque: 8
eq_tapq:anon_101:num_opaque_failed: 0
eq_tapq:anon_101:num_unknown: 0
eq_tapq:anon_101:num_vbucket_set: 0
eq_tapq:anon_101:num_vbucket_set_failed: 0
eq_tapq:anon_101:pending_disconnect: false
eq_tapq:anon_101:reserved: 0
eq_tapq:anon_101:supports_ack: true
eq_tapq:anon_101:type: consumer
eq_tapq:anon_103:connected: true
eq_tapq:anon_103:created: 56
eq_tapq:anon_103:num_checkpoint_end: 0
eq_tapq:anon_103:num_checkpoint_end_failed: 0
eq_tapq:anon_103:num_checkpoint_start: 0
eq_tapq:anon_103:num_checkpoint_start_failed: 0
eq_tapq:anon_103:num_delete: 0
eq_tapq:anon_103:num_delete_failed: 0
eq_tapq:anon_103:num_flush: 0
eq_tapq:anon_103:num_flush_failed: 0
eq_tapq:anon_103:num_mutation: 0
eq_tapq:anon_103:num_mutation_failed: 0
eq_tapq:anon_103:num_opaque: 12
eq_tapq:anon_103:num_opaque_failed: 0
eq_tapq:anon_103:num_unknown: 0
eq_tapq:anon_103:num_vbucket_set: 0
eq_tapq:anon_103:num_vbucket_set_failed: 0
eq_tapq:anon_103:pending_disconnect: false
eq_tapq:anon_103:reserved: 0
eq_tapq:anon_103:supports_ack: true
eq_tapq:anon_103:type: consumer
eq_tapq:anon_109:connected: true
eq_tapq:anon_109:created: 63
eq_tapq:anon_109:num_checkpoint_end: 3
eq_tapq:anon_109:num_checkpoint_end_failed: 0
eq_tapq:anon_109:num_checkpoint_start: 4
eq_tapq:anon_109:num_checkpoint_start_failed: 0
eq_tapq:anon_109:num_delete: 29
eq_tapq:anon_109:num_delete_failed: 0
eq_tapq:anon_109:num_flush: 0
eq_tapq:anon_109:num_flush_failed: 0
eq_tapq:anon_109:num_mutation: 4605
eq_tapq:anon_109:num_mutation_failed: 0
eq_tapq:anon_109:num_opaque: 4
eq_tapq:anon_109:num_opaque_failed: 0
eq_tapq:anon_109:num_unknown: 0
eq_tapq:anon_109:num_vbucket_set: 0
eq_tapq:anon_109:num_vbucket_set_failed: 0
eq_tapq:anon_109:pending_disconnect: false
eq_tapq:anon_109:reserved: 0
eq_tapq:anon_109:supports_ack: true
eq_tapq:anon_109:type: consumer
eq_tapq:anon_110:connected: true
eq_tapq:anon_110:created: 89
eq_tapq:anon_110:num_checkpoint_end: 3
eq_tapq:anon_110:num_checkpoint_end_failed: 0
eq_tapq:anon_110:num_checkpoint_start: 6
eq_tapq:anon_110:num_checkpoint_start_failed: 0
eq_tapq:anon_110:num_delete: 46
eq_tapq:anon_110:num_delete_failed: 0
eq_tapq:anon_110:num_flush: 0
eq_tapq:anon_110:num_flush_failed: 0
eq_tapq:anon_110:num_mutation: 9236
eq_tapq:anon_110:num_mutation_failed: 0
eq_tapq:anon_110:num_opaque: 8
eq_tapq:anon_110:num_opaque_failed: 0
eq_tapq:anon_110:num_unknown: 0
eq_tapq:anon_110:num_vbucket_set: 0
eq_tapq:anon_110:num_vbucket_set_failed: 0
eq_tapq:anon_110:pending_disconnect: false
eq_tapq:anon_110:reserved: 0
eq_tapq:anon_110:supports_ack: true
eq_tapq:anon_110:type: consumer
eq_tapq:anon_82:connected: true
eq_tapq:anon_82:created: 47
eq_tapq:anon_82:num_checkpoint_end: 3
eq_tapq:anon_82:num_checkpoint_end_failed: 0
eq_tapq:anon_82:num_checkpoint_start: 4
eq_tapq:anon_82:num_checkpoint_start_failed: 0
eq_tapq:anon_82:num_delete: 17
eq_tapq:anon_82:num_delete_failed: 0
eq_tapq:anon_82:num_flush: 0
eq_tapq:anon_82:num_flush_failed: 0
eq_tapq:anon_82:num_mutation: 2992
eq_tapq:anon_82:num_mutation_failed: 0
eq_tapq:anon_82:num_opaque: 4
eq_tapq:anon_82:num_opaque_failed: 0
eq_tapq:anon_82:num_unknown: 0
eq_tapq:anon_82:num_vbucket_set: 0
eq_tapq:anon_82:num_vbucket_set_failed: 0
eq_tapq:anon_82:pending_disconnect: false
eq_tapq:anon_82:reserved: 0
eq_tapq:anon_82:supports_ack: true
eq_tapq:anon_82:type: consumer
eq_tapq:anon_98:connected: true
eq_tapq:anon_98:created: 54
eq_tapq:anon_98:num_checkpoint_end: 3
eq_tapq:anon_98:num_checkpoint_end_failed: 0
eq_tapq:anon_98:num_checkpoint_start: 6
eq_tapq:anon_98:num_checkpoint_start_failed: 0
eq_tapq:anon_98:num_delete: 54
eq_tapq:anon_98:num_delete_failed: 0
eq_tapq:anon_98:num_flush: 0
eq_tapq:anon_98:num_flush_failed: 0
eq_tapq:anon_98:num_mutation: 9180
eq_tapq:anon_98:num_mutation_failed: 0
eq_tapq:anon_98:num_opaque: 8
eq_tapq:anon_98:num_opaque_failed: 0
eq_tapq:anon_98:num_unknown: 0
eq_tapq:anon_98:num_vbucket_set: 0
eq_tapq:anon_98:num_vbucket_set_failed: 0
eq_tapq:anon_98:pending_disconnect: false
eq_tapq:anon_98:reserved: 0
eq_tapq:anon_98:supports_ack: true
eq_tapq:anon_98:type: consumer
eq_tapq:replication_ns_1@10.1.3.74:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_seqno: 12322
eq_tapq:replication_ns_1@10.1.3.74:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.74:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.74:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.74:complete: false
eq_tapq:replication_ns_1@10.1.3.74:connected: true
eq_tapq:replication_ns_1@10.1.3.74:created: 42
eq_tapq:replication_ns_1@10.1.3.74:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.74:empty: true
eq_tapq:replication_ns_1@10.1.3.74:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.74:has_item: false
eq_tapq:replication_ns_1@10.1.3.74:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.74:idle: true
eq_tapq:replication_ns_1@10.1.3.74:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.74:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.74:paused: 1
eq_tapq:replication_ns_1@10.1.3.74:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:qlen: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_drain: 9307
eq_tapq:replication_ns_1@10.1.3.74:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.74:rec_fetched: 12305
eq_tapq:replication_ns_1@10.1.3.74:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.74:recv_ack_seqno: 12321
eq_tapq:replication_ns_1@10.1.3.74:reserved: 1
eq_tapq:replication_ns_1@10.1.3.74:seqno_ack_requested: 12321
eq_tapq:replication_ns_1@10.1.3.74:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.74:suspended: false
eq_tapq:replication_ns_1@10.1.3.74:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.74:type: producer
eq_tapq:replication_ns_1@10.1.3.74:vb_filter: { 23, 24, 28 }
eq_tapq:replication_ns_1@10.1.3.74:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.75:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_seqno: 10727
eq_tapq:replication_ns_1@10.1.3.75:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.75:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.75:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.75:complete: false
eq_tapq:replication_ns_1@10.1.3.75:connected: true
eq_tapq:replication_ns_1@10.1.3.75:created: 40
eq_tapq:replication_ns_1@10.1.3.75:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.75:empty: true
eq_tapq:replication_ns_1@10.1.3.75:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.75:has_item: false
eq_tapq:replication_ns_1@10.1.3.75:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.75:idle: true
eq_tapq:replication_ns_1@10.1.3.75:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.75:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.75:paused: 1
eq_tapq:replication_ns_1@10.1.3.75:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:qlen: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_drain: 10026
eq_tapq:replication_ns_1@10.1.3.75:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.75:rec_fetched: 10712
eq_tapq:replication_ns_1@10.1.3.75:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.75:recv_ack_seqno: 10726
eq_tapq:replication_ns_1@10.1.3.75:reserved: 1
eq_tapq:replication_ns_1@10.1.3.75:seqno_ack_requested: 10726
eq_tapq:replication_ns_1@10.1.3.75:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.75:suspended: false
eq_tapq:replication_ns_1@10.1.3.75:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.75:type: producer
eq_tapq:replication_ns_1@10.1.3.75:vb_filter: { 4, 29, 30 }
eq_tapq:replication_ns_1@10.1.3.75:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.76:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_seqno: 16494
eq_tapq:replication_ns_1@10.1.3.76:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.76:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.76:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.76:complete: false
eq_tapq:replication_ns_1@10.1.3.76:connected: true
eq_tapq:replication_ns_1@10.1.3.76:created: 44
eq_tapq:replication_ns_1@10.1.3.76:disconnects: 3
eq_tapq:replication_ns_1@10.1.3.76:empty: true
eq_tapq:replication_ns_1@10.1.3.76:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.76:has_item: false
eq_tapq:replication_ns_1@10.1.3.76:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.76:idle: true
eq_tapq:replication_ns_1@10.1.3.76:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.76:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.76:paused: 1
eq_tapq:replication_ns_1@10.1.3.76:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:qlen: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_drain: 16443
eq_tapq:replication_ns_1@10.1.3.76:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.76:rec_fetched: 16475
eq_tapq:replication_ns_1@10.1.3.76:reconnects: 3
eq_tapq:replication_ns_1@10.1.3.76:recv_ack_seqno: 16493
eq_tapq:replication_ns_1@10.1.3.76:reserved: 1
eq_tapq:replication_ns_1@10.1.3.76:seqno_ack_requested: 16493
eq_tapq:replication_ns_1@10.1.3.76:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.76:suspended: false
eq_tapq:replication_ns_1@10.1.3.76:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.76:type: producer
eq_tapq:replication_ns_1@10.1.3.76:vb_filter: { 14, 31, 32, 40 }
eq_tapq:replication_ns_1@10.1.3.76:vb_filters: 4
eq_tapq:replication_ns_1@10.1.3.78:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_seqno: 7743
eq_tapq:replication_ns_1@10.1.3.78:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.78:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.78:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.78:complete: false
eq_tapq:replication_ns_1@10.1.3.78:connected: true
eq_tapq:replication_ns_1@10.1.3.78:created: 49
eq_tapq:replication_ns_1@10.1.3.78:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.78:empty: false
eq_tapq:replication_ns_1@10.1.3.78:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.78:has_item: false
eq_tapq:replication_ns_1@10.1.3.78:has_queued_item: true
eq_tapq:replication_ns_1@10.1.3.78:idle: false
eq_tapq:replication_ns_1@10.1.3.78:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.78:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.78:paused: 1
eq_tapq:replication_ns_1@10.1.3.78:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:qlen: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_drain: 6199
eq_tapq:replication_ns_1@10.1.3.78:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.78:rec_fetched: 7728
eq_tapq:replication_ns_1@10.1.3.78:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.78:recv_ack_seqno: 7742
eq_tapq:replication_ns_1@10.1.3.78:reserved: 1
eq_tapq:replication_ns_1@10.1.3.78:seqno_ack_requested: 7742
eq_tapq:replication_ns_1@10.1.3.78:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.78:suspended: false
eq_tapq:replication_ns_1@10.1.3.78:total_backlog_size: 1
eq_tapq:replication_ns_1@10.1.3.78:total_noops: 56
eq_tapq:replication_ns_1@10.1.3.78:type: producer
eq_tapq:replication_ns_1@10.1.3.78:vb_filter: { 33, 34, 50 }
eq_tapq:replication_ns_1@10.1.3.78:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.79:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_seqno: 13794
eq_tapq:replication_ns_1@10.1.3.79:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.79:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.79:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.79:complete: false
eq_tapq:replication_ns_1@10.1.3.79:connected: true
eq_tapq:replication_ns_1@10.1.3.79:created: 44
eq_tapq:replication_ns_1@10.1.3.79:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.79:empty: true
eq_tapq:replication_ns_1@10.1.3.79:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.79:has_item: false
eq_tapq:replication_ns_1@10.1.3.79:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.79:idle: true
eq_tapq:replication_ns_1@10.1.3.79:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.79:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.79:paused: 1
eq_tapq:replication_ns_1@10.1.3.79:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:qlen: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_drain: 13324
eq_tapq:replication_ns_1@10.1.3.79:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.79:rec_fetched: 13777
eq_tapq:replication_ns_1@10.1.3.79:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.79:recv_ack_seqno: 13793
eq_tapq:replication_ns_1@10.1.3.79:reserved: 1
eq_tapq:replication_ns_1@10.1.3.79:seqno_ack_requested: 13793
eq_tapq:replication_ns_1@10.1.3.79:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.79:suspended: false
eq_tapq:replication_ns_1@10.1.3.79:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.79:type: producer
eq_tapq:replication_ns_1@10.1.3.79:vb_filter: { 35, 41, 59 }
eq_tapq:replication_ns_1@10.1.3.79:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.80:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_seqno: 6180
eq_tapq:replication_ns_1@10.1.3.80:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.80:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.80:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.80:complete: false
eq_tapq:replication_ns_1@10.1.3.80:connected: true
eq_tapq:replication_ns_1@10.1.3.80:created: 45
eq_tapq:replication_ns_1@10.1.3.80:empty: true
eq_tapq:replication_ns_1@10.1.3.80:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.80:has_item: false
eq_tapq:replication_ns_1@10.1.3.80:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.80:idle: true
eq_tapq:replication_ns_1@10.1.3.80:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.80:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.80:paused: 1
eq_tapq:replication_ns_1@10.1.3.80:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:qlen: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_drain: 6161
eq_tapq:replication_ns_1@10.1.3.80:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.80:rec_fetched: 6173
eq_tapq:replication_ns_1@10.1.3.80:recv_ack_seqno: 6179
eq_tapq:replication_ns_1@10.1.3.80:reserved: 1
eq_tapq:replication_ns_1@10.1.3.80:seqno_ack_requested: 6179
eq_tapq:replication_ns_1@10.1.3.80:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.80:suspended: false
eq_tapq:replication_ns_1@10.1.3.80:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.80:type: producer
eq_tapq:replication_ns_1@10.1.3.80:vb_filter: { 15, 36 }
eq_tapq:replication_ns_1@10.1.3.80:vb_filters: 2
10.1.3.79
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 12
ep_tap_deletes: 3736
ep_tap_fg_fetched: 605813
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 245529
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 616177
ep_tap_total_queue: 0
eq_tapq:anon_505:connected: true
eq_tapq:anon_505:created: 257
eq_tapq:anon_505:num_checkpoint_end: 3
eq_tapq:anon_505:num_checkpoint_end_failed: 0
eq_tapq:anon_505:num_checkpoint_start: 5
eq_tapq:anon_505:num_checkpoint_start_failed: 0
eq_tapq:anon_505:num_delete: 37
eq_tapq:anon_505:num_delete_failed: 0
eq_tapq:anon_505:num_flush: 0
eq_tapq:anon_505:num_flush_failed: 0
eq_tapq:anon_505:num_mutation: 6316
eq_tapq:anon_505:num_mutation_failed: 0
eq_tapq:anon_505:num_opaque: 6
eq_tapq:anon_505:num_opaque_failed: 0
eq_tapq:anon_505:num_unknown: 0
eq_tapq:anon_505:num_vbucket_set: 0
eq_tapq:anon_505:num_vbucket_set_failed: 0
eq_tapq:anon_505:pending_disconnect: false
eq_tapq:anon_505:reserved: 0
eq_tapq:anon_505:supports_ack: true
eq_tapq:anon_505:type: consumer
eq_tapq:anon_524:connected: true
eq_tapq:anon_524:created: 263
eq_tapq:anon_524:num_checkpoint_end: 3
eq_tapq:anon_524:num_checkpoint_end_failed: 0
eq_tapq:anon_524:num_checkpoint_start: 3
eq_tapq:anon_524:num_checkpoint_start_failed: 0
eq_tapq:anon_524:num_delete: 0
eq_tapq:anon_524:num_delete_failed: 0
eq_tapq:anon_524:num_flush: 0
eq_tapq:anon_524:num_flush_failed: 0
eq_tapq:anon_524:num_mutation: 0
eq_tapq:anon_524:num_mutation_failed: 0
eq_tapq:anon_524:num_opaque: 2
eq_tapq:anon_524:num_opaque_failed: 0
eq_tapq:anon_524:num_unknown: 0
eq_tapq:anon_524:num_vbucket_set: 0
eq_tapq:anon_524:num_vbucket_set_failed: 0
eq_tapq:anon_524:pending_disconnect: false
eq_tapq:anon_524:reserved: 0
eq_tapq:anon_524:supports_ack: true
eq_tapq:anon_524:type: consumer
eq_tapq:anon_525:connected: true
eq_tapq:anon_525:created: 263
eq_tapq:anon_525:num_checkpoint_end: 3
eq_tapq:anon_525:num_checkpoint_end_failed: 0
eq_tapq:anon_525:num_checkpoint_start: 5
eq_tapq:anon_525:num_checkpoint_start_failed: 0
eq_tapq:anon_525:num_delete: 457
eq_tapq:anon_525:num_delete_failed: 0
eq_tapq:anon_525:num_flush: 0
eq_tapq:anon_525:num_flush_failed: 0
eq_tapq:anon_525:num_mutation: 7195
eq_tapq:anon_525:num_mutation_failed: 0
eq_tapq:anon_525:num_opaque: 8
eq_tapq:anon_525:num_opaque_failed: 0
eq_tapq:anon_525:num_unknown: 0
eq_tapq:anon_525:num_vbucket_set: 0
eq_tapq:anon_525:num_vbucket_set_failed: 0
eq_tapq:anon_525:pending_disconnect: false
eq_tapq:anon_525:reserved: 0
eq_tapq:anon_525:supports_ack: true
eq_tapq:anon_525:type: consumer
eq_tapq:anon_526:connected: true
eq_tapq:anon_526:created: 263
eq_tapq:anon_526:num_checkpoint_end: 2
eq_tapq:anon_526:num_checkpoint_end_failed: 0
eq_tapq:anon_526:num_checkpoint_start: 2
eq_tapq:anon_526:num_checkpoint_start_failed: 0
eq_tapq:anon_526:num_delete: 0
eq_tapq:anon_526:num_delete_failed: 0
eq_tapq:anon_526:num_flush: 0
eq_tapq:anon_526:num_flush_failed: 0
eq_tapq:anon_526:num_mutation: 0
eq_tapq:anon_526:num_mutation_failed: 0
eq_tapq:anon_526:num_opaque: 2
eq_tapq:anon_526:num_opaque_failed: 0
eq_tapq:anon_526:num_unknown: 0
eq_tapq:anon_526:num_vbucket_set: 0
eq_tapq:anon_526:num_vbucket_set_failed: 0
eq_tapq:anon_526:pending_disconnect: false
eq_tapq:anon_526:reserved: 0
eq_tapq:anon_526:supports_ack: true
eq_tapq:anon_526:type: consumer
eq_tapq:anon_532:connected: true
eq_tapq:anon_532:created: 264
eq_tapq:anon_532:num_checkpoint_end: 3
eq_tapq:anon_532:num_checkpoint_end_failed: 0
eq_tapq:anon_532:num_checkpoint_start: 4
eq_tapq:anon_532:num_checkpoint_start_failed: 0
eq_tapq:anon_532:num_delete: 44
eq_tapq:anon_532:num_delete_failed: 0
eq_tapq:anon_532:num_flush: 0
eq_tapq:anon_532:num_flush_failed: 0
eq_tapq:anon_532:num_mutation: 4102
eq_tapq:anon_532:num_mutation_failed: 0
eq_tapq:anon_532:num_opaque: 4
eq_tapq:anon_532:num_opaque_failed: 0
eq_tapq:anon_532:num_unknown: 0
eq_tapq:anon_532:num_vbucket_set: 0
eq_tapq:anon_532:num_vbucket_set_failed: 0
eq_tapq:anon_532:pending_disconnect: false
eq_tapq:anon_532:reserved: 0
eq_tapq:anon_532:supports_ack: true
eq_tapq:anon_532:type: consumer
eq_tapq:anon_534:connected: true
eq_tapq:anon_534:created: 289
eq_tapq:anon_534:num_checkpoint_end: 4
eq_tapq:anon_534:num_checkpoint_end_failed: 0
eq_tapq:anon_534:num_checkpoint_start: 8
eq_tapq:anon_534:num_checkpoint_start_failed: 0
eq_tapq:anon_534:num_delete: 75
eq_tapq:anon_534:num_delete_failed: 0
eq_tapq:anon_534:num_flush: 0
eq_tapq:anon_534:num_flush_failed: 0
eq_tapq:anon_534:num_mutation: 12382
eq_tapq:anon_534:num_mutation_failed: 0
eq_tapq:anon_534:num_opaque: 10
eq_tapq:anon_534:num_opaque_failed: 0
eq_tapq:anon_534:num_unknown: 0
eq_tapq:anon_534:num_vbucket_set: 0
eq_tapq:anon_534:num_vbucket_set_failed: 0
eq_tapq:anon_534:pending_disconnect: false
eq_tapq:anon_534:reserved: 0
eq_tapq:anon_534:supports_ack: true
eq_tapq:anon_534:type: consumer
eq_tapq:replication_ns_1@10.1.3.74:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_seqno: 12440
eq_tapq:replication_ns_1@10.1.3.74:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.74:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.74:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.74:complete: false
eq_tapq:replication_ns_1@10.1.3.74:connected: true
eq_tapq:replication_ns_1@10.1.3.74:created: 295
eq_tapq:replication_ns_1@10.1.3.74:empty: true
eq_tapq:replication_ns_1@10.1.3.74:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.74:has_item: false
eq_tapq:replication_ns_1@10.1.3.74:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.74:idle: true
eq_tapq:replication_ns_1@10.1.3.74:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.74:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.74:paused: 1
eq_tapq:replication_ns_1@10.1.3.74:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:qlen: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_drain: 12407
eq_tapq:replication_ns_1@10.1.3.74:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.74:rec_fetched: 12429
eq_tapq:replication_ns_1@10.1.3.74:recv_ack_seqno: 12439
eq_tapq:replication_ns_1@10.1.3.74:reserved: 1
eq_tapq:replication_ns_1@10.1.3.74:seqno_ack_requested: 12439
eq_tapq:replication_ns_1@10.1.3.74:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.74:suspended: false
eq_tapq:replication_ns_1@10.1.3.74:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:total_noops: 55
eq_tapq:replication_ns_1@10.1.3.74:type: producer
eq_tapq:replication_ns_1@10.1.3.74:vb_filter: { 26, 46, 47, 62 }
eq_tapq:replication_ns_1@10.1.3.74:vb_filters: 4
eq_tapq:replication_ns_1@10.1.3.75:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_seqno: 75909
eq_tapq:replication_ns_1@10.1.3.75:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.75:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.75:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.75:complete: false
eq_tapq:replication_ns_1@10.1.3.75:connected: true
eq_tapq:replication_ns_1@10.1.3.75:created: 102
eq_tapq:replication_ns_1@10.1.3.75:disconnects: 25
eq_tapq:replication_ns_1@10.1.3.75:empty: true
eq_tapq:replication_ns_1@10.1.3.75:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.75:has_item: false
eq_tapq:replication_ns_1@10.1.3.75:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.75:idle: true
eq_tapq:replication_ns_1@10.1.3.75:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.75:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.75:paused: 1
eq_tapq:replication_ns_1@10.1.3.75:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:qlen: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_drain: 74059
eq_tapq:replication_ns_1@10.1.3.75:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.75:rec_fetched: 75805
eq_tapq:replication_ns_1@10.1.3.75:reconnects: 25
eq_tapq:replication_ns_1@10.1.3.75:recv_ack_seqno: 75908
eq_tapq:replication_ns_1@10.1.3.75:reserved: 1
eq_tapq:replication_ns_1@10.1.3.75:seqno_ack_requested: 75908
eq_tapq:replication_ns_1@10.1.3.75:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.75:suspended: false
eq_tapq:replication_ns_1@10.1.3.75:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:total_noops: 64
eq_tapq:replication_ns_1@10.1.3.75:type: producer
eq_tapq:replication_ns_1@10.1.3.75:vb_filter: { 48, 63 }
eq_tapq:replication_ns_1@10.1.3.75:vb_filters: 2
eq_tapq:replication_ns_1@10.1.3.76:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_seqno: 52423
eq_tapq:replication_ns_1@10.1.3.76:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.76:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.76:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.76:complete: false
eq_tapq:replication_ns_1@10.1.3.76:connected: true
eq_tapq:replication_ns_1@10.1.3.76:created: 102
eq_tapq:replication_ns_1@10.1.3.76:disconnects: 22
eq_tapq:replication_ns_1@10.1.3.76:empty: true
eq_tapq:replication_ns_1@10.1.3.76:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.76:has_item: false
eq_tapq:replication_ns_1@10.1.3.76:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.76:idle: true
eq_tapq:replication_ns_1@10.1.3.76:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.76:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.76:paused: 1
eq_tapq:replication_ns_1@10.1.3.76:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:qlen: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_drain: 52248
eq_tapq:replication_ns_1@10.1.3.76:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.76:rec_fetched: 52346
eq_tapq:replication_ns_1@10.1.3.76:reconnects: 22
eq_tapq:replication_ns_1@10.1.3.76:recv_ack_seqno: 52422
eq_tapq:replication_ns_1@10.1.3.76:reserved: 1
eq_tapq:replication_ns_1@10.1.3.76:seqno_ack_requested: 52422
eq_tapq:replication_ns_1@10.1.3.76:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.76:suspended: false
eq_tapq:replication_ns_1@10.1.3.76:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:total_noops: 64
eq_tapq:replication_ns_1@10.1.3.76:type: producer
eq_tapq:replication_ns_1@10.1.3.76:vb_filter: { 17, 49 }
eq_tapq:replication_ns_1@10.1.3.76:vb_filters: 2
eq_tapq:replication_ns_1@10.1.3.77:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_seqno: 7189
eq_tapq:replication_ns_1@10.1.3.77:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.77:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.77:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.77:complete: false
eq_tapq:replication_ns_1@10.1.3.77:connected: true
eq_tapq:replication_ns_1@10.1.3.77:created: 52
eq_tapq:replication_ns_1@10.1.3.77:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.77:empty: true
eq_tapq:replication_ns_1@10.1.3.77:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.77:has_item: false
eq_tapq:replication_ns_1@10.1.3.77:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.77:idle: true
eq_tapq:replication_ns_1@10.1.3.77:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.77:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.77:paused: 1
eq_tapq:replication_ns_1@10.1.3.77:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:qlen: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_drain: 6887
eq_tapq:replication_ns_1@10.1.3.77:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.77:rec_fetched: 7164
eq_tapq:replication_ns_1@10.1.3.77:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.77:recv_ack_seqno: 7188
eq_tapq:replication_ns_1@10.1.3.77:reserved: 1
eq_tapq:replication_ns_1@10.1.3.77:seqno_ack_requested: 7188
eq_tapq:replication_ns_1@10.1.3.77:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.77:suspended: false
eq_tapq:replication_ns_1@10.1.3.77:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:total_noops: 59
eq_tapq:replication_ns_1@10.1.3.77:type: producer
eq_tapq:replication_ns_1@10.1.3.77:vb_filter: { 42, 50 }
eq_tapq:replication_ns_1@10.1.3.77:vb_filters: 2
eq_tapq:replication_ns_1@10.1.3.78:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_seqno: 17754
eq_tapq:replication_ns_1@10.1.3.78:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.78:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.78:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.78:complete: false
eq_tapq:replication_ns_1@10.1.3.78:connected: true
eq_tapq:replication_ns_1@10.1.3.78:created: 239
eq_tapq:replication_ns_1@10.1.3.78:disconnects: 3
eq_tapq:replication_ns_1@10.1.3.78:empty: true
eq_tapq:replication_ns_1@10.1.3.78:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.78:has_item: false
eq_tapq:replication_ns_1@10.1.3.78:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.78:idle: true
eq_tapq:replication_ns_1@10.1.3.78:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.78:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.78:paused: 1
eq_tapq:replication_ns_1@10.1.3.78:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:qlen: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_drain: 16202
eq_tapq:replication_ns_1@10.1.3.78:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.78:rec_fetched: 17733
eq_tapq:replication_ns_1@10.1.3.78:reconnects: 3
eq_tapq:replication_ns_1@10.1.3.78:recv_ack_seqno: 17753
eq_tapq:replication_ns_1@10.1.3.78:reserved: 1
eq_tapq:replication_ns_1@10.1.3.78:seqno_ack_requested: 17753
eq_tapq:replication_ns_1@10.1.3.78:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.78:suspended: false
eq_tapq:replication_ns_1@10.1.3.78:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:total_noops: 58
eq_tapq:replication_ns_1@10.1.3.78:type: producer
eq_tapq:replication_ns_1@10.1.3.78:vb_filter: { 6, 35, 51, 52 }
eq_tapq:replication_ns_1@10.1.3.78:vb_filters: 4
eq_tapq:replication_ns_1@10.1.3.80:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_seqno: 84791
eq_tapq:replication_ns_1@10.1.3.80:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.80:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.80:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.80:complete: false
eq_tapq:replication_ns_1@10.1.3.80:connected: true
eq_tapq:replication_ns_1@10.1.3.80:created: 102
eq_tapq:replication_ns_1@10.1.3.80:disconnects: 31
eq_tapq:replication_ns_1@10.1.3.80:empty: true
eq_tapq:replication_ns_1@10.1.3.80:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.80:has_item: false
eq_tapq:replication_ns_1@10.1.3.80:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.80:idle: true
eq_tapq:replication_ns_1@10.1.3.80:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.80:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.80:paused: 1
eq_tapq:replication_ns_1@10.1.3.80:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:qlen: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_drain: 83726
eq_tapq:replication_ns_1@10.1.3.80:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.80:rec_fetched: 84664
eq_tapq:replication_ns_1@10.1.3.80:reconnects: 31
eq_tapq:replication_ns_1@10.1.3.80:recv_ack_seqno: 84790
eq_tapq:replication_ns_1@10.1.3.80:reserved: 1
eq_tapq:replication_ns_1@10.1.3.80:seqno_ack_requested: 84790
eq_tapq:replication_ns_1@10.1.3.80:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.80:suspended: false
eq_tapq:replication_ns_1@10.1.3.80:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:total_noops: 64
eq_tapq:replication_ns_1@10.1.3.80:type: producer
eq_tapq:replication_ns_1@10.1.3.80:vb_filter: { 7, 43, 53, 54 }
eq_tapq:replication_ns_1@10.1.3.80:vb_filters: 4
10.1.3.76
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 12
ep_tap_deletes: 3833
ep_tap_fg_fetched: 641739
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 258870
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 655468
ep_tap_total_queue: 0
eq_tapq:anon_517:connected: true
eq_tapq:anon_517:created: 252
eq_tapq:anon_517:num_checkpoint_end: 4
eq_tapq:anon_517:num_checkpoint_end_failed: 0
eq_tapq:anon_517:num_checkpoint_start: 5
eq_tapq:anon_517:num_checkpoint_start_failed: 0
eq_tapq:anon_517:num_delete: 26
eq_tapq:anon_517:num_delete_failed: 0
eq_tapq:anon_517:num_flush: 0
eq_tapq:anon_517:num_flush_failed: 0
eq_tapq:anon_517:num_mutation: 3022
eq_tapq:anon_517:num_mutation_failed: 0
eq_tapq:anon_517:num_opaque: 4
eq_tapq:anon_517:num_opaque_failed: 0
eq_tapq:anon_517:num_unknown: 0
eq_tapq:anon_517:num_vbucket_set: 0
eq_tapq:anon_517:num_vbucket_set_failed: 0
eq_tapq:anon_517:pending_disconnect: false
eq_tapq:anon_517:reserved: 0
eq_tapq:anon_517:supports_ack: true
eq_tapq:anon_517:type: consumer
eq_tapq:anon_519:connected: true
eq_tapq:anon_519:created: 254
eq_tapq:anon_519:num_checkpoint_end: 4
eq_tapq:anon_519:num_checkpoint_end_failed: 0
eq_tapq:anon_519:num_checkpoint_start: 6
eq_tapq:anon_519:num_checkpoint_start_failed: 0
eq_tapq:anon_519:num_delete: 24
eq_tapq:anon_519:num_delete_failed: 0
eq_tapq:anon_519:num_flush: 0
eq_tapq:anon_519:num_flush_failed: 0
eq_tapq:anon_519:num_mutation: 6252
eq_tapq:anon_519:num_mutation_failed: 0
eq_tapq:anon_519:num_opaque: 6
eq_tapq:anon_519:num_opaque_failed: 0
eq_tapq:anon_519:num_unknown: 0
eq_tapq:anon_519:num_vbucket_set: 0
eq_tapq:anon_519:num_vbucket_set_failed: 0
eq_tapq:anon_519:pending_disconnect: false
eq_tapq:anon_519:reserved: 0
eq_tapq:anon_519:supports_ack: true
eq_tapq:anon_519:type: consumer
eq_tapq:anon_531:connected: true
eq_tapq:anon_531:created: 256
eq_tapq:anon_531:num_checkpoint_end: 3
eq_tapq:anon_531:num_checkpoint_end_failed: 0
eq_tapq:anon_531:num_checkpoint_start: 5
eq_tapq:anon_531:num_checkpoint_start_failed: 0
eq_tapq:anon_531:num_delete: 39
eq_tapq:anon_531:num_delete_failed: 0
eq_tapq:anon_531:num_flush: 0
eq_tapq:anon_531:num_flush_failed: 0
eq_tapq:anon_531:num_mutation: 6262
eq_tapq:anon_531:num_mutation_failed: 0
eq_tapq:anon_531:num_opaque: 6
eq_tapq:anon_531:num_opaque_failed: 0
eq_tapq:anon_531:num_unknown: 0
eq_tapq:anon_531:num_vbucket_set: 0
eq_tapq:anon_531:num_vbucket_set_failed: 0
eq_tapq:anon_531:pending_disconnect: false
eq_tapq:anon_531:reserved: 0
eq_tapq:anon_531:supports_ack: true
eq_tapq:anon_531:type: consumer
eq_tapq:anon_539:connected: true
eq_tapq:anon_539:created: 259
eq_tapq:anon_539:num_checkpoint_end: 2
eq_tapq:anon_539:num_checkpoint_end_failed: 0
eq_tapq:anon_539:num_checkpoint_start: 2
eq_tapq:anon_539:num_checkpoint_start_failed: 0
eq_tapq:anon_539:num_delete: 0
eq_tapq:anon_539:num_delete_failed: 0
eq_tapq:anon_539:num_flush: 0
eq_tapq:anon_539:num_flush_failed: 0
eq_tapq:anon_539:num_mutation: 0
eq_tapq:anon_539:num_mutation_failed: 0
eq_tapq:anon_539:num_opaque: 2
eq_tapq:anon_539:num_opaque_failed: 0
eq_tapq:anon_539:num_unknown: 0
eq_tapq:anon_539:num_vbucket_set: 0
eq_tapq:anon_539:num_vbucket_set_failed: 0
eq_tapq:anon_539:pending_disconnect: false
eq_tapq:anon_539:reserved: 0
eq_tapq:anon_539:supports_ack: true
eq_tapq:anon_539:type: consumer
eq_tapq:anon_543:connected: true
eq_tapq:anon_543:created: 260
eq_tapq:anon_543:num_checkpoint_end: 2
eq_tapq:anon_543:num_checkpoint_end_failed: 0
eq_tapq:anon_543:num_checkpoint_start: 3
eq_tapq:anon_543:num_checkpoint_start_failed: 0
eq_tapq:anon_543:num_delete: 21
eq_tapq:anon_543:num_delete_failed: 0
eq_tapq:anon_543:num_flush: 0
eq_tapq:anon_543:num_flush_failed: 0
eq_tapq:anon_543:num_mutation: 3084
eq_tapq:anon_543:num_mutation_failed: 0
eq_tapq:anon_543:num_opaque: 4
eq_tapq:anon_543:num_opaque_failed: 0
eq_tapq:anon_543:num_unknown: 0
eq_tapq:anon_543:num_vbucket_set: 0
eq_tapq:anon_543:num_vbucket_set_failed: 0
eq_tapq:anon_543:pending_disconnect: false
eq_tapq:anon_543:reserved: 0
eq_tapq:anon_543:supports_ack: true
eq_tapq:anon_543:type: consumer
eq_tapq:anon_548:connected: true
eq_tapq:anon_548:created: 263
eq_tapq:anon_548:num_checkpoint_end: 3
eq_tapq:anon_548:num_checkpoint_end_failed: 0
eq_tapq:anon_548:num_checkpoint_start: 3
eq_tapq:anon_548:num_checkpoint_start_failed: 0
eq_tapq:anon_548:num_delete: 0
eq_tapq:anon_548:num_delete_failed: 0
eq_tapq:anon_548:num_flush: 0
eq_tapq:anon_548:num_flush_failed: 0
eq_tapq:anon_548:num_mutation: 0
eq_tapq:anon_548:num_mutation_failed: 0
eq_tapq:anon_548:num_opaque: 2
eq_tapq:anon_548:num_opaque_failed: 0
eq_tapq:anon_548:num_unknown: 0
eq_tapq:anon_548:num_vbucket_set: 0
eq_tapq:anon_548:num_vbucket_set_failed: 0
eq_tapq:anon_548:pending_disconnect: false
eq_tapq:anon_548:reserved: 0
eq_tapq:anon_548:supports_ack: true
eq_tapq:anon_548:type: consumer
eq_tapq:replication_ns_1@10.1.3.74:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_seqno: 15599
eq_tapq:replication_ns_1@10.1.3.74:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.74:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.74:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.74:complete: false
eq_tapq:replication_ns_1@10.1.3.74:connected: true
eq_tapq:replication_ns_1@10.1.3.74:created: 240
eq_tapq:replication_ns_1@10.1.3.74:disconnects: 3
eq_tapq:replication_ns_1@10.1.3.74:empty: true
eq_tapq:replication_ns_1@10.1.3.74:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.74:has_item: false
eq_tapq:replication_ns_1@10.1.3.74:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.74:idle: true
eq_tapq:replication_ns_1@10.1.3.74:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.74:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.74:paused: 1
eq_tapq:replication_ns_1@10.1.3.74:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:qlen: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_drain: 13957
eq_tapq:replication_ns_1@10.1.3.74:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.74:rec_fetched: 15580
eq_tapq:replication_ns_1@10.1.3.74:reconnects: 3
eq_tapq:replication_ns_1@10.1.3.74:recv_ack_seqno: 15598
eq_tapq:replication_ns_1@10.1.3.74:reserved: 1
eq_tapq:replication_ns_1@10.1.3.74:seqno_ack_requested: 15598
eq_tapq:replication_ns_1@10.1.3.74:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.74:suspended: false
eq_tapq:replication_ns_1@10.1.3.74:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:total_noops: 58
eq_tapq:replication_ns_1@10.1.3.74:type: producer
eq_tapq:replication_ns_1@10.1.3.74:vb_filter: { 12, 19, 20, 39 }
eq_tapq:replication_ns_1@10.1.3.74:vb_filters: 4
eq_tapq:replication_ns_1@10.1.3.75:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_seqno: 79657
eq_tapq:replication_ns_1@10.1.3.75:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.75:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.75:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.75:complete: false
eq_tapq:replication_ns_1@10.1.3.75:connected: true
eq_tapq:replication_ns_1@10.1.3.75:created: 101
eq_tapq:replication_ns_1@10.1.3.75:disconnects: 31
eq_tapq:replication_ns_1@10.1.3.75:empty: true
eq_tapq:replication_ns_1@10.1.3.75:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.75:has_item: false
eq_tapq:replication_ns_1@10.1.3.75:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.75:idle: true
eq_tapq:replication_ns_1@10.1.3.75:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.75:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.75:paused: 1
eq_tapq:replication_ns_1@10.1.3.75:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:qlen: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_drain: 77830
eq_tapq:replication_ns_1@10.1.3.75:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.75:rec_fetched: 79547
eq_tapq:replication_ns_1@10.1.3.75:reconnects: 31
eq_tapq:replication_ns_1@10.1.3.75:recv_ack_seqno: 79656
eq_tapq:replication_ns_1@10.1.3.75:reserved: 1
eq_tapq:replication_ns_1@10.1.3.75:seqno_ack_requested: 79656
eq_tapq:replication_ns_1@10.1.3.75:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.75:suspended: false
eq_tapq:replication_ns_1@10.1.3.75:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:total_noops: 63
eq_tapq:replication_ns_1@10.1.3.75:type: producer
eq_tapq:replication_ns_1@10.1.3.75:vb_filter: { 21, 22, 31, 32 }
eq_tapq:replication_ns_1@10.1.3.75:vb_filters: 4
eq_tapq:replication_ns_1@10.1.3.77:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_seqno: 12132
eq_tapq:replication_ns_1@10.1.3.77:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.77:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.77:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.77:complete: false
eq_tapq:replication_ns_1@10.1.3.77:connected: true
eq_tapq:replication_ns_1@10.1.3.77:created: 242
eq_tapq:replication_ns_1@10.1.3.77:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.77:empty: true
eq_tapq:replication_ns_1@10.1.3.77:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.77:has_item: false
eq_tapq:replication_ns_1@10.1.3.77:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.77:idle: true
eq_tapq:replication_ns_1@10.1.3.77:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.77:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.77:paused: 1
eq_tapq:replication_ns_1@10.1.3.77:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:qlen: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_drain: 12093
eq_tapq:replication_ns_1@10.1.3.77:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.77:rec_fetched: 12117
eq_tapq:replication_ns_1@10.1.3.77:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.77:recv_ack_seqno: 12131
eq_tapq:replication_ns_1@10.1.3.77:reserved: 1
eq_tapq:replication_ns_1@10.1.3.77:seqno_ack_requested: 12131
eq_tapq:replication_ns_1@10.1.3.77:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.77:suspended: false
eq_tapq:replication_ns_1@10.1.3.77:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:total_noops: 58
eq_tapq:replication_ns_1@10.1.3.77:type: producer
eq_tapq:replication_ns_1@10.1.3.77:vb_filter: { 13, 23, 24 }
eq_tapq:replication_ns_1@10.1.3.77:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.78:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_seqno: 6312
eq_tapq:replication_ns_1@10.1.3.78:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.78:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.78:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.78:complete: false
eq_tapq:replication_ns_1@10.1.3.78:connected: true
eq_tapq:replication_ns_1@10.1.3.78:created: 290
eq_tapq:replication_ns_1@10.1.3.78:empty: true
eq_tapq:replication_ns_1@10.1.3.78:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.78:has_item: false
eq_tapq:replication_ns_1@10.1.3.78:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.78:idle: true
eq_tapq:replication_ns_1@10.1.3.78:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.78:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.78:paused: 1
eq_tapq:replication_ns_1@10.1.3.78:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:qlen: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_drain: 6293
eq_tapq:replication_ns_1@10.1.3.78:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.78:rec_fetched: 6305
eq_tapq:replication_ns_1@10.1.3.78:recv_ack_seqno: 6311
eq_tapq:replication_ns_1@10.1.3.78:reserved: 1
eq_tapq:replication_ns_1@10.1.3.78:seqno_ack_requested: 6311
eq_tapq:replication_ns_1@10.1.3.78:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.78:suspended: false
eq_tapq:replication_ns_1@10.1.3.78:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:total_noops: 55
eq_tapq:replication_ns_1@10.1.3.78:type: producer
eq_tapq:replication_ns_1@10.1.3.78:vb_filter: { 25, 58 }
eq_tapq:replication_ns_1@10.1.3.78:vb_filters: 2
eq_tapq:replication_ns_1@10.1.3.79:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_seqno: 92140
eq_tapq:replication_ns_1@10.1.3.79:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.79:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.79:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.79:complete: false
eq_tapq:replication_ns_1@10.1.3.79:connected: true
eq_tapq:replication_ns_1@10.1.3.79:created: 101
eq_tapq:replication_ns_1@10.1.3.79:disconnects: 28
eq_tapq:replication_ns_1@10.1.3.79:empty: true
eq_tapq:replication_ns_1@10.1.3.79:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.79:has_item: false
eq_tapq:replication_ns_1@10.1.3.79:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.79:idle: true
eq_tapq:replication_ns_1@10.1.3.79:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.79:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.79:paused: 1
eq_tapq:replication_ns_1@10.1.3.79:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:qlen: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_drain: 89796
eq_tapq:replication_ns_1@10.1.3.79:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.79:rec_fetched: 92024
eq_tapq:replication_ns_1@10.1.3.79:reconnects: 28
eq_tapq:replication_ns_1@10.1.3.79:recv_ack_seqno: 92139
eq_tapq:replication_ns_1@10.1.3.79:reserved: 1
eq_tapq:replication_ns_1@10.1.3.79:seqno_ack_requested: 92139
eq_tapq:replication_ns_1@10.1.3.79:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.79:suspended: false
eq_tapq:replication_ns_1@10.1.3.79:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:total_noops: 64
eq_tapq:replication_ns_1@10.1.3.79:type: producer
eq_tapq:replication_ns_1@10.1.3.79:vb_filter: { 2, 26 }
eq_tapq:replication_ns_1@10.1.3.79:vb_filters: 2
eq_tapq:replication_ns_1@10.1.3.80:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_seqno: 60966
eq_tapq:replication_ns_1@10.1.3.80:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.80:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.80:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.80:complete: false
eq_tapq:replication_ns_1@10.1.3.80:connected: true
eq_tapq:replication_ns_1@10.1.3.80:created: 101
eq_tapq:replication_ns_1@10.1.3.80:disconnects: 22
eq_tapq:replication_ns_1@10.1.3.80:empty: true
eq_tapq:replication_ns_1@10.1.3.80:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.80:has_item: false
eq_tapq:replication_ns_1@10.1.3.80:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.80:idle: true
eq_tapq:replication_ns_1@10.1.3.80:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.80:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.80:paused: 1
eq_tapq:replication_ns_1@10.1.3.80:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:qlen: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_drain: 58901
eq_tapq:replication_ns_1@10.1.3.80:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.80:rec_fetched: 60881
eq_tapq:replication_ns_1@10.1.3.80:reconnects: 22
eq_tapq:replication_ns_1@10.1.3.80:recv_ack_seqno: 60965
eq_tapq:replication_ns_1@10.1.3.80:reserved: 1
eq_tapq:replication_ns_1@10.1.3.80:seqno_ack_requested: 60965
eq_tapq:replication_ns_1@10.1.3.80:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.80:suspended: false
eq_tapq:replication_ns_1@10.1.3.80:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:total_noops: 64
eq_tapq:replication_ns_1@10.1.3.80:type: producer
eq_tapq:replication_ns_1@10.1.3.80:vb_filter: { 3, 27, 49 }
eq_tapq:replication_ns_1@10.1.3.80:vb_filters: 3
10.1.3.80
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 12
ep_tap_deletes: 3616
ep_tap_fg_fetched: 586320
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 240071
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 601341
ep_tap_total_queue: 0
eq_tapq:anon_437:connected: true
eq_tapq:anon_437:created: 243
eq_tapq:anon_437:num_checkpoint_end: 3
eq_tapq:anon_437:num_checkpoint_end_failed: 0
eq_tapq:anon_437:num_checkpoint_start: 4
eq_tapq:anon_437:num_checkpoint_start_failed: 0
eq_tapq:anon_437:num_delete: 12
eq_tapq:anon_437:num_delete_failed: 0
eq_tapq:anon_437:num_flush: 0
eq_tapq:anon_437:num_flush_failed: 0
eq_tapq:anon_437:num_mutation: 3084
eq_tapq:anon_437:num_mutation_failed: 0
eq_tapq:anon_437:num_opaque: 4
eq_tapq:anon_437:num_opaque_failed: 0
eq_tapq:anon_437:num_unknown: 0
eq_tapq:anon_437:num_vbucket_set: 0
eq_tapq:anon_437:num_vbucket_set_failed: 0
eq_tapq:anon_437:pending_disconnect: false
eq_tapq:anon_437:reserved: 0
eq_tapq:anon_437:supports_ack: true
eq_tapq:anon_437:type: consumer
eq_tapq:anon_451:connected: true
eq_tapq:anon_451:created: 245
eq_tapq:anon_451:num_checkpoint_end: 2
eq_tapq:anon_451:num_checkpoint_end_failed: 0
eq_tapq:anon_451:num_checkpoint_start: 4
eq_tapq:anon_451:num_checkpoint_start_failed: 0
eq_tapq:anon_451:num_delete: 29
eq_tapq:anon_451:num_delete_failed: 0
eq_tapq:anon_451:num_flush: 0
eq_tapq:anon_451:num_flush_failed: 0
eq_tapq:anon_451:num_mutation: 6132
eq_tapq:anon_451:num_mutation_failed: 0
eq_tapq:anon_451:num_opaque: 6
eq_tapq:anon_451:num_opaque_failed: 0
eq_tapq:anon_451:num_unknown: 0
eq_tapq:anon_451:num_vbucket_set: 0
eq_tapq:anon_451:num_vbucket_set_failed: 0
eq_tapq:anon_451:pending_disconnect: false
eq_tapq:anon_451:reserved: 0
eq_tapq:anon_451:supports_ack: true
eq_tapq:anon_451:type: consumer
eq_tapq:anon_489:connected: true
eq_tapq:anon_489:created: 255
eq_tapq:anon_489:num_checkpoint_end: 3
eq_tapq:anon_489:num_checkpoint_end_failed: 0
eq_tapq:anon_489:num_checkpoint_start: 3
eq_tapq:anon_489:num_checkpoint_start_failed: 0
eq_tapq:anon_489:num_delete: 0
eq_tapq:anon_489:num_delete_failed: 0
eq_tapq:anon_489:num_flush: 0
eq_tapq:anon_489:num_flush_failed: 0
eq_tapq:anon_489:num_mutation: 0
eq_tapq:anon_489:num_mutation_failed: 0
eq_tapq:anon_489:num_opaque: 2
eq_tapq:anon_489:num_opaque_failed: 0
eq_tapq:anon_489:num_unknown: 0
eq_tapq:anon_489:num_vbucket_set: 0
eq_tapq:anon_489:num_vbucket_set_failed: 0
eq_tapq:anon_489:pending_disconnect: false
eq_tapq:anon_489:reserved: 0
eq_tapq:anon_489:supports_ack: true
eq_tapq:anon_489:type: consumer
eq_tapq:anon_501:connected: true
eq_tapq:anon_501:created: 257
eq_tapq:anon_501:num_checkpoint_end: 4
eq_tapq:anon_501:num_checkpoint_end_failed: 0
eq_tapq:anon_501:num_checkpoint_start: 7
eq_tapq:anon_501:num_checkpoint_start_failed: 0
eq_tapq:anon_501:num_delete: 28
eq_tapq:anon_501:num_delete_failed: 0
eq_tapq:anon_501:num_flush: 0
eq_tapq:anon_501:num_flush_failed: 0
eq_tapq:anon_501:num_mutation: 8504
eq_tapq:anon_501:num_mutation_failed: 0
eq_tapq:anon_501:num_opaque: 8
eq_tapq:anon_501:num_opaque_failed: 0
eq_tapq:anon_501:num_unknown: 0
eq_tapq:anon_501:num_vbucket_set: 0
eq_tapq:anon_501:num_vbucket_set_failed: 0
eq_tapq:anon_501:pending_disconnect: false
eq_tapq:anon_501:reserved: 0
eq_tapq:anon_501:supports_ack: true
eq_tapq:anon_501:type: consumer
eq_tapq:anon_506:connected: true
eq_tapq:anon_506:created: 259
eq_tapq:anon_506:num_checkpoint_end: 2
eq_tapq:anon_506:num_checkpoint_end_failed: 0
eq_tapq:anon_506:num_checkpoint_start: 2
eq_tapq:anon_506:num_checkpoint_start_failed: 0
eq_tapq:anon_506:num_delete: 0
eq_tapq:anon_506:num_delete_failed: 0
eq_tapq:anon_506:num_flush: 0
eq_tapq:anon_506:num_flush_failed: 0
eq_tapq:anon_506:num_mutation: 0
eq_tapq:anon_506:num_mutation_failed: 0
eq_tapq:anon_506:num_opaque: 2
eq_tapq:anon_506:num_opaque_failed: 0
eq_tapq:anon_506:num_unknown: 0
eq_tapq:anon_506:num_vbucket_set: 0
eq_tapq:anon_506:num_vbucket_set_failed: 0
eq_tapq:anon_506:pending_disconnect: false
eq_tapq:anon_506:reserved: 0
eq_tapq:anon_506:supports_ack: true
eq_tapq:anon_506:type: consumer
eq_tapq:anon_517:connected: true
eq_tapq:anon_517:created: 262
eq_tapq:anon_517:num_checkpoint_end: 4
eq_tapq:anon_517:num_checkpoint_end_failed: 0
eq_tapq:anon_517:num_checkpoint_start: 8
eq_tapq:anon_517:num_checkpoint_start_failed: 0
eq_tapq:anon_517:num_delete: 73
eq_tapq:anon_517:num_delete_failed: 0
eq_tapq:anon_517:num_flush: 0
eq_tapq:anon_517:num_flush_failed: 0
eq_tapq:anon_517:num_mutation: 12428
eq_tapq:anon_517:num_mutation_failed: 0
eq_tapq:anon_517:num_opaque: 8
eq_tapq:anon_517:num_opaque_failed: 0
eq_tapq:anon_517:num_unknown: 0
eq_tapq:anon_517:num_vbucket_set: 0
eq_tapq:anon_517:num_vbucket_set_failed: 0
eq_tapq:anon_517:pending_disconnect: false
eq_tapq:anon_517:reserved: 0
eq_tapq:anon_517:supports_ack: true
eq_tapq:anon_517:type: consumer
eq_tapq:replication_ns_1@10.1.3.74:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_seqno: 15732
eq_tapq:replication_ns_1@10.1.3.74:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.74:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.74:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.74:complete: false
eq_tapq:replication_ns_1@10.1.3.74:connected: true
eq_tapq:replication_ns_1@10.1.3.74:created: 251
eq_tapq:replication_ns_1@10.1.3.74:disconnects: 3
eq_tapq:replication_ns_1@10.1.3.74:empty: true
eq_tapq:replication_ns_1@10.1.3.74:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.74:has_item: false
eq_tapq:replication_ns_1@10.1.3.74:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.74:idle: true
eq_tapq:replication_ns_1@10.1.3.74:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.74:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.74:paused: 1
eq_tapq:replication_ns_1@10.1.3.74:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:qlen: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_drain: 15678
eq_tapq:replication_ns_1@10.1.3.74:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.74:rec_fetched: 15711
eq_tapq:replication_ns_1@10.1.3.74:reconnects: 3
eq_tapq:replication_ns_1@10.1.3.74:recv_ack_seqno: 15731
eq_tapq:replication_ns_1@10.1.3.74:reserved: 1
eq_tapq:replication_ns_1@10.1.3.74:seqno_ack_requested: 15731
eq_tapq:replication_ns_1@10.1.3.74:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.74:suspended: false
eq_tapq:replication_ns_1@10.1.3.74:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.74:type: producer
eq_tapq:replication_ns_1@10.1.3.74:vb_filter: { 44, 53, 55, 56 }
eq_tapq:replication_ns_1@10.1.3.74:vb_filters: 4
eq_tapq:replication_ns_1@10.1.3.75:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_seqno: 69245
eq_tapq:replication_ns_1@10.1.3.75:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.75:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.75:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.75:complete: false
eq_tapq:replication_ns_1@10.1.3.75:connected: true
eq_tapq:replication_ns_1@10.1.3.75:created: 132
eq_tapq:replication_ns_1@10.1.3.75:disconnects: 25
eq_tapq:replication_ns_1@10.1.3.75:empty: true
eq_tapq:replication_ns_1@10.1.3.75:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.75:has_item: false
eq_tapq:replication_ns_1@10.1.3.75:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.75:idle: true
eq_tapq:replication_ns_1@10.1.3.75:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.75:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.75:paused: 1
eq_tapq:replication_ns_1@10.1.3.75:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:qlen: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_drain: 68170
eq_tapq:replication_ns_1@10.1.3.75:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.75:rec_fetched: 69148
eq_tapq:replication_ns_1@10.1.3.75:reconnects: 25
eq_tapq:replication_ns_1@10.1.3.75:recv_ack_seqno: 69244
eq_tapq:replication_ns_1@10.1.3.75:reserved: 1
eq_tapq:replication_ns_1@10.1.3.75:seqno_ack_requested: 69244
eq_tapq:replication_ns_1@10.1.3.75:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.75:suspended: false
eq_tapq:replication_ns_1@10.1.3.75:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:total_noops: 63
eq_tapq:replication_ns_1@10.1.3.75:type: producer
eq_tapq:replication_ns_1@10.1.3.75:vb_filter: { 8, 27, 57 }
eq_tapq:replication_ns_1@10.1.3.75:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.76:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_seqno: 57369
eq_tapq:replication_ns_1@10.1.3.76:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.76:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.76:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.76:complete: false
eq_tapq:replication_ns_1@10.1.3.76:connected: true
eq_tapq:replication_ns_1@10.1.3.76:created: 102
eq_tapq:replication_ns_1@10.1.3.76:disconnects: 24
eq_tapq:replication_ns_1@10.1.3.76:empty: true
eq_tapq:replication_ns_1@10.1.3.76:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.76:has_item: false
eq_tapq:replication_ns_1@10.1.3.76:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.76:idle: true
eq_tapq:replication_ns_1@10.1.3.76:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.76:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.76:paused: 1
eq_tapq:replication_ns_1@10.1.3.76:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:qlen: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_drain: 54155
eq_tapq:replication_ns_1@10.1.3.76:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.76:rec_fetched: 57284
eq_tapq:replication_ns_1@10.1.3.76:reconnects: 24
eq_tapq:replication_ns_1@10.1.3.76:recv_ack_seqno: 57368
eq_tapq:replication_ns_1@10.1.3.76:reserved: 1
eq_tapq:replication_ns_1@10.1.3.76:seqno_ack_requested: 57368
eq_tapq:replication_ns_1@10.1.3.76:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.76:suspended: false
eq_tapq:replication_ns_1@10.1.3.76:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:total_noops: 64
eq_tapq:replication_ns_1@10.1.3.76:type: producer
eq_tapq:replication_ns_1@10.1.3.76:vb_filter: { 45, 58 }
eq_tapq:replication_ns_1@10.1.3.76:vb_filters: 2
eq_tapq:replication_ns_1@10.1.3.77:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_seqno: 12646
eq_tapq:replication_ns_1@10.1.3.77:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.77:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.77:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.77:complete: false
eq_tapq:replication_ns_1@10.1.3.77:connected: true
eq_tapq:replication_ns_1@10.1.3.77:created: 246
eq_tapq:replication_ns_1@10.1.3.77:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.77:empty: true
eq_tapq:replication_ns_1@10.1.3.77:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.77:has_item: false
eq_tapq:replication_ns_1@10.1.3.77:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.77:idle: true
eq_tapq:replication_ns_1@10.1.3.77:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.77:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.77:paused: 1
eq_tapq:replication_ns_1@10.1.3.77:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:qlen: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_drain: 11121
eq_tapq:replication_ns_1@10.1.3.77:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.77:rec_fetched: 12629
eq_tapq:replication_ns_1@10.1.3.77:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.77:recv_ack_seqno: 12645
eq_tapq:replication_ns_1@10.1.3.77:reserved: 1
eq_tapq:replication_ns_1@10.1.3.77:seqno_ack_requested: 12645
eq_tapq:replication_ns_1@10.1.3.77:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.77:suspended: false
eq_tapq:replication_ns_1@10.1.3.77:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:total_noops: 58
eq_tapq:replication_ns_1@10.1.3.77:type: producer
eq_tapq:replication_ns_1@10.1.3.77:vb_filter: { 18, 54, 59 }
eq_tapq:replication_ns_1@10.1.3.77:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.78:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_seqno: 10937
eq_tapq:replication_ns_1@10.1.3.78:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.78:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.78:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.78:complete: false
eq_tapq:replication_ns_1@10.1.3.78:connected: true
eq_tapq:replication_ns_1@10.1.3.78:created: 245
eq_tapq:replication_ns_1@10.1.3.78:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.78:empty: true
eq_tapq:replication_ns_1@10.1.3.78:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.78:has_item: false
eq_tapq:replication_ns_1@10.1.3.78:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.78:idle: true
eq_tapq:replication_ns_1@10.1.3.78:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.78:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.78:paused: 1
eq_tapq:replication_ns_1@10.1.3.78:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:qlen: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_drain: 9737
eq_tapq:replication_ns_1@10.1.3.78:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.78:rec_fetched: 10922
eq_tapq:replication_ns_1@10.1.3.78:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.78:recv_ack_seqno: 10936
eq_tapq:replication_ns_1@10.1.3.78:reserved: 1
eq_tapq:replication_ns_1@10.1.3.78:seqno_ack_requested: 10936
eq_tapq:replication_ns_1@10.1.3.78:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.78:suspended: false
eq_tapq:replication_ns_1@10.1.3.78:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:total_noops: 58
eq_tapq:replication_ns_1@10.1.3.78:type: producer
eq_tapq:replication_ns_1@10.1.3.78:vb_filter: { 36, 60, 61 }
eq_tapq:replication_ns_1@10.1.3.78:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.79:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_seqno: 85235
eq_tapq:replication_ns_1@10.1.3.79:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.79:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.79:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.79:complete: false
eq_tapq:replication_ns_1@10.1.3.79:connected: true
eq_tapq:replication_ns_1@10.1.3.79:created: 102
eq_tapq:replication_ns_1@10.1.3.79:disconnects: 30
eq_tapq:replication_ns_1@10.1.3.79:empty: true
eq_tapq:replication_ns_1@10.1.3.79:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.79:has_item: false
eq_tapq:replication_ns_1@10.1.3.79:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.79:idle: true
eq_tapq:replication_ns_1@10.1.3.79:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.79:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.79:paused: 1
eq_tapq:replication_ns_1@10.1.3.79:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:qlen: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_drain: 81210
eq_tapq:replication_ns_1@10.1.3.79:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.79:rec_fetched: 85119
eq_tapq:replication_ns_1@10.1.3.79:reconnects: 30
eq_tapq:replication_ns_1@10.1.3.79:recv_ack_seqno: 85234
eq_tapq:replication_ns_1@10.1.3.79:reserved: 1
eq_tapq:replication_ns_1@10.1.3.79:seqno_ack_requested: 85234
eq_tapq:replication_ns_1@10.1.3.79:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.79:suspended: false
eq_tapq:replication_ns_1@10.1.3.79:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:total_noops: 64
eq_tapq:replication_ns_1@10.1.3.79:type: producer
eq_tapq:replication_ns_1@10.1.3.79:vb_filter: { 9, 62, 63 }
eq_tapq:replication_ns_1@10.1.3.79:vb_filters: 3
10.1.3.78
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 12
ep_tap_deletes: 432
ep_tap_fg_fetched: 80432
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 68432
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 84147
ep_tap_total_queue: 0
eq_tapq:anon_105:connected: true
eq_tapq:anon_105:created: 62
eq_tapq:anon_105:num_checkpoint_end: 3
eq_tapq:anon_105:num_checkpoint_end_failed: 0
eq_tapq:anon_105:num_checkpoint_start: 4
eq_tapq:anon_105:num_checkpoint_start_failed: 0
eq_tapq:anon_105:num_delete: 18
eq_tapq:anon_105:num_delete_failed: 0
eq_tapq:anon_105:num_flush: 0
eq_tapq:anon_105:num_flush_failed: 0
eq_tapq:anon_105:num_mutation: 2970
eq_tapq:anon_105:num_mutation_failed: 0
eq_tapq:anon_105:num_opaque: 4
eq_tapq:anon_105:num_opaque_failed: 0
eq_tapq:anon_105:num_unknown: 0
eq_tapq:anon_105:num_vbucket_set: 0
eq_tapq:anon_105:num_vbucket_set_failed: 0
eq_tapq:anon_105:pending_disconnect: false
eq_tapq:anon_105:reserved: 0
eq_tapq:anon_105:supports_ack: true
eq_tapq:anon_105:type: consumer
eq_tapq:anon_107:connected: true
eq_tapq:anon_107:created: 89
eq_tapq:anon_107:num_checkpoint_end: 2
eq_tapq:anon_107:num_checkpoint_end_failed: 0
eq_tapq:anon_107:num_checkpoint_start: 4
eq_tapq:anon_107:num_checkpoint_start_failed: 0
eq_tapq:anon_107:num_delete: 32
eq_tapq:anon_107:num_delete_failed: 0
eq_tapq:anon_107:num_flush: 0
eq_tapq:anon_107:num_flush_failed: 0
eq_tapq:anon_107:num_mutation: 6262
eq_tapq:anon_107:num_mutation_failed: 0
eq_tapq:anon_107:num_opaque: 6
eq_tapq:anon_107:num_opaque_failed: 0
eq_tapq:anon_107:num_unknown: 0
eq_tapq:anon_107:num_vbucket_set: 0
eq_tapq:anon_107:num_vbucket_set_failed: 0
eq_tapq:anon_107:pending_disconnect: false
eq_tapq:anon_107:reserved: 0
eq_tapq:anon_107:supports_ack: true
eq_tapq:anon_107:type: consumer
eq_tapq:anon_109:connected: true
eq_tapq:anon_109:created: 92
eq_tapq:anon_109:num_checkpoint_end: 2
eq_tapq:anon_109:num_checkpoint_end_failed: 0
eq_tapq:anon_109:num_checkpoint_start: 4
eq_tapq:anon_109:num_checkpoint_start_failed: 0
eq_tapq:anon_109:num_delete: 39
eq_tapq:anon_109:num_delete_failed: 0
eq_tapq:anon_109:num_flush: 0
eq_tapq:anon_109:num_flush_failed: 0
eq_tapq:anon_109:num_mutation: 6254
eq_tapq:anon_109:num_mutation_failed: 0
eq_tapq:anon_109:num_opaque: 6
eq_tapq:anon_109:num_opaque_failed: 0
eq_tapq:anon_109:num_unknown: 0
eq_tapq:anon_109:num_vbucket_set: 0
eq_tapq:anon_109:num_vbucket_set_failed: 0
eq_tapq:anon_109:pending_disconnect: false
eq_tapq:anon_109:reserved: 0
eq_tapq:anon_109:supports_ack: true
eq_tapq:anon_109:type: consumer
eq_tapq:anon_79:connected: true
eq_tapq:anon_79:created: 51
eq_tapq:anon_79:num_checkpoint_end: 3
eq_tapq:anon_79:num_checkpoint_end_failed: 0
eq_tapq:anon_79:num_checkpoint_start: 4
eq_tapq:anon_79:num_checkpoint_start_failed: 0
eq_tapq:anon_79:num_delete: 262
eq_tapq:anon_79:num_delete_failed: 0
eq_tapq:anon_79:num_flush: 0
eq_tapq:anon_79:num_flush_failed: 0
eq_tapq:anon_79:num_mutation: 4478
eq_tapq:anon_79:num_mutation_failed: 0
eq_tapq:anon_79:num_opaque: 6
eq_tapq:anon_79:num_opaque_failed: 0
eq_tapq:anon_79:num_unknown: 0
eq_tapq:anon_79:num_vbucket_set: 0
eq_tapq:anon_79:num_vbucket_set_failed: 0
eq_tapq:anon_79:pending_disconnect: false
eq_tapq:anon_79:reserved: 0
eq_tapq:anon_79:supports_ack: true
eq_tapq:anon_79:type: consumer
eq_tapq:anon_90:connected: true
eq_tapq:anon_90:created: 56
eq_tapq:anon_90:num_checkpoint_end: 2
eq_tapq:anon_90:num_checkpoint_end_failed: 0
eq_tapq:anon_90:num_checkpoint_start: 2
eq_tapq:anon_90:num_checkpoint_start_failed: 0
eq_tapq:anon_90:num_delete: 1509
eq_tapq:anon_90:num_delete_failed: 0
eq_tapq:anon_90:num_flush: 0
eq_tapq:anon_90:num_flush_failed: 0
eq_tapq:anon_90:num_mutation: 0
eq_tapq:anon_90:num_mutation_failed: 0
eq_tapq:anon_90:num_opaque: 6
eq_tapq:anon_90:num_opaque_failed: 0
eq_tapq:anon_90:num_unknown: 0
eq_tapq:anon_90:num_vbucket_set: 0
eq_tapq:anon_90:num_vbucket_set_failed: 0
eq_tapq:anon_90:pending_disconnect: false
eq_tapq:anon_90:reserved: 0
eq_tapq:anon_90:supports_ack: true
eq_tapq:anon_90:type: consumer
eq_tapq:anon_97:connected: true
eq_tapq:anon_97:created: 58
eq_tapq:anon_97:num_checkpoint_end: 4
eq_tapq:anon_97:num_checkpoint_end_failed: 0
eq_tapq:anon_97:num_checkpoint_start: 8
eq_tapq:anon_97:num_checkpoint_start_failed: 0
eq_tapq:anon_97:num_delete: 65
eq_tapq:anon_97:num_delete_failed: 0
eq_tapq:anon_97:num_flush: 0
eq_tapq:anon_97:num_flush_failed: 0
eq_tapq:anon_97:num_mutation: 12204
eq_tapq:anon_97:num_mutation_failed: 0
eq_tapq:anon_97:num_opaque: 8
eq_tapq:anon_97:num_opaque_failed: 0
eq_tapq:anon_97:num_unknown: 0
eq_tapq:anon_97:num_vbucket_set: 0
eq_tapq:anon_97:num_vbucket_set_failed: 0
eq_tapq:anon_97:pending_disconnect: false
eq_tapq:anon_97:reserved: 0
eq_tapq:anon_97:supports_ack: true
eq_tapq:anon_97:type: consumer
eq_tapq:replication_ns_1@10.1.3.74:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_seqno: 9400
eq_tapq:replication_ns_1@10.1.3.74:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.74:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.74:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.74:complete: false
eq_tapq:replication_ns_1@10.1.3.74:connected: true
eq_tapq:replication_ns_1@10.1.3.74:created: 54
eq_tapq:replication_ns_1@10.1.3.74:disconnects: 1
eq_tapq:replication_ns_1@10.1.3.74:empty: true
eq_tapq:replication_ns_1@10.1.3.74:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.74:has_item: false
eq_tapq:replication_ns_1@10.1.3.74:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.74:idle: true
eq_tapq:replication_ns_1@10.1.3.74:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.74:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.74:paused: 1
eq_tapq:replication_ns_1@10.1.3.74:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:qlen: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_drain: 9372
eq_tapq:replication_ns_1@10.1.3.74:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.74:rec_fetched: 9389
eq_tapq:replication_ns_1@10.1.3.74:reconnects: 1
eq_tapq:replication_ns_1@10.1.3.74:recv_ack_seqno: 9399
eq_tapq:replication_ns_1@10.1.3.74:reserved: 1
eq_tapq:replication_ns_1@10.1.3.74:seqno_ack_requested: 9399
eq_tapq:replication_ns_1@10.1.3.74:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.74:suspended: false
eq_tapq:replication_ns_1@10.1.3.74:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.74:type: producer
eq_tapq:replication_ns_1@10.1.3.74:vb_filter: { 37, 51 }
eq_tapq:replication_ns_1@10.1.3.74:vb_filters: 2
eq_tapq:replication_ns_1@10.1.3.75:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_seqno: 9283
eq_tapq:replication_ns_1@10.1.3.75:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.75:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.75:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.75:complete: false
eq_tapq:replication_ns_1@10.1.3.75:connected: true
eq_tapq:replication_ns_1@10.1.3.75:created: 92
eq_tapq:replication_ns_1@10.1.3.75:empty: true
eq_tapq:replication_ns_1@10.1.3.75:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.75:has_item: false
eq_tapq:replication_ns_1@10.1.3.75:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.75:idle: true
eq_tapq:replication_ns_1@10.1.3.75:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.75:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.75:paused: 1
eq_tapq:replication_ns_1@10.1.3.75:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:qlen: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_drain: 9257
eq_tapq:replication_ns_1@10.1.3.75:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.75:rec_fetched: 9274
eq_tapq:replication_ns_1@10.1.3.75:recv_ack_seqno: 9282
eq_tapq:replication_ns_1@10.1.3.75:reserved: 1
eq_tapq:replication_ns_1@10.1.3.75:seqno_ack_requested: 9282
eq_tapq:replication_ns_1@10.1.3.75:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.75:suspended: false
eq_tapq:replication_ns_1@10.1.3.75:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:total_noops: 55
eq_tapq:replication_ns_1@10.1.3.75:type: producer
eq_tapq:replication_ns_1@10.1.3.75:vb_filter: { 33, 38, 60 }
eq_tapq:replication_ns_1@10.1.3.75:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.76:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_seqno: 9393
eq_tapq:replication_ns_1@10.1.3.76:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.76:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.76:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.76:complete: false
eq_tapq:replication_ns_1@10.1.3.76:connected: true
eq_tapq:replication_ns_1@10.1.3.76:created: 52
eq_tapq:replication_ns_1@10.1.3.76:disconnects: 1
eq_tapq:replication_ns_1@10.1.3.76:empty: true
eq_tapq:replication_ns_1@10.1.3.76:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.76:has_item: false
eq_tapq:replication_ns_1@10.1.3.76:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.76:idle: true
eq_tapq:replication_ns_1@10.1.3.76:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.76:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.76:paused: 1
eq_tapq:replication_ns_1@10.1.3.76:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:qlen: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_drain: 9363
eq_tapq:replication_ns_1@10.1.3.76:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.76:rec_fetched: 9382
eq_tapq:replication_ns_1@10.1.3.76:reconnects: 1
eq_tapq:replication_ns_1@10.1.3.76:recv_ack_seqno: 9392
eq_tapq:replication_ns_1@10.1.3.76:reserved: 1
eq_tapq:replication_ns_1@10.1.3.76:seqno_ack_requested: 9392
eq_tapq:replication_ns_1@10.1.3.76:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.76:suspended: false
eq_tapq:replication_ns_1@10.1.3.76:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.76:type: producer
eq_tapq:replication_ns_1@10.1.3.76:vb_filter: { 16, 34, 39 }
eq_tapq:replication_ns_1@10.1.3.76:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.77:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_seqno: 9308
eq_tapq:replication_ns_1@10.1.3.77:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.77:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.77:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.77:complete: false
eq_tapq:replication_ns_1@10.1.3.77:connected: true
eq_tapq:replication_ns_1@10.1.3.77:created: 90
eq_tapq:replication_ns_1@10.1.3.77:empty: true
eq_tapq:replication_ns_1@10.1.3.77:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.77:has_item: false
eq_tapq:replication_ns_1@10.1.3.77:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.77:idle: true
eq_tapq:replication_ns_1@10.1.3.77:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.77:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.77:paused: 1
eq_tapq:replication_ns_1@10.1.3.77:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:qlen: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_drain: 9282
eq_tapq:replication_ns_1@10.1.3.77:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.77:rec_fetched: 9299
eq_tapq:replication_ns_1@10.1.3.77:recv_ack_seqno: 9307
eq_tapq:replication_ns_1@10.1.3.77:reserved: 1
eq_tapq:replication_ns_1@10.1.3.77:seqno_ack_requested: 9307
eq_tapq:replication_ns_1@10.1.3.77:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.77:suspended: false
eq_tapq:replication_ns_1@10.1.3.77:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:total_noops: 55
eq_tapq:replication_ns_1@10.1.3.77:type: producer
eq_tapq:replication_ns_1@10.1.3.77:vb_filter: { 40, 41, 61 }
eq_tapq:replication_ns_1@10.1.3.77:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.79:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_seqno: 14987
eq_tapq:replication_ns_1@10.1.3.79:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.79:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.79:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.79:complete: false
eq_tapq:replication_ns_1@10.1.3.79:connected: true
eq_tapq:replication_ns_1@10.1.3.79:created: 49
eq_tapq:replication_ns_1@10.1.3.79:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.79:empty: true
eq_tapq:replication_ns_1@10.1.3.79:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.79:has_item: false
eq_tapq:replication_ns_1@10.1.3.79:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.79:idle: true
eq_tapq:replication_ns_1@10.1.3.79:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.79:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.79:paused: 1
eq_tapq:replication_ns_1@10.1.3.79:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:qlen: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_drain: 13359
eq_tapq:replication_ns_1@10.1.3.79:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.79:rec_fetched: 14970
eq_tapq:replication_ns_1@10.1.3.79:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.79:recv_ack_seqno: 14986
eq_tapq:replication_ns_1@10.1.3.79:reserved: 1
eq_tapq:replication_ns_1@10.1.3.79:seqno_ack_requested: 14986
eq_tapq:replication_ns_1@10.1.3.79:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.79:suspended: false
eq_tapq:replication_ns_1@10.1.3.79:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.79:type: producer
eq_tapq:replication_ns_1@10.1.3.79:vb_filter: { 25, 42, 43 }
eq_tapq:replication_ns_1@10.1.3.79:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.80:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_seqno: 19398
eq_tapq:replication_ns_1@10.1.3.80:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.80:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.80:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.80:complete: false
eq_tapq:replication_ns_1@10.1.3.80:connected: true
eq_tapq:replication_ns_1@10.1.3.80:created: 37
eq_tapq:replication_ns_1@10.1.3.80:disconnects: 3
eq_tapq:replication_ns_1@10.1.3.80:empty: true
eq_tapq:replication_ns_1@10.1.3.80:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.80:has_item: false
eq_tapq:replication_ns_1@10.1.3.80:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.80:idle: true
eq_tapq:replication_ns_1@10.1.3.80:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.80:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.80:paused: 1
eq_tapq:replication_ns_1@10.1.3.80:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:qlen: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_drain: 17799
eq_tapq:replication_ns_1@10.1.3.80:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.80:rec_fetched: 19375
eq_tapq:replication_ns_1@10.1.3.80:reconnects: 3
eq_tapq:replication_ns_1@10.1.3.80:recv_ack_seqno: 19397
eq_tapq:replication_ns_1@10.1.3.80:reserved: 1
eq_tapq:replication_ns_1@10.1.3.80:seqno_ack_requested: 19397
eq_tapq:replication_ns_1@10.1.3.80:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.80:suspended: false
eq_tapq:replication_ns_1@10.1.3.80:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.80:type: producer
eq_tapq:replication_ns_1@10.1.3.80:vb_filter: { 5, 44, 45, 52 }
eq_tapq:replication_ns_1@10.1.3.80:vb_filters: 4
10.1.3.75
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 12
ep_tap_deletes: 3531
ep_tap_fg_fetched: 543366
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 254729
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 552693
ep_tap_total_queue: 0
eq_tapq:anon_482:connected: true
eq_tapq:anon_482:created: 253
eq_tapq:anon_482:num_checkpoint_end: 3
eq_tapq:anon_482:num_checkpoint_end_failed: 0
eq_tapq:anon_482:num_checkpoint_start: 4
eq_tapq:anon_482:num_checkpoint_start_failed: 0
eq_tapq:anon_482:num_delete: 12
eq_tapq:anon_482:num_delete_failed: 0
eq_tapq:anon_482:num_flush: 0
eq_tapq:anon_482:num_flush_failed: 0
eq_tapq:anon_482:num_mutation: 2968
eq_tapq:anon_482:num_mutation_failed: 0
eq_tapq:anon_482:num_opaque: 4
eq_tapq:anon_482:num_opaque_failed: 0
eq_tapq:anon_482:num_unknown: 0
eq_tapq:anon_482:num_vbucket_set: 0
eq_tapq:anon_482:num_vbucket_set_failed: 0
eq_tapq:anon_482:pending_disconnect: false
eq_tapq:anon_482:reserved: 0
eq_tapq:anon_482:supports_ack: true
eq_tapq:anon_482:type: consumer
eq_tapq:anon_487:connected: true
eq_tapq:anon_487:created: 254
eq_tapq:anon_487:num_checkpoint_end: 3
eq_tapq:anon_487:num_checkpoint_end_failed: 0
eq_tapq:anon_487:num_checkpoint_start: 3
eq_tapq:anon_487:num_checkpoint_start_failed: 0
eq_tapq:anon_487:num_delete: 0
eq_tapq:anon_487:num_delete_failed: 0
eq_tapq:anon_487:num_flush: 0
eq_tapq:anon_487:num_flush_failed: 0
eq_tapq:anon_487:num_mutation: 0
eq_tapq:anon_487:num_mutation_failed: 0
eq_tapq:anon_487:num_opaque: 2
eq_tapq:anon_487:num_opaque_failed: 0
eq_tapq:anon_487:num_unknown: 0
eq_tapq:anon_487:num_vbucket_set: 0
eq_tapq:anon_487:num_vbucket_set_failed: 0
eq_tapq:anon_487:pending_disconnect: false
eq_tapq:anon_487:reserved: 0
eq_tapq:anon_487:supports_ack: true
eq_tapq:anon_487:type: consumer
eq_tapq:anon_491:connected: true
eq_tapq:anon_491:created: 256
eq_tapq:anon_491:num_checkpoint_end: 4
eq_tapq:anon_491:num_checkpoint_end_failed: 0
eq_tapq:anon_491:num_checkpoint_start: 7
eq_tapq:anon_491:num_checkpoint_start_failed: 0
eq_tapq:anon_491:num_delete: 45
eq_tapq:anon_491:num_delete_failed: 0
eq_tapq:anon_491:num_flush: 0
eq_tapq:anon_491:num_flush_failed: 0
eq_tapq:anon_491:num_mutation: 9174
eq_tapq:anon_491:num_mutation_failed: 0
eq_tapq:anon_491:num_opaque: 8
eq_tapq:anon_491:num_opaque_failed: 0
eq_tapq:anon_491:num_unknown: 0
eq_tapq:anon_491:num_vbucket_set: 0
eq_tapq:anon_491:num_vbucket_set_failed: 0
eq_tapq:anon_491:pending_disconnect: false
eq_tapq:anon_491:reserved: 0
eq_tapq:anon_491:supports_ack: true
eq_tapq:anon_491:type: consumer
eq_tapq:anon_498:connected: true
eq_tapq:anon_498:created: 259
eq_tapq:anon_498:num_checkpoint_end: 4
eq_tapq:anon_498:num_checkpoint_end_failed: 0
eq_tapq:anon_498:num_checkpoint_start: 4
eq_tapq:anon_498:num_checkpoint_start_failed: 0
eq_tapq:anon_498:num_delete: 0
eq_tapq:anon_498:num_delete_failed: 0
eq_tapq:anon_498:num_flush: 0
eq_tapq:anon_498:num_flush_failed: 0
eq_tapq:anon_498:num_mutation: 0
eq_tapq:anon_498:num_mutation_failed: 0
eq_tapq:anon_498:num_opaque: 2
eq_tapq:anon_498:num_opaque_failed: 0
eq_tapq:anon_498:num_unknown: 0
eq_tapq:anon_498:num_vbucket_set: 0
eq_tapq:anon_498:num_vbucket_set_failed: 0
eq_tapq:anon_498:pending_disconnect: false
eq_tapq:anon_498:reserved: 0
eq_tapq:anon_498:supports_ack: true
eq_tapq:anon_498:type: consumer
eq_tapq:anon_508:connected: true
eq_tapq:anon_508:created: 262
eq_tapq:anon_508:num_checkpoint_end: 2
eq_tapq:anon_508:num_checkpoint_end_failed: 0
eq_tapq:anon_508:num_checkpoint_start: 3
eq_tapq:anon_508:num_checkpoint_start_failed: 0
eq_tapq:anon_508:num_delete: 23
eq_tapq:anon_508:num_delete_failed: 0
eq_tapq:anon_508:num_flush: 0
eq_tapq:anon_508:num_flush_failed: 0
eq_tapq:anon_508:num_mutation: 3052
eq_tapq:anon_508:num_mutation_failed: 0
eq_tapq:anon_508:num_opaque: 3
eq_tapq:anon_508:num_opaque_failed: 0
eq_tapq:anon_508:num_unknown: 0
eq_tapq:anon_508:num_vbucket_set: 0
eq_tapq:anon_508:num_vbucket_set_failed: 0
eq_tapq:anon_508:pending_disconnect: false
eq_tapq:anon_508:reserved: 0
eq_tapq:anon_508:supports_ack: true
eq_tapq:anon_508:type: consumer
eq_tapq:anon_510:connected: true
eq_tapq:anon_510:created: 290
eq_tapq:anon_510:num_checkpoint_end: 3
eq_tapq:anon_510:num_checkpoint_end_failed: 0
eq_tapq:anon_510:num_checkpoint_start: 6
eq_tapq:anon_510:num_checkpoint_start_failed: 0
eq_tapq:anon_510:num_delete: 49
eq_tapq:anon_510:num_delete_failed: 0
eq_tapq:anon_510:num_flush: 0
eq_tapq:anon_510:num_flush_failed: 0
eq_tapq:anon_510:num_mutation: 9208
eq_tapq:anon_510:num_mutation_failed: 0
eq_tapq:anon_510:num_opaque: 8
eq_tapq:anon_510:num_opaque_failed: 0
eq_tapq:anon_510:num_unknown: 0
eq_tapq:anon_510:num_vbucket_set: 0
eq_tapq:anon_510:num_vbucket_set_failed: 0
eq_tapq:anon_510:pending_disconnect: false
eq_tapq:anon_510:reserved: 0
eq_tapq:anon_510:supports_ack: true
eq_tapq:anon_510:type: consumer
eq_tapq:replication_ns_1@10.1.3.74:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.74:ack_seqno: 12838
eq_tapq:replication_ns_1@10.1.3.74:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.74:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.74:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.74:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.74:complete: false
eq_tapq:replication_ns_1@10.1.3.74:connected: true
eq_tapq:replication_ns_1@10.1.3.74:created: 237
eq_tapq:replication_ns_1@10.1.3.74:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.74:empty: true
eq_tapq:replication_ns_1@10.1.3.74:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.74:has_item: false
eq_tapq:replication_ns_1@10.1.3.74:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.74:idle: true
eq_tapq:replication_ns_1@10.1.3.74:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.74:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.74:paused: 1
eq_tapq:replication_ns_1@10.1.3.74:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.74:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.74:qlen: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_drain: 12799
eq_tapq:replication_ns_1@10.1.3.74:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.74:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.74:rec_fetched: 12823
eq_tapq:replication_ns_1@10.1.3.74:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.74:recv_ack_seqno: 12837
eq_tapq:replication_ns_1@10.1.3.74:reserved: 1
eq_tapq:replication_ns_1@10.1.3.74:seqno_ack_requested: 12837
eq_tapq:replication_ns_1@10.1.3.74:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.74:suspended: false
eq_tapq:replication_ns_1@10.1.3.74:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.74:total_noops: 58
eq_tapq:replication_ns_1@10.1.3.74:type: producer
eq_tapq:replication_ns_1@10.1.3.74:vb_filter: { 10, 11, 48 }
eq_tapq:replication_ns_1@10.1.3.74:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.76:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_seqno: 79639
eq_tapq:replication_ns_1@10.1.3.76:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.76:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.76:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.76:complete: false
eq_tapq:replication_ns_1@10.1.3.76:connected: true
eq_tapq:replication_ns_1@10.1.3.76:created: 50
eq_tapq:replication_ns_1@10.1.3.76:disconnects: 34
eq_tapq:replication_ns_1@10.1.3.76:empty: true
eq_tapq:replication_ns_1@10.1.3.76:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.76:has_item: false
eq_tapq:replication_ns_1@10.1.3.76:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.76:idle: true
eq_tapq:replication_ns_1@10.1.3.76:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.76:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.76:paused: 1
eq_tapq:replication_ns_1@10.1.3.76:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:qlen: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_drain: 78778
eq_tapq:replication_ns_1@10.1.3.76:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.76:rec_fetched: 79517
eq_tapq:replication_ns_1@10.1.3.76:reconnects: 34
eq_tapq:replication_ns_1@10.1.3.76:recv_ack_seqno: 79638
eq_tapq:replication_ns_1@10.1.3.76:reserved: 1
eq_tapq:replication_ns_1@10.1.3.76:seqno_ack_requested: 79638
eq_tapq:replication_ns_1@10.1.3.76:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.76:suspended: false
eq_tapq:replication_ns_1@10.1.3.76:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:total_noops: 67
eq_tapq:replication_ns_1@10.1.3.76:type: producer
eq_tapq:replication_ns_1@10.1.3.76:vb_filter: { 0, 12, 13 }
eq_tapq:replication_ns_1@10.1.3.76:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.77:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_seqno: 26678
eq_tapq:replication_ns_1@10.1.3.77:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.77:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.77:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.77:complete: false
eq_tapq:replication_ns_1@10.1.3.77:connected: true
eq_tapq:replication_ns_1@10.1.3.77:created: 50
eq_tapq:replication_ns_1@10.1.3.77:disconnects: 4
eq_tapq:replication_ns_1@10.1.3.77:empty: true
eq_tapq:replication_ns_1@10.1.3.77:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.77:has_item: false
eq_tapq:replication_ns_1@10.1.3.77:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.77:idle: true
eq_tapq:replication_ns_1@10.1.3.77:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.77:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.77:paused: 1
eq_tapq:replication_ns_1@10.1.3.77:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:qlen: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_drain: 25247
eq_tapq:replication_ns_1@10.1.3.77:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.77:rec_fetched: 26637
eq_tapq:replication_ns_1@10.1.3.77:reconnects: 4
eq_tapq:replication_ns_1@10.1.3.77:recv_ack_seqno: 26677
eq_tapq:replication_ns_1@10.1.3.77:reserved: 1
eq_tapq:replication_ns_1@10.1.3.77:seqno_ack_requested: 26677
eq_tapq:replication_ns_1@10.1.3.77:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.77:suspended: false
eq_tapq:replication_ns_1@10.1.3.77:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:total_noops: 60
eq_tapq:replication_ns_1@10.1.3.77:type: producer
eq_tapq:replication_ns_1@10.1.3.77:vb_filter: { 1, 14, 15, 38 }
eq_tapq:replication_ns_1@10.1.3.77:vb_filters: 4
eq_tapq:replication_ns_1@10.1.3.78:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_seqno: 12450
eq_tapq:replication_ns_1@10.1.3.78:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.78:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.78:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.78:complete: false
eq_tapq:replication_ns_1@10.1.3.78:connected: true
eq_tapq:replication_ns_1@10.1.3.78:created: 244
eq_tapq:replication_ns_1@10.1.3.78:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.78:empty: true
eq_tapq:replication_ns_1@10.1.3.78:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.78:has_item: false
eq_tapq:replication_ns_1@10.1.3.78:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.78:idle: true
eq_tapq:replication_ns_1@10.1.3.78:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.78:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.78:paused: 1
eq_tapq:replication_ns_1@10.1.3.78:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:qlen: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_drain: 12164
eq_tapq:replication_ns_1@10.1.3.78:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.78:rec_fetched: 12433
eq_tapq:replication_ns_1@10.1.3.78:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.78:recv_ack_seqno: 12449
eq_tapq:replication_ns_1@10.1.3.78:reserved: 1
eq_tapq:replication_ns_1@10.1.3.78:seqno_ack_requested: 12449
eq_tapq:replication_ns_1@10.1.3.78:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.78:suspended: false
eq_tapq:replication_ns_1@10.1.3.78:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:total_noops: 58
eq_tapq:replication_ns_1@10.1.3.78:type: producer
eq_tapq:replication_ns_1@10.1.3.78:vb_filter: { 16, 21, 22 }
eq_tapq:replication_ns_1@10.1.3.78:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.79:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_seqno: 56528
eq_tapq:replication_ns_1@10.1.3.79:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.79:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.79:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.79:complete: false
eq_tapq:replication_ns_1@10.1.3.79:connected: true
eq_tapq:replication_ns_1@10.1.3.79:created: 50
eq_tapq:replication_ns_1@10.1.3.79:disconnects: 22
eq_tapq:replication_ns_1@10.1.3.79:empty: true
eq_tapq:replication_ns_1@10.1.3.79:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.79:has_item: false
eq_tapq:replication_ns_1@10.1.3.79:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.79:idle: true
eq_tapq:replication_ns_1@10.1.3.79:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.79:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.79:paused: 1
eq_tapq:replication_ns_1@10.1.3.79:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:qlen: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_drain: 56340
eq_tapq:replication_ns_1@10.1.3.79:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.79:rec_fetched: 56445
eq_tapq:replication_ns_1@10.1.3.79:reconnects: 22
eq_tapq:replication_ns_1@10.1.3.79:recv_ack_seqno: 56527
eq_tapq:replication_ns_1@10.1.3.79:reserved: 1
eq_tapq:replication_ns_1@10.1.3.79:seqno_ack_requested: 56527
eq_tapq:replication_ns_1@10.1.3.79:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.79:suspended: false
eq_tapq:replication_ns_1@10.1.3.79:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:total_noops: 67
eq_tapq:replication_ns_1@10.1.3.79:type: producer
eq_tapq:replication_ns_1@10.1.3.79:vb_filter: { 17, 29, 57 }
eq_tapq:replication_ns_1@10.1.3.79:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.80:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_seqno: 72204
eq_tapq:replication_ns_1@10.1.3.80:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.80:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.80:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.80:complete: false
eq_tapq:replication_ns_1@10.1.3.80:connected: true
eq_tapq:replication_ns_1@10.1.3.80:created: 50
eq_tapq:replication_ns_1@10.1.3.80:disconnects: 28
eq_tapq:replication_ns_1@10.1.3.80:empty: true
eq_tapq:replication_ns_1@10.1.3.80:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.80:has_item: false
eq_tapq:replication_ns_1@10.1.3.80:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.80:idle: true
eq_tapq:replication_ns_1@10.1.3.80:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.80:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.80:paused: 1
eq_tapq:replication_ns_1@10.1.3.80:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:qlen: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_drain: 69401
eq_tapq:replication_ns_1@10.1.3.80:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.80:rec_fetched: 72095
eq_tapq:replication_ns_1@10.1.3.80:reconnects: 28
eq_tapq:replication_ns_1@10.1.3.80:recv_ack_seqno: 72203
eq_tapq:replication_ns_1@10.1.3.80:reserved: 1
eq_tapq:replication_ns_1@10.1.3.80:seqno_ack_requested: 72203
eq_tapq:replication_ns_1@10.1.3.80:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.80:suspended: false
eq_tapq:replication_ns_1@10.1.3.80:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:total_noops: 66
eq_tapq:replication_ns_1@10.1.3.80:type: producer
eq_tapq:replication_ns_1@10.1.3.80:vb_filter: { 18, 30 }
eq_tapq:replication_ns_1@10.1.3.80:vb_filters: 2
10.1.3.74
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 12
ep_tap_deletes: 511
ep_tap_fg_fetched: 93058
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 77934
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 93746
ep_tap_total_queue: 0
eq_tapq:anon_108:connected: true
eq_tapq:anon_108:created: 57
eq_tapq:anon_108:num_checkpoint_end: 4
eq_tapq:anon_108:num_checkpoint_end_failed: 0
eq_tapq:anon_108:num_checkpoint_start: 5
eq_tapq:anon_108:num_checkpoint_start_failed: 0
eq_tapq:anon_108:num_delete: 1610
eq_tapq:anon_108:num_delete_failed: 0
eq_tapq:anon_108:num_flush: 0
eq_tapq:anon_108:num_flush_failed: 0
eq_tapq:anon_108:num_mutation: 3186
eq_tapq:anon_108:num_mutation_failed: 0
eq_tapq:anon_108:num_opaque: 6
eq_tapq:anon_108:num_opaque_failed: 0
eq_tapq:anon_108:num_unknown: 0
eq_tapq:anon_108:num_vbucket_set: 0
eq_tapq:anon_108:num_vbucket_set_failed: 0
eq_tapq:anon_108:pending_disconnect: false
eq_tapq:anon_108:reserved: 0
eq_tapq:anon_108:supports_ack: true
eq_tapq:anon_108:type: consumer
eq_tapq:anon_110:connected: true
eq_tapq:anon_110:created: 58
eq_tapq:anon_110:num_checkpoint_end: 2
eq_tapq:anon_110:num_checkpoint_end_failed: 0
eq_tapq:anon_110:num_checkpoint_start: 4
eq_tapq:anon_110:num_checkpoint_start_failed: 0
eq_tapq:anon_110:num_delete: 32
eq_tapq:anon_110:num_delete_failed: 0
eq_tapq:anon_110:num_flush: 0
eq_tapq:anon_110:num_flush_failed: 0
eq_tapq:anon_110:num_mutation: 6292
eq_tapq:anon_110:num_mutation_failed: 0
eq_tapq:anon_110:num_opaque: 6
eq_tapq:anon_110:num_opaque_failed: 0
eq_tapq:anon_110:num_unknown: 0
eq_tapq:anon_110:num_vbucket_set: 0
eq_tapq:anon_110:num_vbucket_set_failed: 0
eq_tapq:anon_110:pending_disconnect: false
eq_tapq:anon_110:reserved: 0
eq_tapq:anon_110:supports_ack: true
eq_tapq:anon_110:type: consumer
eq_tapq:anon_118:connected: true
eq_tapq:anon_118:created: 60
eq_tapq:anon_118:num_checkpoint_end: 4
eq_tapq:anon_118:num_checkpoint_end_failed: 0
eq_tapq:anon_118:num_checkpoint_start: 6
eq_tapq:anon_118:num_checkpoint_start_failed: 0
eq_tapq:anon_118:num_delete: 12
eq_tapq:anon_118:num_delete_failed: 0
eq_tapq:anon_118:num_flush: 0
eq_tapq:anon_118:num_flush_failed: 0
eq_tapq:anon_118:num_mutation: 4716
eq_tapq:anon_118:num_mutation_failed: 0
eq_tapq:anon_118:num_opaque: 6
eq_tapq:anon_118:num_opaque_failed: 0
eq_tapq:anon_118:num_unknown: 0
eq_tapq:anon_118:num_vbucket_set: 0
eq_tapq:anon_118:num_vbucket_set_failed: 0
eq_tapq:anon_118:pending_disconnect: false
eq_tapq:anon_118:reserved: 0
eq_tapq:anon_118:supports_ack: true
eq_tapq:anon_118:type: consumer
eq_tapq:anon_121:connected: true
eq_tapq:anon_121:created: 61
eq_tapq:anon_121:num_checkpoint_end: 3
eq_tapq:anon_121:num_checkpoint_end_failed: 0
eq_tapq:anon_121:num_checkpoint_start: 5
eq_tapq:anon_121:num_checkpoint_start_failed: 0
eq_tapq:anon_121:num_delete: 42
eq_tapq:anon_121:num_delete_failed: 0
eq_tapq:anon_121:num_flush: 0
eq_tapq:anon_121:num_flush_failed: 0
eq_tapq:anon_121:num_mutation: 6478
eq_tapq:anon_121:num_mutation_failed: 0
eq_tapq:anon_121:num_opaque: 6
eq_tapq:anon_121:num_opaque_failed: 0
eq_tapq:anon_121:num_unknown: 0
eq_tapq:anon_121:num_vbucket_set: 0
eq_tapq:anon_121:num_vbucket_set_failed: 0
eq_tapq:anon_121:pending_disconnect: false
eq_tapq:anon_121:reserved: 0
eq_tapq:anon_121:supports_ack: true
eq_tapq:anon_121:type: consumer
eq_tapq:anon_128:connected: true
eq_tapq:anon_128:created: 95
eq_tapq:anon_128:num_checkpoint_end: 4
eq_tapq:anon_128:num_checkpoint_end_failed: 0
eq_tapq:anon_128:num_checkpoint_start: 8
eq_tapq:anon_128:num_checkpoint_start_failed: 0
eq_tapq:anon_128:num_delete: 73
eq_tapq:anon_128:num_delete_failed: 0
eq_tapq:anon_128:num_flush: 0
eq_tapq:anon_128:num_flush_failed: 0
eq_tapq:anon_128:num_mutation: 12334
eq_tapq:anon_128:num_mutation_failed: 0
eq_tapq:anon_128:num_opaque: 10
eq_tapq:anon_128:num_opaque_failed: 0
eq_tapq:anon_128:num_unknown: 0
eq_tapq:anon_128:num_vbucket_set: 0
eq_tapq:anon_128:num_vbucket_set_failed: 0
eq_tapq:anon_128:pending_disconnect: false
eq_tapq:anon_128:reserved: 0
eq_tapq:anon_128:supports_ack: true
eq_tapq:anon_128:type: consumer
eq_tapq:anon_94:connected: true
eq_tapq:anon_94:created: 48
eq_tapq:anon_94:num_checkpoint_end: 3
eq_tapq:anon_94:num_checkpoint_end_failed: 0
eq_tapq:anon_94:num_checkpoint_start: 4
eq_tapq:anon_94:num_checkpoint_start_failed: 0
eq_tapq:anon_94:num_delete: 1513
eq_tapq:anon_94:num_delete_failed: 0
eq_tapq:anon_94:num_flush: 0
eq_tapq:anon_94:num_flush_failed: 0
eq_tapq:anon_94:num_mutation: 2992
eq_tapq:anon_94:num_mutation_failed: 0
eq_tapq:anon_94:num_opaque: 6
eq_tapq:anon_94:num_opaque_failed: 0
eq_tapq:anon_94:num_unknown: 0
eq_tapq:anon_94:num_vbucket_set: 0
eq_tapq:anon_94:num_vbucket_set_failed: 0
eq_tapq:anon_94:pending_disconnect: false
eq_tapq:anon_94:reserved: 0
eq_tapq:anon_94:supports_ack: true
eq_tapq:anon_94:type: consumer
eq_tapq:replication_ns_1@10.1.3.75:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.75:ack_seqno: 20368
eq_tapq:replication_ns_1@10.1.3.75:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.75:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.75:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.75:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.75:complete: false
eq_tapq:replication_ns_1@10.1.3.75:connected: true
eq_tapq:replication_ns_1@10.1.3.75:created: 37
eq_tapq:replication_ns_1@10.1.3.75:disconnects: 3
eq_tapq:replication_ns_1@10.1.3.75:empty: true
eq_tapq:replication_ns_1@10.1.3.75:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.75:has_item: false
eq_tapq:replication_ns_1@10.1.3.75:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.75:idle: true
eq_tapq:replication_ns_1@10.1.3.75:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.75:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.75:paused: 1
eq_tapq:replication_ns_1@10.1.3.75:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.75:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.75:qlen: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_drain: 20309
eq_tapq:replication_ns_1@10.1.3.75:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.75:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.75:rec_fetched: 20345
eq_tapq:replication_ns_1@10.1.3.75:reconnects: 3
eq_tapq:replication_ns_1@10.1.3.75:recv_ack_seqno: 20367
eq_tapq:replication_ns_1@10.1.3.75:reserved: 1
eq_tapq:replication_ns_1@10.1.3.75:seqno_ack_requested: 20367
eq_tapq:replication_ns_1@10.1.3.75:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.75:suspended: false
eq_tapq:replication_ns_1@10.1.3.75:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.75:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.75:type: producer
eq_tapq:replication_ns_1@10.1.3.75:vb_filter: { 0, 1, 19, 46 }
eq_tapq:replication_ns_1@10.1.3.75:vb_filters: 4
eq_tapq:replication_ns_1@10.1.3.76:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.76:ack_seqno: 12561
eq_tapq:replication_ns_1@10.1.3.76:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.76:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.76:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.76:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.76:complete: false
eq_tapq:replication_ns_1@10.1.3.76:connected: true
eq_tapq:replication_ns_1@10.1.3.76:created: 38
eq_tapq:replication_ns_1@10.1.3.76:disconnects: 3
eq_tapq:replication_ns_1@10.1.3.76:empty: true
eq_tapq:replication_ns_1@10.1.3.76:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.76:has_item: false
eq_tapq:replication_ns_1@10.1.3.76:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.76:idle: true
eq_tapq:replication_ns_1@10.1.3.76:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.76:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.76:paused: 1
eq_tapq:replication_ns_1@10.1.3.76:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.76:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.76:qlen: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_drain: 12512
eq_tapq:replication_ns_1@10.1.3.76:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.76:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.76:rec_fetched: 12542
eq_tapq:replication_ns_1@10.1.3.76:reconnects: 3
eq_tapq:replication_ns_1@10.1.3.76:recv_ack_seqno: 12560
eq_tapq:replication_ns_1@10.1.3.76:reserved: 1
eq_tapq:replication_ns_1@10.1.3.76:seqno_ack_requested: 12560
eq_tapq:replication_ns_1@10.1.3.76:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.76:suspended: false
eq_tapq:replication_ns_1@10.1.3.76:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.76:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.76:type: producer
eq_tapq:replication_ns_1@10.1.3.76:vb_filter: { 2, 3, 10, 37 }
eq_tapq:replication_ns_1@10.1.3.76:vb_filters: 4
eq_tapq:replication_ns_1@10.1.3.77:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.77:ack_seqno: 17146
eq_tapq:replication_ns_1@10.1.3.77:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.77:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.77:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.77:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.77:complete: false
eq_tapq:replication_ns_1@10.1.3.77:connected: true
eq_tapq:replication_ns_1@10.1.3.77:created: 40
eq_tapq:replication_ns_1@10.1.3.77:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.77:empty: true
eq_tapq:replication_ns_1@10.1.3.77:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.77:has_item: false
eq_tapq:replication_ns_1@10.1.3.77:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.77:idle: true
eq_tapq:replication_ns_1@10.1.3.77:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.77:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.77:paused: 1
eq_tapq:replication_ns_1@10.1.3.77:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.77:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.77:qlen: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_drain: 17098
eq_tapq:replication_ns_1@10.1.3.77:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.77:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.77:rec_fetched: 17127
eq_tapq:replication_ns_1@10.1.3.77:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.77:recv_ack_seqno: 17145
eq_tapq:replication_ns_1@10.1.3.77:reserved: 1
eq_tapq:replication_ns_1@10.1.3.77:seqno_ack_requested: 17145
eq_tapq:replication_ns_1@10.1.3.77:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.77:suspended: false
eq_tapq:replication_ns_1@10.1.3.77:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.77:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.77:type: producer
eq_tapq:replication_ns_1@10.1.3.77:vb_filter: { 4, 11, 47 }
eq_tapq:replication_ns_1@10.1.3.77:vb_filters: 3
eq_tapq:replication_ns_1@10.1.3.78:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.78:ack_seqno: 6313
eq_tapq:replication_ns_1@10.1.3.78:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.78:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.78:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.78:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.78:complete: false
eq_tapq:replication_ns_1@10.1.3.78:connected: true
eq_tapq:replication_ns_1@10.1.3.78:created: 89
eq_tapq:replication_ns_1@10.1.3.78:empty: true
eq_tapq:replication_ns_1@10.1.3.78:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.78:has_item: false
eq_tapq:replication_ns_1@10.1.3.78:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.78:idle: true
eq_tapq:replication_ns_1@10.1.3.78:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.78:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.78:paused: 1
eq_tapq:replication_ns_1@10.1.3.78:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.78:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.78:qlen: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_drain: 6294
eq_tapq:replication_ns_1@10.1.3.78:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.78:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.78:rec_fetched: 6306
eq_tapq:replication_ns_1@10.1.3.78:recv_ack_seqno: 6312
eq_tapq:replication_ns_1@10.1.3.78:reserved: 1
eq_tapq:replication_ns_1@10.1.3.78:seqno_ack_requested: 6312
eq_tapq:replication_ns_1@10.1.3.78:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.78:suspended: false
eq_tapq:replication_ns_1@10.1.3.78:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.78:total_noops: 55
eq_tapq:replication_ns_1@10.1.3.78:type: producer
eq_tapq:replication_ns_1@10.1.3.78:vb_filter: { 5, 55 }
eq_tapq:replication_ns_1@10.1.3.78:vb_filters: 2
eq_tapq:replication_ns_1@10.1.3.79:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.79:ack_seqno: 12490
eq_tapq:replication_ns_1@10.1.3.79:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.79:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.79:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.79:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.79:complete: false
eq_tapq:replication_ns_1@10.1.3.79:connected: true
eq_tapq:replication_ns_1@10.1.3.79:created: 90
eq_tapq:replication_ns_1@10.1.3.79:empty: true
eq_tapq:replication_ns_1@10.1.3.79:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.79:has_item: false
eq_tapq:replication_ns_1@10.1.3.79:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.79:idle: true
eq_tapq:replication_ns_1@10.1.3.79:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.79:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.79:paused: 1
eq_tapq:replication_ns_1@10.1.3.79:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.79:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.79:qlen: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_drain: 12457
eq_tapq:replication_ns_1@10.1.3.79:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.79:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.79:rec_fetched: 12479
eq_tapq:replication_ns_1@10.1.3.79:recv_ack_seqno: 12489
eq_tapq:replication_ns_1@10.1.3.79:reserved: 1
eq_tapq:replication_ns_1@10.1.3.79:seqno_ack_requested: 12489
eq_tapq:replication_ns_1@10.1.3.79:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.79:suspended: false
eq_tapq:replication_ns_1@10.1.3.79:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.79:total_noops: 55
eq_tapq:replication_ns_1@10.1.3.79:type: producer
eq_tapq:replication_ns_1@10.1.3.79:vb_filter: { 6, 7, 20, 56 }
eq_tapq:replication_ns_1@10.1.3.79:vb_filters: 4
eq_tapq:replication_ns_1@10.1.3.80:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.80:ack_seqno: 9298
eq_tapq:replication_ns_1@10.1.3.80:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.80:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.80:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.80:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.80:complete: false
eq_tapq:replication_ns_1@10.1.3.80:connected: true
eq_tapq:replication_ns_1@10.1.3.80:created: 39
eq_tapq:replication_ns_1@10.1.3.80:disconnects: 2
eq_tapq:replication_ns_1@10.1.3.80:empty: true
eq_tapq:replication_ns_1@10.1.3.80:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.80:has_item: false
eq_tapq:replication_ns_1@10.1.3.80:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.80:idle: true
eq_tapq:replication_ns_1@10.1.3.80:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.80:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.80:paused: 1
eq_tapq:replication_ns_1@10.1.3.80:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.80:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.80:qlen: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_drain: 9264
eq_tapq:replication_ns_1@10.1.3.80:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.80:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.80:rec_fetched: 9285
eq_tapq:replication_ns_1@10.1.3.80:reconnects: 2
eq_tapq:replication_ns_1@10.1.3.80:recv_ack_seqno: 9297
eq_tapq:replication_ns_1@10.1.3.80:reserved: 1
eq_tapq:replication_ns_1@10.1.3.80:seqno_ack_requested: 9297
eq_tapq:replication_ns_1@10.1.3.80:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.80:suspended: false
eq_tapq:replication_ns_1@10.1.3.80:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.80:total_noops: 57
eq_tapq:replication_ns_1@10.1.3.80:type: producer
eq_tapq:replication_ns_1@10.1.3.80:vb_filter: { 8, 9, 28 }
eq_tapq:replication_ns_1@10.1.3.80:vb_filters: 3