I am using Couchbase 4.1.0-5005 Community Edition (build-5005). I have a main cluster and a remote single node cluster with XDCR
In my XDCR topology the remote cluster suffered a downtime on 2017-09-02 13:40. Since then - the replication counter under http://remotecb:8091/pools/default/tasks is showing an increasingly changesLeft counter. In goxdcr.log it shows that changesLeft = 0:
cff1f24ee22368f5886da678a834aed0/main/main total_docs=37998167, docs_processed=37998167, changes_left=0
The statisticsManager is prompting the error in the log every now and then, with the timestamp it had happened on:
StatisticsManager 2017-09-04T04:40:39.775Z [INFO] Stats for pipeline cff1f24ee22368f5886da678a834aed0/main/main-117446212 {“CkptMgr”: {“num_checkpoints”: 76, “num_failedckpts”: 0, “time_committing”: {“count”: 76, “max”: 1890000, “mean”: 1.411092105263158e+06, “min”: 349000}}, “Errors”: “[{"time":"2017-09-02T13:42:17.77474215Z","errMsg":"Failed to connect to cluster reference remoteCluster/Js1sCS-GqWs1N_tlKoOhrhXySSeDEodJSwvJzK-JhLc=\n"},{"time":"2017-09-02T13:26:45.813451011Z","errMsg":"Error constructing vb couchApiBase map for bucket main on remote cluster Secondary because of failure to retrieve bucket info\n\n err = Get http://remotecb:8091/pools/default/buckets/main?bucket_uuid=6482a8b3e4d372b94f5132c9e3b9071d: dial tcp remotecb:8091: i/o timeout"},{"time":"2017-09-02T13:25:48.216987029Z","errMsg":"Error constructing vb couchApiBase map for bucket main on remote cluster Secondary because of failure to retrieve bucket info\n\n err = Get http://remotecb:8091/pools/default/buckets/main?bucket_uuid=6482a8b3e4d372b94f5132c9e3b9071d: dial tcp remotecb:8091: i/o timeout"},{"time":"2017-09-02T13:24:40.626231985Z","errMsg":"xmem_cff1f24ee22368f5886da678a834aed0/main/remotecb:11210_1:Failed to repair connections to target cluster."}]”, “Overview”: {“bandwidth_usage”: 187492, “changes_left”: 0, “data_replicated”: 1719611301, “dcp_datach_length”: 0, “dcp_dispatch_time”: 46, “deletion_docs_written”: 31264, “deletion_failed_cr_source”: 0, “deletion_filtered”: 0, “deletion_received_from_dcp”: 31264, “docs_checked”: 37996020, “docs_failed_cr_source”: 240, “docs_filtered”: 0, “docs_opt_repd”: 32782, “docs_processed”: 37998196, “docs_received_from_dcp”: 126152, “docs_rep_queue”: 0, “docs_written”: 125912, “expiry_docs_written”: 0, “expiry_failed_cr_source”: 0, “expiry_filtered”: 0, “expiry_received_from_dcp”: 0, “num_checkpoints”: 76, “num_failedckpts”: 0, “rate_doc_checks”: 1, “rate_doc_opt_repd”: 0, “rate_received_from_dcp”: 2, “rate_replicated”: 2, “resp_wait_time”: 0, “set_docs_written”: 94648, “set_failed_cr_source”: 240, “set_filtered”: 0, “set_received_from_dcp”: 94888, “size_rep_queue”: 0, “time_committing”: 1411092, “wtavg_docs_latency”: 1, “wtavg_meta_latency”: 0}, “Progress”: “Pipeline is running”, “Router_dcp_cff1f24ee22368f5886da678a834aed0/main/main_172.19.12.20:11210_0”: {“deletion_filtered”: 0, “docs_filtered”: 0, “expiry_filtered”: 0, “set_filtered”: 0}, “Router_dcp_cff1f24ee22368f5886da678a834aed0/main/main_172.19.12.20:11210_1”: {“deletion_filtered”: 0, “docs_filtered”: 0, “expiry_filtered”: 0, “set_filtered”: 0}, “Status”: “Replicating”, “dcp_cff1f24ee22368f5886da678a834aed0/main/main_172.19.12.20:11210_0”: {“dcp_datach_length”: 0, “dcp_dispatch_time”: {“count”: 62618, “max”: 13505, “mean”: 61.123, “min”: 4}, “deletion_received_from_dcp”: 15148, “docs_received_from_dcp”: 62618, “expiry_received_from_dcp”: 0, “set_received_from_dcp”: 47470}, “dcp_cff1f24ee22368f5886da678a834aed0/main/main_172.19.12.20:11210_1”: {“dcp_datach_length”: 0, “dcp_dispatch_time”: {“count”: 63534, “max”: 2054, “mean”: 32.864, “min”: 4}, “deletion_received_from_dcp”: 16116, “docs_received_from_dcp”: 63534, “expiry_received_from_dcp”: 0, “set_received_from_dcp”: 47418}, “xmem_cff1f24ee22368f5886da678a834aed0/main/main_remotecb:11210_0”: {“data_replicated”: 915832191, “deletion_docs_written”: 15148, “deletion_failed_cr_source”: 0, “docs_failed_cr_source”: 107, “docs_opt_repd”: 16388, “docs_rep_queue”: 0, “docs_written”: 62511, “expiry_docs_written”: 0, “expiry_failed_cr_source”: 0, “resp_wait_time”: {“count”: 62511, “max”: 33, “mean”: 0.624, “min”: 0}, “set_docs_written”: 47363, “set_failed_cr_source”: 107, “size_rep_queue”: 0, “wtavg_docs_latency”: {“count”: 62511, “max”: 61, “mean”: 1.723, “min”: 0}, “wtavg_meta_latency”: {“count”: 46228, “max”: 73, “mean”: 0.3, “min”: 0}}, “xmem_cff1f24ee22368f5886da678a834aed0/main/main_remotecb:11210_1”: {“data_replicated”: 803779110, “deletion_docs_written”: 16116, “deletion_failed_cr_source”: 0, “docs_failed_cr_source”: 133, “docs_opt_repd”: 16394, “docs_rep_queue”: 0, “docs_written”: 63401, “expiry_docs_written”: 0, “expiry_failed_cr_source”: 0, “resp_wait_time”: {“count”: 63401, “max”: 30, “mean”: 0.619, “min”: 0}, “set_docs_written”: 47285, “set_failed_cr_source”: 133, “size_rep_queue”: 0, “wtavg_docs_latency”: {“count”: 63401, “max”: 31, “mean”: 1.606, “min”: 0}, “wtavg_meta_latency”: {“count”: 47139, “max”: 20, “mean”: 0.303, “min”: 0}}}
My question is: how to recosiliate the REST API counter or the conflict in the XDCR, as I don’t know what to believe