http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-6763 Couchbase Server * Place holder for adding error logging on the Source XDCR cluster. <br/> - Will add more as we come across more use-cases/scenarios. <br/> <br/> Replication failure due to the following reason should be logged - This will be helpful for support to troubleshoot errors and for any end-user. <br/> *Today most of the replication failures are debugged using ns_server logs, we should move the error displaying on the UI as well. <br/> <br/> Errors <br/> ----------------- <br/> - Source replication cluster reference cannot be deleted if there is a replication is set up (another bug to track this - <a href="http://www.couchbase.com/issues/browse/MB-6843" title="Correctly handle deleting cluster reference that&#39;s used by currently defined replications"><strike>MB-6843</strike></a>) <br/> - Source replication is deleted - issue user-visible log message saying &quot;Replication has been deleted&quot; on source cluster <br/> - Source bucket is deleted - (1) Issue warning message to the user on delete bucket that replication is going on (2). issue user-visible log message saying &quot;Bucket has been deleted, replication to remote bucket has stopped&quot; on source cluster (3) Error message in the XDCR page on the replication impacted saying &quot;Bucket has been delete, XDCR has stopped&quot; <br/> - Source bucket is flushed - Cannot be done. <br/> <br/> <br/> For errors on the destination node and replication times out: raise this error on the source node . <br/> <br/> <br/> MB-6763

XDCR Error Logging

Bug Blocker Closed Fixed Public Aleksey Kondratenko Ketaki Gangal Thu, 27 Sep 2012 13:13:12 -0500 Thu, 10 Jan 2013 01:14:17 -0600 Wed, 10 Oct 2012 14:18:41 -0500 2.0 2.0 cross-datacenter-replication 0 2 fyi, .. <a href="http://www.couchbase.com/issues/browse/MB-5611">http://www.couchbase.com/issues/browse/MB-5611</a> Pasting useful message that we need to provide better errors on from <a href="http://www.couchbase.com/issues/browse/MB-5611" title="XDCR: UI should show the reason why replication failed "><strike>MB-5611</strike></a> <br/> <br/> Some common XDCR failure reasons: <br/> <br/> 1 db_not_found error: when node is unresponsive, for e.g: <br/> &quot;could not open <a href="http://Administrator:">http://Administrator:</a> <br/> *****@10.3.3.28:8092/default%2f120%3b093c0a978eb59342ea52d87eae424bb3/&quot; <br/> <br/> 2 badmatch,{error,corrupted_data}, Erlang-related corruption <br/> &nbsp;&nbsp;&nbsp;&nbsp;[{couch_compress,decompress,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_doc,with_uncompressed_body,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_doc,to_json_base64,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{xdc_vbucket_rep_worker,maybe_flush_docs,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,foldl,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{xdc_vbucket_rep_worker,local_process_batch,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{xdc_vbucket_rep_worker,queue_fetch_loop,4}] <br/> <br/> 3 checkpoint_commit_failure <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{bad_return_value, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{checkpoint_commit_failure, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;Failure on target commit: {error,&lt;&lt;\&quot;not_found\&quot;&gt;&gt;}&quot;&gt;&gt;}} <br/> <br/> 4 http_request_failed <br/> xdc_replicator:handle_info:282] Worker &lt;0.11173.72&gt; died with reason: {http_request_failed,&quot;POST&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;<a href="http://10.3.121.33:8092/default%2F684/_bulk_docs">http://10.3.121.33:8092/default%2F684/_bulk_docs</a>&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{error,{code,500}}} <br/> <br/> &nbsp;&nbsp;Replicator: couldn&#39;t write document <br/> xdc_replicator_worker:flush_docs:111] Replicator: couldn&#39;t <br/> write document ``, revision ``, <br/> to target database `<a href="http://10.3.121.33:8092/default%2F683/">http://10.3.121.33:8092/default%2F683/</a>`. Error: ``, reason: ``. <br/> <br/> 5 replicator_died <br/> {replicator_died, {&#39;EXIT&#39;,&lt;15849.2212.0&gt;, {badmatch,{error,closed}}}} <br/> <br/> 6 bulk_set_vbucket_state_failed <br/> General error seen when rebalance fails due to vbucket_map not ready (possibly) <br/> that may cause replication to fail. Hi Dipti, <br/> <br/> Thanks for organizing the meeting. Actually today XDCR has already have the API to expose errors to ns_server. The API is within XDCR replication manager (xdc_rep_manager:latest_errors()) and when called, it will return the last 10 errors for each bucket which are actively replicating. Alk will expose these errors (or at least some of them) on UI. Alk will also determine where to expose these msgs. <br/> <br/> If users feel these error msgs are hard to understand, we can change it later to make it more user-friendly. At this time, we just need to ask Alk expose them to UI. Let me know if any other questions. <br/> <br/> <br/> Thanks! <br/> <br/> Junyi Nothing to do within XDCR. All ns_server work. Fixed &amp; approved but still sits in gerrit: <a href="http://review.couchbase.org/#/c/21459/">http://review.couchbase.org/#/c/21459/</a><br/> <br/> it is naive, but so are &#39;errors&#39; from xdcr Date of First Response Fri, 28 Sep 2012 15:46:22 -0500 Rank 3925 Sprint Status Time In Status