Hi,
I am using couchbase server since three years but in these three I didn’t find issue like this, I have upgrade my couchbase clusters from version 5 to 6, but after upgradation, there is issue in XDCR replication version2 could anyone please help me in this. find the error log below
Error:
2018-11-26T15:12:45.810+05:30 [INFO] UPR_STREAMREQ with rollback 0 for vb 75 Failed:
2018-11-26T15:12:45.810+05:30 [INFO] UPR_STREAMREQ for vb 205 successful
2018-11-26T15:12:45.815+05:30 [INFO] Rollback response. body=[0 0 0 0 0 0 0 0], headerBuf=[129 83 0 0 0 0 0 35 0 0 0 8 0 1 1 35 0 0 0 0 0 0 0 0]
2018-11-26T15:12:45.815+05:30 [INFO] Rollback seqno is 0 for response with opaque 65827
2018-11-26T15:12:45.815+05:30 [INFO] UPR_STREAMREQ with rollback 0 for vb 291 Failed:
2018-11-26T15:12:46.010+05:30 [INFO] UPR_STREAMREQ for vb 209 successful
2018-11-26T15:12:46.015+05:30 [INFO] UPR_STREAMREQ for vb 669 successful
2018-11-26T15:12:46.015+05:30 INFO GOXDCR.PipelineMgr: Pipeline updater scheduled time is up. Executing pipeline updater
2018-11-26T15:12:46.015+05:30 INFO GOXDCR.PipelineMgr: Replication status is updated now, current status=name={c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB}, status={Pending}, errors={[{“time”:“2018-11-26T15:12:36.015225717+05:30”,“errMsg”:“Pipeline c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB failed to start”},{“time”:“2018-11-26T15:12:35.98313521+05:30”,“errMsg”:“xmem_c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB_192.168.0.4:11210_0:Xmem is stuck”},{“time”:“2018-11-26T15:12:19.379364596+05:30”,“errMsg”:“xmem_c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB_192.168.0.3:11210_2:Xmem is stuck”}]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline failed to start, err=map[genericPipeline.SetState.Pipeline_running:Pipeline c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB failed to start]}
2018-11-26T15:12:46.015+05:30 INFO GOXDCR.PipelineMgr: Replication c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB’s status is changed, update now
2018-11-26T15:12:46.015+05:30 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB. Current error(s)=r.update_err_ch : xmem_c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB_192.168.0.4:11210_0:Xmem is stuck
genericPipeline.SetState.Pipeline_running : Pipeline c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB failed to start
2018-11-26T15:12:46.015+05:30 INFO GOXDCR.PipelineMgr: Try to stop pipeline c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB
2018-11-26T15:12:46.015+05:30 INFO GOXDCR.PipelineMgr: Trying to stop the pipeline c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB
2018-11-26T15:12:46.215+05:30 [INFO] UPR_STREAMREQ for vb 246 successful
2018-11-26T15:12:46.215+05:30 [INFO] Rollback response. body=[0 0 0 0 0 0 0 0], headerBuf=[129 83 0 0 0 0 0 35 0 0 0 8 0 1 1 197 0 0 0 0 0 0 0 0]
2018-11-26T15:12:46.215+05:30 [INFO] Rollback seqno is 0 for response with opaque 65989
2018-11-26T15:12:46.215+05:30 [INFO] UPR_STREAMREQ with rollback 0 for vb 453 Failed:
2018-11-26T15:12:46.510+05:30 [INFO] Rollback response. body=[0 0 0 0 0 0 0 0], headerBuf=[129 83 0 0 0 0 0 35 0 0 0 8 0 1 0 98 0 0 0 0 0 0 0 0]
2018-11-26T15:12:46.510+05:30 [INFO] Rollback seqno is 0 for response with opaque 65634
2018-11-26T15:12:46.510+05:30 [INFO] UPR_STREAMREQ with rollback 0 for vb 98 Failed:
2018-11-26T15:12:46.515+05:30 [INFO] UPR_STREAMREQ for vb 597 successful
2018-11-26T15:12:46.567+05:30 INFO GOXDCR.PipelineMgr: checkpipeline spec=&{c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB yyu2G4rxkwBOgRQcO22OFA== couchbasDB 64209f702f17d2547078e1f2b31a313f c4c4f3776892ea4953d86aa656bd20a7 couchbasDB e1c37a21b735a602759f4c01d43b9af9 0xc459e3ee60 [131 108 0 0 0 9 104 2 109 0 0 0 32 49 57 53 48 56 51 99 52 100 50 49 52 101 101 97 98 51 50 101 52 101 102 51 52 53 50 50 102 49 54 97 56 104 2 97 9 110 5 0 219 156 158 211 14 104 2 109 0 0 0 32 50 48 57 56 53 55 51 52 51 53 98 101 100 100 101 54 53 101 50 49 101 50 101 99 49 50 53 57 53 100 100 56 104 2 97 2 110 5 0 100 29 111 213 14 104 2 109 0 0 0 32 50 102 99 57 101 50 48 55 49 48 54 49 57 51 57 52 55 55 97 101 101 102 98 50 52 102 55 49 97 50 99 56 104 2 97 1 110 5 0 115 105 186 211 14 104 2 109 0 0 0 32 51 56 100 52 97 50 51 53 53 97 53 53 57 49 100 49 48 49 48 97 98 98 55 55 50 55 55 57 53 101 56 97 104 2 97 2 110 5 0 241 50 112 213 14 104 2 109 0 0 0 32 56 48 52 101 98 52 101 56 101 49 48 98 55 98 50 52 101 57 99 48 55 53 50 49 100 56 51 53 98 51 52 51 104 2 97 3 110 5 0 10 86 180 211 14 104 2 109 0 0 0 32 56 52 101 55 54 54 55 101 49 50 50 50 98 51 99 54 51 98 102 55 57 54 52 56 100 54 97 48 99 51 52 101 104 2 97 4 110 5 0 12 146 22 213 14 104 2 109 0 0 0 32 57 57 49 97 54 55 99 55 102 53 48 100 100 48 100 98 49 99 50 53 52 57 99 55 53 51 51 97 49 98 98 49 104 2 97 1 110 5 0 239 90 184 211 14 104 2 109 0 0 0 32 100 98 49 54 102 48 97 48 97 50 48 101 100 97 52 53 48 102 52 57 57 55 98 101 49 57 48 97 51 53 98 52 104 2 97 1 110 5 0 92 162 22 213 14 104 2 109 0 0 0 32 102 55 100 100 56 97 50 52 54 48 98 51 100 55 51 102 56 51 53 97 49 101 100 102 54 100 100 50 57 51 55 50 104 2 97 9 110 5 0 85 51 112 213 14 106]}, uuid=64209f702f17d2547078e1f2b31a313f
2018-11-26T15:12:46.715+05:30 [INFO] Rollback response. body=[0 0 0 0 0 0 0 0], headerBuf=[129 83 0 0 0 0 0 35 0 0 0 8 0 1 1 42 0 0 0 0 0 0 0 0]
2018-11-26T15:12:46.715+05:30 [INFO] Rollback seqno is 0 for response with opaque 65834
2018-11-26T15:12:46.715+05:30 [INFO] UPR_STREAMREQ with rollback 0 for vb 298 Failed:
2018-11-26T15:12:46.715+05:30 [INFO] UPR_STREAMREQ for vb 325 successful
2018-11-26T15:12:46.715+05:30 [INFO] UPR_STREAMREQ for vb 956 successful
2018-11-26T15:12:46.915+05:30 [INFO] Rollback response. body=[0 0 0 0 0 0 0 0], headerBuf=[129 83 0 0 0 0 0 35 0 0 0 8 0 1 1 40 0 0 0 0 0 0 0 0]
2018-11-26T15:12:46.915+05:30 [INFO] Rollback seqno is 0 for response with opaque 65832
2018-11-26T15:12:46.915+05:30 [INFO] UPR_STREAMREQ with rollback 0 for vb 296 Failed:
2018-11-26T15:12:46.954+05:30 INFO GOXDCR.PipelineMgr: checkpipeline spec=&{c4c4f3776892ea4953d86aa656bd20a7/viralshare-sessions/viralshare-sessions je1Oke2dvJIOFbzPZf4ZlA== viralshare-sessions 3efc23cab768acbcb18b29e46f5d8d6b c4c4f3776892ea4953d86aa656bd20a7 viralshare-sessions 9c823d4c6df44827e4d6739dc72d7f8e 0xc45c198dc0 [131 108 0 0 0 4 104 2 109 0 0 0 32 50 48 57 56 53 55 51 52 51 53 98 101 100 100 101 54 53 101 50 49 101 50 101 99 49 50 53 57 53 100 100 56 104 2 97 11 110 5 0 93 42 111 213 14 104 2 109 0 0 0 32 51 56 100 52 97 50 51 53 53 97 53 53 57 49 100 49 48 49 48 97 98 98 55 55 50 55 55 57 53 101 56 97 104 2 97 1 110 5 0 219 50 112 213 14 104 2 109 0 0 0 32 56 52 101 55 54 54 55 101 49 50 50 50 98 51 99 54 51 98 102 55 57 54 52 56 100 54 97 48 99 51 52 101 104 2 97 11 110 5 0 209 145 22 213 14 104 2 109 0 0 0 32 102 55 100 100 56 97 50 52 54 48 98 51 100 55 51 102 56 51 53 97 49 101 100 102 54 100 100 50 57 51 55 50 104 2 97 3 110 5 0 11 51 112 213 14 106]}, uuid=3efc23cab768acbcb18b29e46f5d8d6b
2018-11-26T15:12:46.957+05:30 INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/TZcIFm5rRetkUMxwbsYuJDWt0IBkfy5BBMitPjVMRN4= with 192.168.0.1:8091 and ActiveHttpsHostName with
2018-11-26T15:12:47.010+05:30 [INFO] Rollback response. body=[0 0 0 0 0 0 0 0], headerBuf=[129 83 0 0 0 0 0 35 0 0 0 8 0 1 0 149 0 0 0 0 0 0 0 0]
2018-11-26T15:12:47.010+05:30 [INFO] Rollback seqno is 0 for response with opaque 65685
2018-11-26T15:12:47.010+05:30 [INFO] UPR_STREAMREQ with rollback 0 for vb 149 Failed:
2018-11-26T15:12:47.190+05:30 INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes:
Original: id:remoteCluster/TZcIFm5rRetkUMxwbsYuJDWt0IBkfy5BBMitPjVMRN4=; uuid:c4c4f3776892ea4953d86aa656bd20a7; name:cb-sg-iweb; hostName:192.168.0.2:8091; userName:Administrator; password:xxxx; secureType:none; certificate:; clientCertificate:; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 108 0 0 0 6 104 2 109 0 0 0 32 49 57 53 48 56 51 99 52 100 50 49 52 101 101 97 98 51 50 101 52 101 102 51 52 53 50 50 102 49 54 97 56 104 2 97 4 110 5 0 126 139 174 211 14 104 2 109 0 0 0 32 50 48 57 56 53 55 51 52 51 53 98 101 100 100 101 54 53 101 50 49 101 50 101 99 49 50 53 57 53 100 100 56 104 2 97 1 110 5 0 62 6 111 213 14 104 2 109 0 0 0 32 56 48 52 101 98 52 101 56 101 49 48 98 55 98 50 52 101 57 99 48 55 53 50 49 100 56 51 53 98 51 52 51 104 2 97 10 110 5 0 125 143 174 211 14 104 2 109 0 0 0 32 56 52 101 55 54 54 55 101 49 50 50 50 98 51 99 54 51 98 102 55 57 54 52 56 100 54 97 48 99 51 52 101 104 2 97 2 110 5 0 36 148 62 213 14 104 2 109 0 0 0 32 57 57 49 97 54 55 99 55 102 53 48 100 100 48 100 98 49 99 50 53 52 57 99 55 53 51 51 97 49 98 98 49 104 2 97 4 110 5 0 0 247 164 211 14 104 2 109 0 0 0 32 102 55 100 100 56 97 50 52 54 48 98 51 100 55 51 102 56 51 53 97 49 101 100 102 54 100 100 50 57 51 55 50 104 2 97 1 110 5 0 222 249 110 213 14 106] [192.168.0.2:8091 192.168.0.3:8091 192.168.0.1:8091 192.168.0.4:8091]
Actual staged changes now in memory: id:remoteCluster/TZcIFm5rRetkUMxwbsYuJDWt0IBkfy5BBMitPjVMRN4=; uuid:c4c4f3776892ea4953d86aa656bd20a7; name:cb-sg-iweb; hostName:192.168.0.2:8091; userName:Administrator; password:xxxx; secureType:none; certificate:; clientCertificate:; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 108 0 0 0 6 104 2 109 0 0 0 32 49 57 53 48 56 51 99 52 100 50 49 52 101 101 97 98 51 50 101 52 101 102 51 52 53 50 50 102 49 54 97 56 104 2 97 4 110 5 0 126 139 174 211 14 104 2 109 0 0 0 32 50 48 57 56 53 55 51 52 51 53 98 101 100 100 101 54 53 101 50 49 101 50 101 99 49 50 53 57 53 100 100 56 104 2 97 1 110 5 0 62 6 111 213 14 104 2 109 0 0 0 32 56 48 52 101 98 52 101 56 101 49 48 98 55 98 50 52 101 57 99 48 55 53 50 49 100 56 51 53 98 51 52 51 104 2 97 10 110 5 0 125 143 174 211 14 104 2 109 0 0 0 32 56 52 101 55 54 54 55 101 49 50 50 50 98 51 99 54 51 98 102 55 57 54 52 56 100 54 97 48 99 51 52 101 104 2 97 2 110 5 0 36 148 62 213 14 104 2 109 0 0 0 32 57 57 49 97 54 55 99 55 102 53 48 100 100 48 100 98 49 99 50 53 52 57 99 55 53 51 51 97 49 98 98 49 104 2 97 4 110 5 0 0 247 164 211 14 104 2 109 0 0 0 32 102 55 100 100 56 97 50 52 54 48 98 51 100 55 51 102 56 51 53 97 49 101 100 102 54 100 100 50 57 51 55 50 104 2 97 1 110 5 0 222 249 110 213 14 106] [192.168.0.2:8091 192.168.0.3:8091 192.168.0.1:8091 192.168.0.4:8091]
2018-11-26T15:12:47.210+05:30 [INFO] Rollback response. body=[0 0 0 0 0 0 0 0], headerBuf=[129 83 0 0 0 0 0 35 0 0 0 8 0 1 0 150 0 0 0 0 0 0 0 0]
2018-11-26T15:12:47.210+05:30 [INFO] Rollback seqno is 0 for response with opaque 65686
2018-11-26T15:12:47.210+05:30 [INFO] UPR_STREAMREQ with rollback 0 for vb 150 Failed:
2018-11-26T15:12:47.215+05:30 [INFO] UPR_STREAMREQ for vb 250 successful
2018-11-26T15:12:47.447+05:30 INFO GOXDCR.PipelineMgr: checkpipeline spec=&{c4c4f3776892ea4953d86aa656bd20a7/couchbasDB2/couchbasDB2 fKV2rj16TwbfKVFCbemhOg== couchbasDB2 4ceed8b3907d16285c2e0b4c59f84ae8 c4c4f3776892ea4953d86aa656bd20a7 couchbasDB2 5af1962c7c46cee961b0111979d75b79 0xc4574628c0 [131 108 0 0 0 5 104 2 109 0 0 0 32 51 56 100 52 97 50 51 53 53 97 53 53 57 49 100 49 48 49 48 97 98 98 55 55 50 55 55 57 53 101 56 97 104 2 97 1 110 5 0 53 36 112 213 14 104 2 109 0 0 0 32 52 97 54 97 56 54 99 99 98 101 55 99 54 98 98 56 51 101 99 50 51 101 53 102 100 57 54 57 101 99 102 57 104 2 97 1 110 5 0 212 249 188 212 14 104 2 109 0 0 0 32 56 52 101 55 54 54 55 101 49 50 50 50 98 51 99 54 51 98 102 55 57 54 52 56 100 54 97 48 99 51 52 101 104 2 97 6 110 5 0 251 145 22 213 14 104 2 109 0 0 0 32 100 98 49 54 102 48 97 48 97 50 48 101 100 97 52 53 48 102 52 57 57 55 98 101 49 57 48 97 51 53 98 52 104 2 97 1 110 5 0 93 162 22 213 14 104 2 109 0 0 0 32 102 55 100 100 56 97 50 52 54 48 98 51 100 55 51 102 56 51 53 97 49 101 100 102 54 100 100 50 57 51 55 50 104 2 97 2 110 5 0 211 50 112 213 14 106]}, uuid=4ceed8b3907d16285c2e0b4c59f84ae8
2018-11-26T15:12:47.510+05:30 [INFO] Rollback response. body=[0 0 0 0 0 0 0 0], headerBuf=[129 83 0 0 0 0 0 35 0 0 0 8 0 1 0 121 0 0 0 0 0 0 0 0]
2018-11-26T15:12:47.510+05:30 [INFO] Rollback seqno is 0 for response with opaque 65657
2018-11-26T15:12:47.510+05:30 [INFO] UPR_STREAMREQ with rollback 0 for vb 121 Failed:
2018-11-26T15:12:47.515+05:30 [INFO] UPR_STREAMREQ for vb 680 successful
2018-11-26T15:12:47.710+05:30 [INFO] UPR_STREAMREQ for vb 208 successful
2018-11-26T15:12:47.715+05:30 [INFO] Rollback response. body=[0 0 0 0 0 0 0 0], headerBuf=[129 83 0 0 0 0 0 35 0 0 0 8 0 1 3 77 0 0 0 0 0 0 0 0]
2018-11-26T15:12:47.715+05:30 [INFO] Rollback seqno is 0 for response with opaque 66381
2018-11-26T15:12:47.715+05:30 [INFO] UPR_STREAMREQ with rollback 0 for vb 845 Failed:
2018-11-26T15:12:47.733+05:30 INFO GOXDCR.PipelineMgr: Replication Status = map[c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB:name={c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB}, status={Pending}, errors={[{“time”:“2018-11-26T15:12:36.015225717+05:30”,“errMsg”:“Pipeline c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB failed to start”},{“time”:“2018-11-26T15:12:35.98313521+05:30”,“errMsg”:“xmem_c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB_192.168.0.4:11210_0:Xmem is stuck”},{“time”:“2018-11-26T15:12:19.379364596+05:30”,“errMsg”:“xmem_c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB_192.168.0.3:11210_2:Xmem is stuck”}]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline failed to start, err=map[genericPipeline.SetState.Pipeline_running:Pipeline c4c4f3776892ea4953d86aa656bd20a7/couchbasDB/couchbasDB failed to start]}
c4c4f3776892ea4953d86aa656bd20a7/viralshare-sessions/viralshare-sessions:name={c4c4f3776892ea4953d86aa656bd20a7/viralshare-sessions/viralshare-sessions}, status={Replicating}, errors={}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}
c4c4f3776892ea4953d86aa656bd20a7/couchbasDB2/couchbasDB2:name={c4c4f3776892ea4953d86aa656bd20a7/couchbasDB2/couchbasDB2}, status={Replicating}, errors={}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}