CBRESTORE - Connection reset by peer

Couchbase Server
1

I was trying the backup/restore capabilities of couchbase and ran into a restore issue:

I run the couchbase docker image and set up my bucket. Then I attach with ‘docker exec -it couch /bin/bash’ to get a console inside the container (since cbbackup does not work from “outside”, like described in the docs).

First, I create a backup with:
cbbackup couchbase://localhost:8091 /backup -u admin -p xyz

But when I try to restore the backup with:
cbrestore /backup -u admin -p xyz couchbase://localhost:8091 -vvvv

…it fails with the following message:

2018-05-21 18:52:18,972: mt cbrestore...
2018-05-21 18:52:18,973: mt  source : /backup
2018-05-21 18:52:18,974: mt  sink   : couchbase://localhost:8091
2018-05-21 18:52:18,974: mt  opts   : {'username': '<xxx>', 'verbose': 4, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'dcp_consumer_queue_length': 1000.0, 'data_only': 0.0, 'uncompress': 0.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, 'cbb_max_mb': 100000.0, 'report': 5.0, 'mcd_compatible': 1.0, 'try_xwm': 1.0, 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'flow_control': 1.0, 'batch_max_size': 1000.0, 'seqno': 0.0, 'design_doc_only': 0.0, 'allow_recovery_vb_remap': 0.0, 'recv_min_bytes': 4096.0}, 'collection': None, 'ssl': False, 'threads': 4, 'to_date': None, 'key': None, 'password': '<xxx>', 'id': None, 'bucket_source': None, 'silent': False, 'dry_run': False, 'from_date': None, 'bucket_destination': None, 'add': False, 'vbucket_list': None, 'separator': '::'}
2018-05-21 18:52:18,975: mt source_class: <class 'pump_bfd.BFDSource'>
2018-05-21 18:52:18,981: mt sink_class: <class 'pump_cb.CBSink'>
2018-05-21 18:52:18,991: mt Starting new HTTP connection (1): localhost
2018-05-21 18:52:19,003: mt "GET /pools/default/buckets HTTP/1.1" 200 9959
2018-05-21 18:52:19,011: mt source_bucket: geldapp
2018-05-21 18:52:19,012: mt sink_bucket: geldapp
2018-05-21 18:52:19,012: mt source_buckets: geldapp
2018-05-21 18:52:19,012: mt bucket: geldapp
2018-05-21 18:52:19,013: mt  source_nodes: 127.0.0.1:8091
2018-05-21 18:52:19,015: mt  enqueueing node: 127.0.0.1:8091
2018-05-21 18:52:19,020: w0  node: 127.0.0.1:8091
2018-05-21 18:52:19,021: w0 sink_bucket: geldapp
2018-05-21 18:52:19,023: w0   connect_db: /backup/2018-05-21T184918Z/2018-05-21T184918Z-full/bucket-geldapp/node-127.0.0.1%3A8091/data-0000.cbb
2018-05-21 18:52:19,031: mt   connect_db: /backup/2018-05-21T184918Z/2018-05-21T184918Z-full/bucket-geldapp/node-127.0.0.1%3A8091/data-0000.cbb
2018-05-21 18:52:19,142: s0 error: recv exception: [Errno 104] Connection reset by peer
2018-05-21 18:52:19,144: s0 MCSink exception:
2018-05-21 18:52:19,148: s0 error: async operation: error: MCSink exception:  on sink: couchbase://localhost:8091(geldapp@127.0.0.1:8091)
2018-05-21 18:52:19,148: w0   pump (/backup(geldapp@127.0.0.1:8091)->couchbase://localhost:8091(geldapp@127.0.0.1:8091)) done.
2018-05-21 18:52:19,149: w0   source : /backup(geldapp@127.0.0.1:8091)
2018-05-21 18:52:19,149: w0   sink   : couchbase://localhost:8091(geldapp@127.0.0.1:8091)
2018-05-21 18:52:19,149: w0          :                total |       last |    per sec
2018-05-21 18:52:19,149: w0  node: 127.0.0.1:8091, done; rv: error: MCSink exception:
error: MCSink exception:

I’m using couchbase:latest (ac51235d7b842c5ca1dc3a102838063b0d51830de7f6f8438bd8a02dfb665bb0).

Am I doing something wrong?

2

I’m just realizing, I’m not the only one having this problem:

@househippo, you asked for verbose logging. Here it is :slight_smile:

3

More information:

I set up a brand new VM and tried to reproduce the error with a sample db. Actually backup and restore worked fine.

Then I copied the backup from the original machine and tried to restore it - same error as on the original post.
I did another cbbackup from the original machine and tried it again on the new VM - same error.

So if you want, I can provide you the 120KB backup.tar.gz including a small script to setup a minimal docker test environment in which you can reproduce the error every time.

If so, please contact me via email, since the backup contains some personal data which I do not want to upload here.

4

Hi Thomas,
Getting the same problem as you.
Did you find what the problem was… and how to solve it?
Thx for sharing…

5

I am having the same issue. Using cbbackupwrapper then cbrestorewrapper, I see “broken pipe on sink” errors (or sometimes “connection reset by peer on sink”).

This only started when upgrading to CB 5.0.1 and 5.1.1. I used to use v4.5.1 and cbbackupwrapper/cbrestorewrapper worked fine.

Also this only happens when the backup is made with cbbackupwrapper on Ubuntu. If I make a backup on Mac (with the same commands) I can restore this backup fine on both Mac and Ubuntu. However, I can’t restore the backup made on Ubuntu on either Mac or Ubuntu.

Any hints would be appreciated.

Thanks,
Giles

1 Like
6

I have the exact same problem on the exact same version 5.1.1. Have you came across any solutions for restoring the backup made on Ubuntu ?

7

I found that the cbbackupwrapper and cbrestorewrapper from CB 4.5.1 still work even though I’m running CB 5.1.1. Therefore I have downloaded the Mac version of CB 4.5.1 and unpacked this to a local directory. I don’t run this version (5.1.1 is still running) but cd to the bin directory within the package and just run cbbackupwrapper/cbrestorewrapper from there.

cd Downloads/couchbase-server-community_4/Couchbase\ Server.app/Contents/Resources/couchbase-core/bin
./cbbackupwrapper http://<ubuntu_host>:8091 ~/cbbackup -b <bucket> -u <cbuser> -p <cbpassword>
./cbrestorewrapper ~/cbbackup http://<mac_host>:8091 -x rehash=1 -b <bucket> -B <bucket> -u <cbuser> -p <cbpassword>
1 Like
8

A “Connection reset by peer” is a symptoms that can be caused by a number of different problems, Network issue, Server going down, etc. Unfortunately each case has to be investigated in turn and we have to be careful about assuming all cases have the same root cause.

That being said there was a race condition in the underlying code used by cbackup, cbrestore and cbtransfer that could trigger this issue, which is fixed in Couchbase Server 5.5.0.

If you upload the logs from the cluster and cbbackup I will have a look.

9

This solution fixed our problem with 5.1.1 CE. We will continue to backup using this solution until there is a fix for the community edition.

10

The same issue in Couchbase 5.1. Luckly we have different passwords in production and test environment. Its looks like it try to access the source server where the backup was taken, even being a completely different server.

C:\Program Files\Couchbase\Server\bin>.\cbrestore --bucket-source microdado --bucket-destination microdado_pp1 E:\Couchbase_Backup\ --username xxxxxx -password “*****” http://wtesterest01v:8091 -x conflict_resolve=0 -x rehash=1
…2018-09-21 20:34:49,359: s0 error: async operation: error: conn.send() exception: [Errno 10054] An existing connection was forcibly closed by the remote host on sink: http:\wtesterest01v:8091(microdado@WPRODSERVER03V.ibge.gov.br:8091)
2018-09-21 20:35:54,404: s3 error: async operation: error: conn.send() exception: [Errno 10053] An established connection was aborted by the software in your host machine on sink: http:\wtesterest01v:8091(microdado@WPRODSERVER04v.ibge.gov.br:8091)
2018-09-21 20:37:09,759: s2 error: async operation: error: conn.send() exception: [Errno 10053] An established connection was aborted by the software in your host machine on sink: http:\wtesterest01v:8091(microdado@WPRODSERVER02v.ibge.gov.br:8091)
error: conn.send() exception: [Errno 10054] An existing connection was forcibly closed by the remote host

1 Like
11

hi
did you manage to find a fix. I have the same issue. it tried to connect to the server it was backed on. I am using 5.1.1 CE.

thanks

12

hi
did you manage to find a fix. I have the same issue. it tried to connect to the server it was backed on. I am using 5.1.1 CE.

thanks

13

Nothing yet, we are waiting a SR opened without response.

14

Hello, I have the same issue.

node: couchbase.docker-c:8091, done; rv: error: MCSink MC error: 135
error: conn.sendall() exception: [Errno 104] Connection reset by peer

I inspected couchbase server logs and I obeserved this in file memcached.log.0
00003.txt when the error hapened

Invalid format specified for DEL_WITH_META - 135 - closing connection packet:mcbp::header: magic:0x80, opcode:0xa8, keylen:17, extlen:28, datatype:0x4, specific:811, bodylen:45, opaque:0x2000000, rawextras:00000000000000a2515665a52647000000

15

Hello @jean-maxime,

I have the same issue.

Unfortunately Connection reset by peer is a symptom that can be caused by a number of different root causes, it’s unlikely to be the same issue.

It would be best to open a new post for this case.

Invalid format specified for DEL_WITH_META - 135 - closing connection packet:mcbp::header: magic:0x80, opcode:0xa8, keylen:17, extlen:28, datatype:0x4, specific:811, bodylen:45, opaque:0x2000000, rawextras:00000000000000a2515665a52647000000

From the error presented the remote end closed the connection because it received a malformed packet. Which version of Couchbase Server is being used?

16

Backup was created on couchbase version enterprise 5.1.0 and reload on 5.1.0 too.
I tried on 5.5.0 but I got same issue too

17

@jean-maxime Are you using Couchbase Sync-Gateway with this Cluster?

18

@pvarley yes we use Sync Gateway

19

@pvarley do you think my problem is due to Sync-Gateway ?

20

@jean-maxime I believe you are hitting MB-31224 . Sync-Gateway makes use of Extended Attributes
 (XATTRs). This is fixed in Couchbase Server 6.0, the issue is that for deleted documents (Tombstones) cbbackup was not correctly storing the Extended Attribute. When it came to doing the restore it would send a malformed packet which would cause the connection to be closed.