Couchbase 6.5 cbrestore - error in pump_mc.py: struct.error: int too large to convert

Hi,
We are facing issue in restoring cbbackup from Couchbase 4.5.1 to 6.5.1. Process terminates with following error.

[root@localhost bkp]# /opt/couchbase/bin/cbrestore -u Administrator -p ******* /root/bkp/ http://127.0.0.1:8091 --bucket-source=person-data --bucket-destination=person-data
Exception in thread s0:] 17.7% (114000/estimated 642907 msgs)
Traceback (most recent call last):
File “/opt/couchbase/lib/python/runtime/lib/python3.7/threading.py”, line 917, in _bootstrap_inner
self.run()
File “/opt/couchbase/lib/python/runtime/lib/python3.7/threading.py”, line 865, in run
self._target(*self._args, **self._kwargs)
File “/opt/couchbase/lib/python/pump_mc.py”, line 112, in run
rv, batch, need_backoff = self.scatter_gather(mconns, batch)
File “/opt/couchbase/lib/python/pump_cb.py”, line 88, in scatter_gather
vbucket_id=vbucket_id)
File “/opt/couchbase/lib/python/pump_mc.py”, line 233, in send_msgs
conf_res) # type: ignore
File “/opt/couchbase/lib/python/pump_mc.py”, line 595, in cmd_request
ext = struct.pack(“>IIQQI”, flg, exp, int.from_bytes(meta, byteorder=‘big’), cas, force)
struct.error: int too large to convert

It may not be the same as Loading.... See the discussion on the following thread.
Cbrestore - Couchbase 4.5 dump to Couchbase 6.5 shows error after 17% of restoring

@jefinjeeva https://issues.couchbase.com/browse/MB-39788 tracks this issue

1 Like

Jeeva,
CB 6.5 is having lot enhancements when compared to 4.5.1.
i recommmend doing a rolling upgrade of 4.5 .1 to atleast 5.5 0r till 6.5 and then take a dump and restore it to 6.5.

Thanks
Rohith

@rvuppala, @pvarley
I have tried XDCR to copy documents from 4.5.1 to 6.5. It successfully copied all documents. Then, I tried to take a dump from 6.5. It showed the same kind of error in CBBACKUP.
Hope this finding help to fix the issue.

@jefinjeeva Sorry for the slow reply on this case, I spent some time last week trying to reproduce this issue and I could not. If you have the data still around and can still reproduce this problem. I would be interested in getting some extra debug information from it.

There is good news, from 7.0.0 cbbackupmgr is supported for CE and EE. It’s more performant than cbbackup/cbrestore and more resilient . I would recommend looking at cbbackupmgr in the future.

@pvarley The issue was only reproduced in our production environment. Maybe some documents in production would have lots of updates. As it is our production data I could not share it. I suspect the issue is occurring while restoring some documents which is having lots of updates on them. Is there any option to know which document causes the issue? Do we have any logs for CBRestore? We are getting the same error still in 6.6 CE.

It was good news for us. I have already checked 7.0.0. Waiting for the release. When would be the stable release of the community edition?

Also got this error when backup from 6.0.0 to 6.5.0.
I’ve also tried 6.5.1, 6.6.0 and 7.1.1, all failed.
Now I need to switch from CentOS6 to Ubuntu18, is it possible to install 6.0.0 in Ubuntu18?

Got the same error when trying to restore the backup taken from 6.0.0 to 6.5.0. tried upgrading 6.6.0. and failed.
bucket is restoring partially and giving this error.
If someone found the reason or a workaround , please let me know ?