I did some tests after 2.5 to 4.1 migration and it turned out that the whole restore process is now at least 5 times slower than it was on 2.5 server.
There is no visible difference with the small buckets with few thousands documents, but the buckets with more than 100 thousands restores much slower.
Generally a backup with ~2 million documents in few buckets was restored in 7 minutes on CB 2.5.
so I removed CB 2.5 and installed 4.1, I used the same memory size on for CB and the same bucket sizes
and restored the same backup as previously - it took 55 minutes. (almost 8 times slower)
I though it might be some kind of conversion when restore 2.5 backup on 4.1 server so after that I created a backup on this new 4.1 server and after recreating all buckets I restored it back on 4.1 server. It took 37minutes (5 times slower)
I repeated this twice plus one time different windows machines, with very similar results.
We have the same issues with our CBs on Linux. The restore time has increased significantly.
so just want to know if this is a bug or a new CB feature?
Very sad to say that, but it’s a “well-known feature”:
Using cbbackupmgr Tool
Couchbase Backup Manager is an enterprise-grade backup and restore utility that is available in the Couchbase Server Enterprise Edition only. Designed for the Enterprise Edition, it replaces the cbbackup and cbrestore tools as the primary and recommended means of backup and restore for Enterprise customers from version 4.5 and above. It enables backup and restore of data, indexes, and bucket configurations at a dramatically increased speed over the previous generation tools.
I’m unsure why the cbrestore from 2.5 is working so much faster for you than the 4.1 cbrestore. It is true that we have added an enterprise version for backup/restore called cbbackupmgr, but there should not be any performance regressions from our current tools. Also, this isn’t an issue I’ve heard about previously from other users so I’m not sure why this is running so slowly for you.
One option is that you can try using cbbackupwrapper and cbrestorewrapper instead of cbbackup/cbrestore. These commands allow running “multi-threaded” versions of cbbackup/cbrestore and should improve performance since cbbackup and cbrestore are only single threaded and always have been.
Also, you can file a ticket at couchbase.com/issues with your server logs and backup logs if you like and I can look at the logs to see if there is anything that stands out. If you do this please run the backup with -vv to get verbose output.
cbbackup actually works in a very similar way to XDCR. Both use DCP in order to stream data out of the server and this is probably the closest thing you can get to just having a direct dump from memory since DCP will stream items from the in-memory cache and avoid going to disk when possible. In your examples, where are you running your cluster and what type of disk are you writing to. I suspect that you are using a disk that has a 5MB/sec max IO throughput. Can you run iostat while the backup is taking place to confirm whether or not this is the case?
Another thing that I’d be interested in knowing is why the system is unresponsive. Is it due to memory, CPU, or disk IO usage?
cbbakup.exe is working fine from directory with space in name and without.
cbbakupwrapper.exe doesn’t work from directory with space in name, and this error message is from cbbackupwrapper.exe
anyway, once moved to no-space directory it works fine.
I compared times for standard cbbackup and cbbackupwrapper, for 5 attempts.
CBBackupwrapper does backups in 21 minutes.
CBBackup does backups in 26 minutes
so there is a small progress with wrapper,
Unfortunately I wasn’t able to test cbrestorewrapper as it just hangs at the end
I can see on CB server that all buckets have been restored but the command doesn’t return, I was waiting for 2hours before I killed the process. It hangs every time.
@mikew what kind of server log would you need to analyse this issue?
As soon as I find a new server for testing I will create the ticket. as now I cannot install 2.5.2 on my local machine, facing “Computing space requirements” issue and the workaround with registry trick doesn’t work for me.
It is only 2.5.2 installation issues as 4.1.1 can be installed and uninstalled without any problems.
I am afraid my general feeling about CB is not very positive