[MB-4884] couchbase backup utility should run integrity check after backup and before vacuuming Created: 09/Mar/12  Updated: 09/Jan/13  Resolved: 26/Mar/12

Status: Closed
Project: Couchbase Server
Component/s: tools
Affects Version/s: 1.7.1, 1.7.1.1, 1.7.2, 1.8.0
Fix Version/s: 1.8.1, 1.8.2
Security Level: Public

Type: Bug Priority: Major
Reporter: Farshid Ghods Assignee: Bin Cui
Resolution: Fixed Votes: 0
Labels: 1.8.1-release-notes, customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos 5.8 64-bit


 Description   
one of the users reported :

Backup of default done
Vacuum
of default done
Backup of default-0.mb done
Error: database disk image is malformed
Vacuum of default-0.mb done
Backup of default-1.mb done
Error: database disk image is malformed
Vacuum of default-1.mb done
Backup of default-2.mb done
Vacuum of default-2.mb done
Backup of default-3.mb done
Error: database disk image is malformed
Vacuum of default-3.mb done

the user then ran integrity check on the corrupted backup shard :

-sh-3.2$ /opt/couchbase/bin/sqlite3 default-1.mb
SQLite version 3.7.2
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> PRAGMA integrity_check;
*** in database main ***
On tree page 6543 cell 0: 4 of 6 pages missing from overflow list starting at 2719613
On tree page 7161 cell 0: 2 of 6 pages missing from overflow list starting at 3548396
On tree page 347385 cell 0: 1 of 2 pages missing from overflow list starting at 4781681
On tree page 248234 cell 0: 2nd reference to page 5647436
On tree page 361270 cell 0: 2 of 3 pages missing from overflow list starting at 4384604
On tree page 505716 cell 0: 5 of 8 pages missing from overflow list starting at 1540962
On tree page 1298298 cell 0: 8 of 12 pages missing from overflow list starting at 5226912
On tree page 1490361 cell 0: 1 of 2 pages missing from overflow list starting at 5517260
On tree page 1559833 cell 0: 1 of 2 pages missing from overflow list starting at 3237937
On tree page 1408893 cell 0: 13 of 19 pages missing from overflow list starting at 2522699
On tree page 3001939 cell 0: 2nd reference to page 4510212
On tree page 1331840 cell 0: 1 of 2 pages missing from overflow list starting at 5378314
On tree page 2068325 cell 0: 11 of 12 pages missing from overflow list starting at 3128187
On tree page 2210730 cell 0: 7 of 8 pages missing from overflow list starting at 5482006
On tree page 1540338 cell 0: 2nd reference to page 3862335
On tree page 1958772 cell 0: 1 of 2 pages missing from overflow list starting at 3120119
On tree page 2899988 cell 0: 2nd reference to page 5482006
On tree page 3477578 cell 0: 2nd reference to page 5518857
On tree page 3606737 cell 0: 2nd reference to page 5227907
On tree page 4500400 cell 0: 2 of 3 pages missing from overflow list starting at 2081640
On tree page 2648274 cell 0: 1 of 2 pages missing from overflow list starting at 4268285
On tree page 1589224 cell 0: 2nd reference to page 3548396
On tree page 3239597 cell 0: 2nd reference to page 2010905
On tree page 4433825 cell 0: 2nd reference to page 4062110
On tree page 882949 cell 0: 2nd reference to page 4438689
On tree page 2696470 cell 0: 1 of 2 pages missing from overflow list starting at 5485665
On tree page 1805738 cell 0: 1 of 2 pages missing from overflow list starting at 5360333
On tree page 4680065 cell 1: 2nd reference to page 3371159
On tree page 4506277 cell 0: 1 of 2 pages missing from overflow list starting at 2827305
On tree page 4719656 cell 0: 1 of 2 pages missing from overflow list starting at 1688045
On tree page 3116190 cell 0: 2nd reference to page 1409168
On tree page 5509032 cell 0: 2nd reference to page 4165388
On tree page 2049533 cell 1: Rowid 105665 out of order (max larger than parent max of 105661)
On tree page 5494237 cell 0: Rowid 105662 out of order (min less than parent min of 105665)
On tree page 5538158 cell 1: 2nd reference to page 2950787
On tree page 3449976 cell 0: 2nd reference to page 875478
On tree page 1605890 cell 4: 2 of 3 pages missing from overflow list starting at 4643092
On tree page 1605890 cell 4: Rowid 108215 out of order (max larger than parent max of 108210)
On tree page 2800825 cell 0: Rowid 108216 out of order (max larger than parent max of 108214)
On tree page 3338950 cell 0: Rowid 108215 out of order (min less than parent min of 108216)
On tree page 3433440 cell 0: 2nd reference to page 1424424
Page 2457439: btreeInitPage() returns error code 11
On tree page 5607184 cell 75: Child page depth differs
On tree page 5607184 cell 76: Child page depth differs
On tree page 486521 cell 0: 2 of 4 pages missing from overflow list starting at 316405
On tree page 493239 cell 0: 1 of 2 pages missing from overflow list starting at 5351072
On tree page 493208 cell 0: 1 of 2 pages missing from overflow list starting at 5256055
On tree page 160940 cell 0: 1 of 2 pages missing from overflow list starting at 5554097
On tree page 1019204 cell 0: 2 of 3 pages missing from overflow list starting at 4217052
On tree page 1218789 cell 0: 2nd reference to page 316405
On tree page 1549565 cell 0: 1 of 4 pages missing from overflow list starting at 4112133
On tree page 1129128 cell 0: 1 of 3 pages missing from overflow list starting at 3874575
On tree page 1558712 cell 0: 1 of 2 pages missing from overflow list starting at 5279593
On tree page 1892214 cell 0: 1 of 2 pages missing from overflow list starting at 5586509
On tree page 1300281 cell 0: 1 of 2 pages missing from overflow list starting at 5285166
On tree page 1220910 cell 0: 2 of 3 pages missing from overflow list starting at 5128098
On tree page 3070292 cell 1: 2nd reference to page 5555189
On tree page 3181687 cell 0: 2nd reference to page 5255690
On tree page 3560123 cell 0: 2nd reference to page 1630143
On tree page 2748034 cell 0: 1 of 2 pages missing from overflow list starting at 3993856
On tree page 3705983 cell 0: 2nd reference to page 5599995
On tree page 2152439 cell 0: 1 of 2 pages missing from overflow list starting at 3400534
On tree page 425083 cell 2: 1 of 2 pages missing from overflow list starting at 4305563
On tree page 3605923 cell 0: 2nd reference to page 3024261
On tree page 4305105 cell 0: 2nd reference to page 3436968
On tree page 3872405 cell 0: 1 of 2 pages missing from overflow list starting at 1230546
On tree page 1399679 cell 0: 1 of 2 pages missing from overflow list starting at 3257397
On tree page 924249 cell 0: 1 of 2 pages missing from overflow list starting at 5482794
On tree page 3578548 cell 0: 2nd reference to page 5482794
On tree page 635650 cell 0: 10 of 11 pages missing from overflow list starting at 5430517
On tree page 782697 cell 0: 1 of 2 pages missing from overflow list starting at 5594886
On tree page 3021006 cell 0: 1 of 2 pages missing from overflow list starting at 2120611
On tree page 4798998 cell 0: 2nd reference to page 917219
On tree page 666848 cell 0: 1 of 2 pages missing from overflow list starting at 4444655
On tree page 5260799 cell 0: 2nd reference to page 5430517
On tree page 5431912 cell 0: 2nd reference to page 4444655
On tree page 5473718 cell 0: 1 of 2 pages missing from overflow list starting at 727464
On tree page 932894 cell 3: Rowid 105594 out of order (max larger than parent max of 105590)
On tree page 4125868 cell 1: 2nd reference to page 4681418
On tree page 4125868 cell 2: Rowid 105591 out of order (min less than parent min of 105594)
On tree page 5500578 cell 0: 2nd reference to page 3691475
On tree page 5505656 cell 2: 2nd reference to page 4996604
On tree page 5316722 cell 0: 2nd reference to page 1606735
Page 550004: btreeInitPage() returns error code 11
On tree page 5541521 cell 63: Child page depth differs
On tree page 5541521 cell 64: Child page depth differs


 Comments   
Comment by Farshid Ghods [ 09/Mar/12 ]
we can add a line to our backup script to run "PRAGMA integrity_check;" after the backup and report that backup has failed
Generated at Wed May 22 15:39:14 CDT 2013 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.