Couchbase Server

mbrestore tool does not support restoring from WAL'ed database files

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Won't Fix
  • Affects Version/s: 1.7 alpha 2, 1.8.0
  • Fix Version/s: 1.7 alpha 2
  • Component/s: tools
  • Security Level: Public
  • Labels:
    None

Description

This error is presented:
tcarpenter@sports-mb-test02:/opt/membase/1.6.4.1/data/ns_1]$ /tmp/python271/Python-2.7.1/python /opt/membase/1.6.4.1/bin/ep_engine/management/mbrestore -a default default-0.mb default-1.mb default-2.mb default-3.mb
Traceback (most recent call last):
  File "/opt/membase/1.6.4.1/bin/ep_engine/management/mbrestore", line 147, in <module>
    main()
  File "/opt/membase/1.6.4.1/bin/ep_engine/management/mbrestore", line 105, in main
    db.executemany("attach ? as ?", zip(db_filenames, attached_dbs))
sqlite3.OperationalError: file is encrypted or is not a database

According to Dustin, this is because the mbrestore tool does not currently support WAL db files and the workaround is to run: /opt/membase/bin/ep_engine/management/sqlite3 <filename> 'pragma journal_mode=delete' on all the db files you are trying to restore.

Please add the above workaround to the mbrestore script itself.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Sean Lynch added a comment -
I think it'd be even better to require a newer version of sqlite for Python, or ship our own sqlite module or even Python runtime or frozen scripts.
Show
Sean Lynch added a comment - I think it'd be even better to require a newer version of sqlite for Python, or ship our own sqlite module or even Python runtime or frozen scripts.
Sean Lynch made changes -
Field Original Value New Value
Assignee Dustin Sallings [ dustin ]
Hide
Permalink
Perry Krug added a comment -
A Pivotal Tracker story has been created for this Issue: http://www.pivotaltracker.com/story/show/9309485
Show
Perry Krug added a comment - A Pivotal Tracker story has been created for this Issue: http://www.pivotaltracker.com/story/show/9309485
Hide
Permalink
Perry Krug added a comment -
This is not a python version problem, it's a problem with WAL.
Show
Perry Krug added a comment - This is not a python version problem, it's a problem with WAL.
Sharon Barr made changes -
Assignee Dustin Sallings [ dustin ] Bin Cui [ bcui ]
Fix Version/s 1.7 beta [ 10110 ]
Sharon Barr made changes -
Fix Version/s 1.7 alpha 1 [ 10170 ]
Fix Version/s 1.7 beta [ 10110 ]
Sharon Barr made changes -
Affects Version/s 1.7 Alpha 2 [ 10180 ]
Affects Version/s 1.6.4.1 [ 10141 ]
Sharon Barr made changes -
Fix Version/s 1.7 Alpha 2 [ 10180 ]
Fix Version/s 1.7 alpha 1 [ 10170 ]
Bin Cui made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Farshid Ghods made changes -
Flagged [Release Note]
Hide
Permalink
Perry Krug added a comment -
Perry Krug deleted the linked story in Pivotal Tracker
Show
Perry Krug added a comment - Perry Krug deleted the linked story in Pivotal Tracker
Hide
Permalink
Sandeep Mukherjee added a comment -
Running the workaround:

run: /opt/membase/bin/ep_engine/management/sqlite3 <filename> 'pragma journal_mode=delete'

gives an error:

sqlite3.exe: Error: too many options: "journal_mode=delete'"
Use -help for a list of options.
Show
Sandeep Mukherjee added a comment - Running the workaround: run: /opt/membase/bin/ep_engine/management/sqlite3 <filename> 'pragma journal_mode=delete' gives an error: sqlite3.exe: Error: too many options: "journal_mode=delete'" Use -help for a list of options.
Hide
Permalink
Perry Krug added a comment -
This issue should be resolved and the workaround not needed in post 1.7 releases...can you verify that?
Show
Perry Krug added a comment - This issue should be resolved and the workaround not needed in post 1.7 releases...can you verify that?
Hide
Permalink
Sandeep Mukherjee added a comment -
Well here is my problem. I have data which ranges upto 57 Gig with 11 million entries. The data was generated using the version 1.6.5. If i try to load the data to the new version will it be backward compatible?
Show
Sandeep Mukherjee added a comment - Well here is my problem. I have data which ranges upto 57 Gig with 11 million entries. The data was generated using the version 1.6.5. If i try to load the data to the new version will it be backward compatible?
Hide
Permalink
Perry Krug added a comment -
There are a few strategies for converting the data. They're not directly compatible, but you can either upgrade the database (http://www.couchbase.org/wiki/display/membase/Membase+Server+1.7) or use our restore tool to pull the data out and replay it into a new cluster (http://www.couchbase.org/wiki/display/membase/Backup+and+Restore+with+Membase). The latter step is where this bug came from, and if you use the latest mbrestore script (provided with the 1.7 installation) you won't need this workaround.
Show
Perry Krug added a comment - There are a few strategies for converting the data. They're not directly compatible, but you can either upgrade the database ( http://www.couchbase.org/wiki/display/membase/Membase+Server+1.7 ) or use our restore tool to pull the data out and replay it into a new cluster ( http://www.couchbase.org/wiki/display/membase/Backup+and+Restore+with+Membase ). The latter step is where this bug came from, and if you use the latest mbrestore script (provided with the 1.7 installation) you won't need this workaround.
Hide
Permalink
Perry Krug added a comment -
This has been reported on 1.8:
"When trying to restore from snapshot with cbrestore, it fails with:

"Either the metadata db file is not specified
or the backup files need to be upgraded to version 2.
Please use cbdbupgrade for upgrade or contact support."

But using cbbackup on snapshot data and restoring afterwards works."
Show
Perry Krug added a comment - This has been reported on 1.8: "When trying to restore from snapshot with cbrestore, it fails with: "Either the metadata db file is not specified or the backup files need to be upgraded to version 2. Please use cbdbupgrade for upgrade or contact support." But using cbbackup on snapshot data and restoring afterwards works."
Perry Krug made changes -
Resolution Fixed [ 1 ]
Status Resolved [ 5 ] Reopened [ 4 ]
Perry Krug made changes -
Affects Version/s 1.8.0 [ 10248 ]
Steve Yen made changes -
Assignee Bin Cui [ bcui ] Steve Yen [ steve ]
Peter Wansch made changes -
Component/s tools [ 10223 ]
Component/s cli [ 10011 ]
Hide
Permalink
Peter Wansch added a comment -
Steve, since the workaround did the trick, can this issue be closed?
Show
Peter Wansch added a comment - Steve, since the workaround did the trick, can this issue be closed?
Hide
Permalink
Steve Yen added a comment -
Additionally, as part of the workaround, the user should also run "pragma wal_checkpoint(FULL)" to incorporate any WAL files before changing the journal_mode.

So...

  pragma wal_checkpoint(FULL);
  pragma journal_mode=delete;

See...

  http://www.sqlite.org/pragma.html#pragma_wal_checkpoint

One last workaround, possibly already mentioned, is to run cbrestore on a box with updated python and python sqlite3 binding.
Show
Steve Yen added a comment - Additionally, as part of the workaround, the user should also run "pragma wal_checkpoint(FULL)" to incorporate any WAL files before changing the journal_mode. So...   pragma wal_checkpoint(FULL);   pragma journal_mode=delete; See...    http://www.sqlite.org/pragma.html#pragma_wal_checkpoint One last workaround, possibly already mentioned, is to run cbrestore on a box with updated python and python sqlite3 binding.
Steve Yen made changes -
Status Reopened [ 4 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Steve Yen made changes -
Resolution Fixed [ 1 ]
Status Resolved [ 5 ] Reopened [ 4 ]
Hide
Permalink
Steve Yen added a comment -
Related to Perry's April 13, 2012 comment, another workaround to handle the "backup files need to be upgrade to version 2" issue...

IF the user can determine that the files _definitely_ came from the right version (e.g., backup of 1.8.1 and trying to restore to 1.8.1), then perhaps the user_version pragma information did not transfer correctly.

For example, use sqlite3 to run...

  pragma user_version;

The result should be '2'. If not then a workaround fix would be, in sqlite3 to use the following across all the sqlite files...

  pragma user_version=2;
Show
Steve Yen added a comment - Related to Perry's April 13, 2012 comment, another workaround to handle the "backup files need to be upgrade to version 2" issue... IF the user can determine that the files _definitely_ came from the right version (e.g., backup of 1.8.1 and trying to restore to 1.8.1), then perhaps the user_version pragma information did not transfer correctly. For example, use sqlite3 to run...   pragma user_version; The result should be '2'. If not then a workaround fix would be, in sqlite3 to use the following across all the sqlite files...   pragma user_version=2;
Steve Yen made changes -
Status Reopened [ 4 ] Resolved [ 5 ]
Resolution Won't Fix [ 2 ]

People

  • Assignee:
    Steve Yen
    Reporter:
    Perry Krug
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: