Backupping membase 1.6.5

 That's pretty odd...the mbbackup script uses the sqlite backup API which is designed to be done on a live running database.  Is it possible your user permissions are not sufficient enough?

Perry

 Hello Perry, sorry for the late reply to your post. Quite busy arround here.

Which permissions are you talking about perry ? I tried backupping as root and as the user membase, and both are giving the errors that its locked.

And all the files that belong to membase are of the user membase and group membase.

 Yes, those were the permissions I was referring to...

I really don't know what's going on, I was able to test this myself on 1.6.5 with success.

Can you give me the output of 'ls -lR /membase/'?

Also, this shouldn't be a problem, but are you writing constantly into the database or is it "idle"/read-only?

Thanks

Perry

/membase/: total 24
drwxr-xr-x 4 membase membase 4096 2011-03-04 16:15 backups
drwx------ 2 membase membase 16384 2011-01-11 14:56 lost+found
drwxr-xr-x 2 membase membase 4096 2011-03-07 21:16 ns_1

/membase/backups: total 8
drwxr-xr-x 2 membase membase 4096 2011-03-04 16:16 fri_mar_4_15_15_51
drwxr-xr-x 2 membase membase 4096 2011-02-22 17:20 tue_feb_22_16_00_26

/membase/backups/fri_mar_4_15_15_51: total 162940
-rw-r--r-- 1 membase membase 45056 2011-03-04 16:15 avatars
-rw-r--r-- 1 membase membase 45056 2011-03-04 16:16 Avatars
-rw-r--r-- 1 membase membase 0 2011-03-04 16:16 avatars-0.mb
-rw-r--r-- 1 membase membase 307200 2011-03-04 16:16 Avatars-0.mb
-rw-r--r-- 1 membase membase 307200 2011-03-04 16:16 Avatars-1.mb
-rw-r--r-- 1 membase membase 45056 2011-03-04 16:16 sessions
-rw-r--r-- 1 membase membase 89729024 2011-03-04 16:16 sessions-0.mb
-rw-r--r-- 1 membase membase 45056 2011-03-04 16:16 test
-rw-r--r-- 1 membase membase 76324864 2011-03-04 16:16 test-0.mb

/membase/backups/tue_feb_22_16_00_26: total 210048
-rw-r--r-- 1 membase membase 45056 2011-02-22 17:00 avatars
-rw-r--r-- 1 membase membase 45056 2011-02-22 17:20 Avatars
-rw-r--r-- 1 membase membase 0 2011-02-22 17:20 avatars-0.mb
-rw-r--r-- 1 membase membase 307200 2011-02-22 17:20 Avatars-0.mb
-rw-r--r-- 1 membase membase 307200 2011-02-22 17:20 Avatars-1.mb
-rw-r--r-- 1 membase membase 307200 2011-02-22 17:20 Avatars-2.mb
-rw-r--r-- 1 membase membase 307200 2011-02-22 17:20 Avatars-3.mb
-rw-r--r-- 1 membase membase 45056 2011-02-22 17:20 sessions
-rw-r--r-- 1 membase membase 53033984 2011-02-22 17:20 sessions-0.mb
-rw-r--r-- 1 membase membase 0 2011-02-22 17:20 sessions-1.mb
-rw-r--r-- 1 membase membase 45056 2011-02-22 17:20 test
-rw-r--r-- 1 membase membase 39450624 2011-02-22 17:20 test-0.mb
-rw-r--r-- 1 membase membase 40585216 2011-02-22 17:20 test-1.mb
-rw-r--r-- 1 membase membase 40585216 2011-02-22 17:20 test-2.mb
-rw-r--r-- 1 membase membase 40017920 2011-02-22 17:20 test-3.mb

/membase/lost+found: total 0 /membase/ns_1: total 157537492
-rw-r--r-- 1 membase membase 45056 2011-03-07 21:15 avatars
-rw-r--r-- 1 membase membase 45056 2011-02-11 15:34 Avatars
-rw-r--r-- 1 membase membase 40080152576 2011-03-07 21:16 avatars-0.mb
-rw-r--r-- 1 membase membase 307200 2011-02-11 15:16 Avatars-0.mb
-rw-r--r-- 1 membase membase 40191405056 2011-03-07 21:16 avatars-1.mb
-rw-r--r-- 1 membase membase 307200 2011-02-11 15:16 Avatars-1.mb
-rw-r--r-- 1 membase membase 40178411520 2011-03-07 21:16 avatars-2.mb
-rw-r--r-- 1 membase membase 307200 2011-02-11 15:16 Avatars-2.mb
-rw-r--r-- 1 membase membase 40163972096 2011-03-07 21:16 avatars-3.mb
-rw-r--r-- 1 membase membase 307200 2011-02-11 15:16 Avatars-3.mb
-rw-r--r-- 1 membase membase 46080 2011-03-07 21:15 sessions
-rw-r--r-- 1 membase membase 97074176 2011-03-07 21:16 sessions-0.mb
-rw-r--r-- 1 membase membase 95870976 2011-03-07 21:16 sessions-1.mb
-rw-r--r-- 1 membase membase 97060864 2011-03-07 21:16 sessions-2.mb
-rw-r--r-- 1 membase membase 97207296 2011-03-07 21:16 sessions-3.mb
-rw-r--r-- 1 membase membase 45056 2011-03-07 21:15 test
-rw-r--r-- 1 membase membase 76324864 2011-02-28 17:31 test-0.mb
-rw-r--r-- 1 membase membase 81430528 2011-02-28 17:31 test-1.mb
-rw-r--r-- 1 membase membase 81430528 2011-02-28 17:31 test-2.mb
-rw-r--r-- 1 membase membase 76324864 2011-02-28 17:31 test-3.mb

Also what quite useful to know is, i wrote a backup script that i have to run and it backups all the buckets automatically. script source :
#!/bin/bash
#DATE format : tue_feb_22_15_20_50
DATE=`/bin/date -u |/usr/bin/awk '{ print $1, $2, $3, $4; }' | sed s/" "/_/g | sed s/:/_/g | awk {'print tolower($_)'}`
# Membase vbucket location LOCATION=/membase/ns_1
# membase backup location BACKUPDIR=/membase/backups
#Management directiory of membase MANAGEMENTDIR=/opt/membase/1.6.5/bin/ep_engine/management/
#backuping all buckets.... BUCKETS=`ls -l $LOCATION |awk {'print $8'} |egrep -v "*.mb|*-[a-zA-Z1-0]|^$"`
if [ ! -d $MANAGEMENTDIR ]; then
echo "Cannot find membase 1.6.5 management directory"
exit 1
fi
if [ ! -f $MANAGEMENTDIR/mbbackup ]; then
echo "Cannot find membase 1.6.5 mbackup script"
exit 1
fi
if [ ! -d $BACKUPDIR/$DATE ]; then
mkdir $BACKUPDIR/$DATE
fi
if [ ! -d $BACKUPDIR/$DATE ]; then
echo "I didnt create $BACKUPDIR/$DATE please fix me"
exit 1
else
for bucket in $BUCKETS
do
cd $MANAGEMENTDIR
/usr/bin/python mbbackup $LOCATION/$bucket $BACKUPDIR/$DATE
echo "Backup for $bucket complete it is located in $BACKUPDIR/$DATE"
done
fi

But also when i run the command manually, i get the same errors. - Siebren

Do you get the errors for all buckets or just the sessions one?

Can you try setting up a "test" server and seeing if you can reproduce this there?

Perry

I am able to setup a test server with membase 1.6.5. I need to do this at the office tough, Currently i'm not.
When i get the test results, ill post them in this thread.

I'll keep you posted.

-
Siebren

Hello Perry,

I tried to reproduce the problem on our test server and the backup script i used on our production server and it is working on the test server.

Output :

membase@membasetest:/$ bash backupMembaseTest.sh
Back-upping buckets : test
Backup of test done
Vacuum of test done
Backup of test-0.mb done
Vacuum of test-0.mb done
Backup of test-1.mb done
Vacuum of test-1.mb done
Backup of test-2.mb done
Vacuum of test-2.mb done
Backup of test-3.mb done
Vacuum of test-3.mb done
Backup for test complete it is located in /membase/backups/tue_mar_8_10_36_44

Only the configuration of the membase files is different.

I'll try to make it identical to our production server and test it again.

Even with an identical setup it still works

membase@membasetest:/home/gp$ bash backupMembaseTest.sh
test2
Backup of test2 done
Vacuum of test2 done
Backup of test2-0.mb done
Vacuum of test2-0.mb done
Backup of test2-1.mb done
Vacuum of test2-1.mb done
Backup of test2-2.mb done
Vacuum of test2-2.mb done
Backup of test2-3.mb done
Vacuum of test2-3.mb done
Backup for test2 complete it is located in /membase/backups/tue_mar_8_11_14_40

Any suggestions ?

 Okay, so at least we know it works overall, now it's just about figuring out why it doesn't work on your other deployment.  

Is the other one in prouduction?  if not, could you shut the service down (/etc/init.d/membase-server stop) and then try the backup script again?  If that still doesn't work, then at least we know it's not a problem with the software itself...

Thanks for your patience and cooperation.

Perry

 Yes it is running in production.

 Okay, then lets try to diagnose this "non-disruptively" :-)

Can you run this command:

sudo /opt/membase/bin/ep_engine/management/sqlite3 /membase/ns_1/sessions-0.mb '.backup /membase/backup/<timestamp>/sessions-0/mb'

I don't believe sqlite will print out an error in some cases, so check the size of the resulting file to know whether it worked properly or not.

Perry

That seems to work,

ls -alh output
membase@membase:/membase/backups/testbackup09_09_2011_11_28_35# ls -alh
total 94M
drwxr-xr-x 2 membase membase 4.0K 2011-03-09 11:32 .
drwxr-xr-x 5 membase membase 4.0K 2011-03-09 11:29 ..
-rw-r--r-- 1 membase membase 94M 2011-03-09 11:32 sessions-0.mb

original file :
membase@membase:/membase/ns_1# ls -alh |grep sessions-0
-rw-r--r-- 1 membase membase 94M 2011-03-09 11:33 sessions-0.mb

command i ran to backup

/opt/membase/bin/ep_engine/management/sqlite3 /membase/ns_1/sessions-0.mb '.backup /membase/backups/testbackup09_09_2011_11_28_35/sessions-0.mb'

 Okay.  I'm sorry for the repetition, but can you confirm that this does NOT work (it "should" be calling the same thing but could be a bug in the mbbackup script):

cd /opt/membase/1.6.5/bin/ep_engine/management/

./mbbackup /membase/ns_1/sessions /membase/backups/<timestamp>/

If that doesn't work, could you grab the latest mbbackup script from the source code and try with that?

Thanks

Hello Perry,

I tried it again with the mbbackup script of 1.6.5 and it fails again.

membase@db10:~/1.6.5/bin/ep_engine/management$ ./mbbackup /membase/ns_1/sessions /membase/backups/
Backup of sessions done
Vacuum of sessions done
Backup of sessions-0.mb done
Vacuum of sessions-0.mb done
Error: database is locked
Try backup sessions-1.mb again
Error: database is locked

With the new mbbackup script located in the source code it fails as well.

membase@db10:~/1.6.5/bin/ep_engine/management$ ./mbbackup /membase/ns_1/sessions /membase/backups/testbackup_10_03_2011_2/
Backup of sessions done
Vacuum of sessions done
Backup of sessions-0.mb done
Vacuum of sessions-0.mb done
Error: database is locked
Try backup sessions-1.mb again

 Hi, Siebren. Can you tell us what kind of file system this is? And what OS?

There are reports of locking issues when using sqlite on NFS or other networked file systems. For example:

 ubuntuforums.org/showthread.php?t=780891

If this is the case for you, a first step would be to test some very simple sqlite commands on the same filesystem to verify that this is the cause, and then to look for solutions specific to your filesystem. There may be a fix in a more recent version of the filesystem, or you may be able to access the disk locally.

Tim

uname -a :
Linux db10 2.6.32-24-server #39-Ubuntu SMP Wed Jul 28 06:21:40 UTC 2010 x86_64 GNU/Linux

with local storage, no nfs mounts on the server what so ever.

Also, i am not able to restart the server for updates. Since it is running in production and we don't have a second server that we can re-balance to, at least not yet.
And the session bucket shows no problem in the logs as far as i can see them.

 No, the problems don't seem related to the performance of the system...just something of a weird state on that bucket.  It would be really good if you could add another node to the cluster and remove this one with a rebalance.

Perry

 Also, based upon the previous posts, you can successfully back up this system if need be.  It's a manual process using the sqlite commands, but the functionality is the same so you're not in any trouble.

Perry