Prebuilt Database copy not completing

Mobile Couchbase Lite
21

Thanks for that, @dshyu. Logs (adb logcat) and a corrupted database would be helpful.

22

@blake.meike, is your email the same form as jianmin’s? I’ll share the databases and logs in the onedrive I shared with him.

23

Yes. Thanks a bunch.

24

@blake.meike,

It looks like I misspoke. I think when I still had the database config directory set for internal storage, I saw the database copy successfully copy over all 8 GB. However, now that I have the directory set for external storage, the database copy is only bringing over 205 MB.

I also sent you a link to the onedrive with the corrupted database as well as the logs that I pulled from that run.

25

Oh! That’s a really good clue. Just to confirm, this is emulated external storage, not actually a removable card, right?

Got the DB and logs. Thanks.

26

Correct. This is part of the 512 GB base available to our S10.

27

I’m sorry… Can you confirm that? I see a Samsung S10 as having either 128 or 256 GB total RAM, with support for up to 512 G external card.

28

The phone model number is SM-G973U1 so I’m pretty sure it’s the unlocked version.

The specs listed here for the unlocked S10 include one with 512 GB Storage and 8GB Ram.

29

Great. Thanks for the confirmation!

30

@dshyu : I cannot repro this. I had some ideas but they have not proved useful.

The standard procedure for using a database that is embedded in an application goes like this:

  1. The db is packaged in the app as a zip file: unzip it
  2. Use Database.copy(path, name, config) to create a unique instance of the db, somewhere.
  3. Delete the original, unpacked db.
  4. Open the new, unique instance.

I’m guessing that this is, more or less, what you are doing (as suggested by the TravelSample code that you reference above)

I would like to know what happens if you, instead, open the unzipped database, directly, before you copy it (step #2).

I would also be very interested in knowing if there was any difference between unzipping the file directly to internal storage (the application sandbox, Context.getFilesDir(), usually /data/data/<your app's package name>/files, or, perhaps /data/user/<small number>/<your app's package name>/files) as opposed to what happens if you copy it to external storage (Context.getExternalFilesDir("dbs"), somewhere like: /storage/emulated/0/Android/data/<your app's package name>/files/dbs)

Since we have been unable to reproduce this, your help is very definitely appreciated.

31

@blake.meike ,

So the way I was running this, and it seemed to work on small database sizes (5 to 10 entries) was:

When the database directory was still in internal storage (data/user/0/com./files/)

  1. Sync down data from gateway and generate sync.cblite2 folder.
  2. Copy sync.cblite2 folder to external storage
  3. Clear data from app and uninstall app.
  4. Reinstall app. App checks external storage for the existence of the cblite2 folder.
  5. App makes Database.copy call
  6. App makes new Database call after copy finishes

I also tried setting the config directory for the database to external storage so I could skip step 2 however that hasn’t worked either.

So I am differing in that I never zip/unzip the database and I don’t delete the original unpacked db. I can try deleting the original database before making the new Database call. I can also try zipping and db and then unzipping it to internal storage.

Can you clarify what you mean by opening the unzipped database before copying it? Do you want me to use the cblite inspector?

32

Yeah. Sorry for the lack of clarity.

The call to Database.copy is important. It’s main purpose is that it gives a particular DB instance (the specific copy DB now installed on this particular device) a unique ID. That’s important so that the SyncGateway can optimize its choice of which updates to send to the device.

Before the copy, though, the database should be perfectly functional. What I am hoping you can do is to open the DB, as is, in external storage, with no call to copy. I’m trying to figure out exactly when the DB is getting corrupted: the copy to external storage, or the copy back. If you just open the device in external storage, we should be able to separate the two scenarios.

Btw, I’m going to verify this but using Database.copy may not be the right thing to do in your case. The DB that you copy to external storage already has an ID unique to the device. When you copy it to external storage, you give it a new ID and when you copy it back you change that ID yet again. While this shouldn’t cause any serious problems, it is going to mean that the SGW is going to treat it as a new, previously un-seen device, forcing it to do a complete, un-optimized update.

1 Like
33

Ok so have the cblite2 file in external storage and then use new Database with that location in external storage defined in the config?

Just to confirm, the config for the preloaded database and the app that’s loading the preloading database has to match correct? It just occurred to me that while switching baselines and testing, I may have forgotten to match directories in the configs.

34

Yes. Please just open, directly, the db that you copied to external storage. You should be able to do that like this:

Database db = new Database(
    "myDb",
    new DatabaseConfiguration(ctxt..getExternalFilesDir("dbBackup").getAbsolutePath()));

… and, yes: new Database("myDb", newConfig("/foo/bar")) will open a database named “/foo/bar/myDb.cblite2”, if it exists. If it does not exist, it will create it (if it can). If your config points to the wrong place, you will just get a new database.

35

That worked. After creating a db and then just opening it without the copy call in the new instance of the app, I was able to open the db and successfully query the count.

36

@dshyu Excellent! This is the most useful bit of information that we’ve had on this issue.

I would like to ask you to try one more thing, if you would: Open the db on the external store, as you are doing now. Then close it, copy it to internal storage (as you were doing before) and open it again in the internal store. If you would try that a few times and confirm that the external copy is unlikely to be corrupt, but the internal copy IS likely to be corrupt, we will really have something to work with.

Again, your efforts are hugely appreciated

37

Just want to make sure I’m testing the right thing.

  1. Repeat the previous test where I just open the database without copying it.
  2. Close the database with Database.close call.
  3. Same instance of app (or can this be a new instance of the app), use Database.copy call.
  4. Confirm Database corruption.

I’m a little hung up on the fact that right now, the database config has the directory set to external storage. If I change the database config to go to internal storage, wouldn’t that cause issues?

38

Yeah… you have it exactly right.

If I understand, correctly, you are invoking Database.copy twice:

  1. after a complete sync, you copy the DB to external storage
  2. after you uninstall and re-install the app, you copy the DB from external storage, back to internal storage

It is my understanding that, after step #2, the DB is frequently corrupted. I would like to know it it is already corrupted after step #1

As for the database config: you cannot modify the configuration that a database holds, if that is your concern. When you hand a configuration to an instance of a Database, it gets copied. If you ask a database for its configuration, what you get is a copy. You cannot affect the configuration of a Database, including its directory, by changing something in a DatabaseConfiguration object.

39

Sorry for the misunderstanding but first copy of the DB to external storage was done through ADB terminal:

adb exec-out run-as {app package} cp -R /data/user/0/{app package}/files/sync.cblite2 /storage/emulated/0

This was a work around to pull the DB off a phone when it was stored in internal memory. Since it was tied to the instance of the app, we couldn’t ever see the DB unless it was moved to external storage.

The second copy is correct but there are a two test cases that I want to further explain so I don’t confuse the matter.

Test Case 1: DatabaseConfiguration path is default or set to internal memory (/data/user/0/{app package})

  1. Pull database from internal memory to external memory as described above
  2. Reinstall app.
  3. Call database.copy targeting database location in external memory.

Test Case 2: DatabaseConfiguration path is set to somewhere in external memory (Environment.getExternalStorageDirectory())

  1. Move DB in external memory to temporary folder in external memory
  2. Reinstall app.
  3. Call database.copy targeting database location in external memory.
1 Like
40

Got it. Thanks for the clarification.

I think that we are pretty certain that your Test Case 1 reveals DB corruption. When you copy the DB into internal memory, using Database.copy, after copying it out to external memory (using ADB), you frequently find that the DB is corrupt.

I don’t think the test you describe in test case #2 is of that much interest.

The testcase in which I am most interested, at this point, is:

Test Case #3

  1. Pull database from internal memory to external memory as described above
  2. Reinstall app.
  3. Open the database at its current location in external memory. No call to Database.copy. Just build a DB config that points to the external directory.

There is one other test that might be of interest. Especially given our current hypotheses about the problem, it is not essential, but it might be interesting;

Test Case #4:

  1. Pull database from internal memory to external memory as described above
  2. Reinstall app.
  3. Copy the database from external memory to internal memory, using ADB. Do not use Database.copy to move it.

Your help on this has been invaluable. We may have something for you to try out, shortly.

1 Like