What is the proper way to install a pre-built database

Following a discussion with @priya.rajagopal about the proper way of creating a working database in your app using a prebuilt database, I’ve become uncertain if I’m doing it the right way - or perhaps has an issue that needs to be fixed.

I have used code provided by @borrrden from 2017 for (iOs and Android) :slight_smile:

Basically, I have zipped the database (a clean, newly created db that has finished sync.) into the app package, and then on first launch I unpack it an copies it into place. This is the code I use (here for iOS):

public class DatabaseSeedService : IDatabaseSeedService
{
    const string ZIP_NAME = "angler.zip";
    public async Task CopyDatabaseAsync(string directoryPath)
    {
        Directory.CreateDirectory(directoryPath);
        var zipFile = Path.Combine(NSBundle.MainBundle.ResourcePath, ZIP_NAME);
        var assetStream = File.OpenRead(zipFile);
        using (var archive = new ZipArchive(assetStream, ZipArchiveMode.Read))
        {
            foreach (var entry in archive.Entries)
            {
                var entryPath = Path.Combine(directoryPath, entry.FullName);
                if (entryPath.EndsWith("/", System.StringComparison.Ordinal))
                {
                    Directory.CreateDirectory(entryPath);
                }
                else
                {
                    using (var entryStream = entry.Open())
                    using (var writeStream = File.OpenWrite(entryPath))
                    {
                        await entryStream.CopyToAsync(writeStream).ConfigureAwait(false);
                    }
                }
            }
        }
    }

But from what you say I’m a little concerned that this code does give me an issue?

  1. Should I unpack the database and then copy it and delete the first instance?
  2. Should I do something to all of the apps already running out there? If so what can I do?

I’m now using CB Lite 2.8.4 (Community Edition) and Xamarin.Forms for my app.

Hi @blake.meike

Well, the question was moved from another thread to avoid hijacking it - so perhaps some of the content has gone missing. But what concerned me was that using the code above it seems that I’m missing a database.copy() method in the flow. So the second question is really what I should do (if anything) to all of the instances of the app already installed out there?

From the sample that you link to I can see that I should actually unzip the database to a temp. location/file and then do a database.copy() to put it in the right place (and then remove the temp . copy - although the sample doesn’t do that…).

Alas, I did not do a database.copy() until now - so is there anything I can/should do to the apps already installed, e.g. during startup after upgrade?

Yeah. you must do a database copy. The code to which I referred you shows that. Unzip, copy, delete unziped.

If you are saying that there are copies of your application, out there, that are not copying the db, I’m surprised that they work at all. If they don’t work, I think you’ll need to update them. If they do work, though, by some fluke… well, cool.

Ok, very interesting - and sad that I didn’t see that when originally designing this :frowning:

So, I guess I should design an upgrade task that will copy the database to a temp location and copy it back again - and remove the temp. copy…

… and just for the reference, as far as I can see, the example does not remove the temp. copy… - but perhaps I just don’t understand those utility classes in Java :wink:

I doubt that there is a reason to do that. If the applications are working, they are working. Let me verify…

1 Like

I stand corrected. You definitely should copy those databases.

The reason is that every db has a UUID associated with it. That helps the replicator tell them apart. Changes to the db from all of your applications are going to look as though they are changes from one instance of the app. That will make replication take an unnecessarily long time.

Copying the DB will change its UUID.

1 Like

Ok, thanks… Will need to figure out the best way to do that - and test it…

And I do think that I’m experiencing the “unnecessarily long time” sync. - so good that there is an explanation… Just a little sad that I didn’t know from the beginning…

Also need to fix the code for first install of the app…

But thanks for clarifying!

Not sure which “temp” copy you are referring to but the copyDatabase() API does this at a high level

  1. Copies the specified database to a temp folder.
  2. Opens the copied database
  3. Calls resetUUID() to reset UUID (This is key to preventing all the clients from having same UUID)
  4. Closes the database
  5. Moves the database from the temp directory to the destination directory.

The documentation on prebuilt database has admittedly been updated since 2017 but it clarifies the same

Use the API’s Database.Copy() method; this ensures that a UUID is generated for each copy — see: Code

Ah… got you. I was missing that the Database.copy() implicitly moves the database instead of making a copy :slight_smile:

I was assuming that if I unzip the database to eg. “DB1” and then do a Database.copy("DB1", "DB2",null) then “DB1” would still be sitting in the phone and needed to be deleted. But if I understand you correctly “DB1” is actually removed after the copy has completed and “DB2” has been created?

So to “fix” the issue in the app where the database is already installed (without Database.copy()) I really need to do a Database.copy() to a “another” file and once more back to the original name? I assume I cannot do a copy to the same location (=“itself”) to get it to set the new UUID()?

I don’t think that’s quite the correct interpretation here. There are a few things here that collide in an awkward way so to start off, the “regular” way of doing this is just to pass in a path to your read only prebuilt database artifact that is distributed as an asset along with your application. The easiest way to envision this is to think of the way iOS handles things with NSBundle. You get the path out of NSBundle, pass that to the copy API, and it will “install” the database into its final location ready to be used.

What could go wrong? Well…Android doesn’t give you paths to your assets. Instead it gives you streams and that’s not compatible with the way the API is. So the workaround to this is to extract the stream to a temporary location and use the temporary location as a path for the copy API. You will need to delete this afterwords (or you could put it in a cache directory and hope that the OS deletes it for you).

The confusing part about this explanation is that there are now 2 temp directories involved on Android vs everything else. In order to ensure atomicity the way the move is performed is by first copying the database to a system temp directory, modifying it there, and then only moving it once it succeeds since an intra-filesystem move is an atomic operation. So the flow looks like this

  • Android: Assets - copied by you → your chosen temp directory - copied by CBL → CBL chosen temp directory - moved → final location
  • Everything else: Assets - copied by CBL → CBL chosen temp directory - moved → final location

Hope this makes things clearer and I didn’t just confuse you more :smiley:

Thanks Jim

I think I get it now… The only “extra” step is that I need to unzip the “asset” before doing the Database.copy(). So I guess that makes the code behave more “similar” for both Android and iOS? … and that I have to remove the “unzipped” copy once it has been copy by CBL to the final destination - for both platforms.

What is the best approach for the databases I have already installed prior to this change? Should I just do a Database.copy() twice as I tried to line out above? First to a temp db and then from temp db back to final destination.

Or can I reset the UUID() directly?

@borrrden , any suggestions to this? Thanks!

Ok, I dived into the dotnet version of the code you linked to (also written by @borrrden :slight_smile: ). And in that code: mobile-travel-sample/LoginModel.cs at 753e5d408c49e48e58741e26ece872c0c6efac3c · couchbaselabs/mobile-travel-sample · GitHub I cannot find a Database.copy()

As far as I can see then the call to await copier.CopyDatabaseAsync(userFolder); “just” moves the builtin database into the right directory - as a “file”. But it does not perform a CBL Database.copy() - or I’ve gone blind in the search… :slight_smile: Essentially, it just does what my code above does - apart from the part where I unzip the file.

I tried to do a

Database.Copy(tmpFile,"angler",options)

where I have set the options to point at the final destination (with this code):

dbFolder = Path.Combine(defaultDirectory, "data");
if (!Directory.Exists(dbFolder))
{
   Directory.CreateDirectory(dbFolder);
}
options.Directory = dbFolder;

I have tried to use “angler” and “angler.cblite2” as file names in the copy but in both cases I end up with an error:

Couchbase.Lite.CouchbaseLiteException: CouchbaseLiteException (LiteCoreDomain / 21): file/data is not in the requested format.   

... and the stack trace:

LiteCore.Interop:
NativeHandler.ThrowOrHandle () C:\Jenkins\workspace\couchbase-lite-net-edition-build\couchbase-lite-net-ee\couchbase-lite-net\src\LiteCore\src\LiteCore.Shared\Interop\NativeHandler.cs:250

LiteCore.Interop:
NativeHandler.Execute (LiteCore.Interop.C4TryLogicDelegate1 block) C:\Jenkins\workspace\couchbase-lite-net-edition-build\couchbase-lite-net-ee\couchbase-lite-net\src\LiteCore\src\LiteCore.Shared\Interop\NativeHandler.cs:164

LiteCore:
LiteCoreBridge.Check (LiteCore.Interop.C4TryLogicDelegate1 block) C:\Jenkins\workspace\couchbase-lite-net-edition-build\couchbase-lite-net-ee\couchbase-lite-net\src\LiteCore\src\LiteCore.Shared\Interop\LiteCoreBridge.cs:32

Couchbase.Lite:
Database.Copy (System.String path, System.String name, Couchbase.Lite.DatabaseConfiguration config) C:\Jenkins\workspace\couchbase-lite-net-edition-build\couchbase-lite-net-ee\couchbase-lite-net\src\Couchbase.Lite.Shared\API\Database\Database.cs:385

Do you by any chance have an example of how to use Database.Copy() in a dotnet context. I’m using Xamarin.Forms to code the app in C#

You cannot reset the UUID manually. The easiest thing to do would be to move the folder somewhere else and then move it back in as if you were calling Database.Copy for the first time. As far as how to use it, Priya previously linked a section of our docs that is dedicated to this. You can find it here. For the first argument, you’d pass in your unzipped cblite2 folder path that is in the temp directory you chose.

Ok, I think I got it to work now by using this code:

// Borrow this functionality from Couchbase Lite
var dftFolder = Service.GetInstance<IDefaultDirectoryResolver>().DefaultDirectory();
var tmpFolder = Path.Combine(dftFolder, "tmp");
await seedService.CopyDatabaseAsync(tmpFolder);
Database.Copy(Path.Combine(tmpFolder, "angler.cblite2"), "angler", options);

I think the issue I had was that I did not have the full path to the directory to copy from entirely right.

However, the “copy” in “tmp” is not removed. As far as I understood from our discussions it should - so I guess I will have to clean it up manually to not have it taking up space…

One of the things I was hoping would disappear is a bunch of messages like this:

[DbDataStore]   Received ID: User:94D5A056E32C6065C1258499003EDFB2 (access removed)

These are probably related to the fact that the “User” type of documents have earlier been replicated to the app. As this was changed (in the sync formula in the sync .gateway) I took the database offline and ran a full resin on it. The database copy built for the “pre-built” package was then created after that.

Is there some extra “intermediate” steps I should have taken for the database not to know anything about some documents having been replicated in the past but not included anymore after it was created?

These tests were run on the iPhone simulator - if that makes any difference…

Ok, the extra messages may be irrelevant as the final number of replications seems correct:

[DbDataStore]   Sending: Doc.id: WebInfo:318, Error: Couchbase.Lite.CouchbaseWebsocketException: CouchbaseLiteException (WebSocketDomain / 403): sg admin required.
[DbDataStore]   Sending: Doc.id: WebInfo:438, Error: Couchbase.Lite.CouchbaseWebsocketException: CouchbaseLiteException (WebSocketDomain / 403): sg admin required.
[DbDataStore]   Sending: Doc.id: WebInfo:300, Error: Couchbase.Lite.CouchbaseWebsocketException: CouchbaseLiteException (WebSocketDomain / 403): sg admin required.
[DbDataStore] PushAndPull Replicator: 75978387/75978387, activity = Idle
[DbDataStore] Repl. completed on: 12-05-2021 15:09:02. Duration=00:00:06.2909540, pulls=38, pushes=0

Like the messages above telling that some WebInfo type documents could not be synced back to the server (which is correct). They have probably been removed after the pre-built database was created…

Just worried that it was a symptom of something that needed to be done - and potentially wasted extra time on the first sync?

Just for reference - if someone comes back there then this is the code that I now use to copy the database to generate the unique UUID (only on upgrade from a specific version or prior). Please note that the options object points a ‘data’ subdirectory from the root folde (the dbFolder variable)r:

// Copy to the root, remove the original db - and then copy to the 'data' folder and remove the copy in the root folder
Database.Copy(Path.Combine(dbFolder, AppConstants.DB_NAME + ".cblite2"), AppConstants.DB_NAME, null);
Database.Delete(AppConstants.DB_NAME, dbFolder);
Database.Copy(Path.Combine(rootFolder, AppConstants.DB_NAME + ".cblite2"), AppConstants.DB_NAME, options);
Database.Delete(AppConstants.DB_NAME, rootFolder);

This seems to work - and runs pretty fast so the user shouldn’t really note it…

@jda, you get extra points for saying “dived” instead of “dove”. As my high-school English teacher used to say: “A dove is a bird”

1 Like

There could be subtle nuances that I’m not able to see… - as a Dane :innocent: