Trying to run multiple replicators in parellel to speed up the download resulting in "database is locked" error every now and then

In our usecase we have around 3 millions db rows to download which takes around 2 hours which is a bit unacceptable in our usecase. So to speed up the process we decided to run Multiple replicators in parellel but that leads to about 10% of the inserts failing with the following message.

{N8litecore4repl8InserterE#42} Failed to insert ‘781346ec-60a7-4edb-b4fc-ff9baxe7f68c’ #1-x345ascf8f3f4e3c3e3cf97fe1b20493abd5e : LiteCore Busy, “database busy/locked”

Looks like when a thread is writing to the DB then other threads are unable to write. But the thread which failed to write instead of either waiting or reattempting the insertion, it just moves on to the next row

I don’t think that is going to work. The database is a file on the file system. It cannot be updated by multiple threads at the same time.

Let me see if I can find an expert to suggest alternatives…

Unfortunately I’ve also run into this problem using just one replicator during long running replications.
It’s rare but when it happens clients that have completed replication and you expect to have fully updated documents instead have randomly outdated documents. The problem is that the replicator never retries failed documents when this happens. I would expect it to retry on the next replication attempt. Currently the only way I’ve found to force it to retry a document is to create and save another revision of the failed document on the server.

@hyling If you have encountered either a locked database during replication or randomly updated documents, please report a bug. Neither one of those things should happen, ever.

I do have another thread that writes to the database while replication is happening. I assumed it was the same problem you described above, multiple threads can’t write to the database file at the same time?

Yeah, in that case, you are right.

Unlike large DB systems, SQLite is just a complicated way of writing to a file. The OS won’t let multiple threads do that at the same time…