I have a pre-built db with size of 12GB, after I call ChangeEncryptionKey function to change the key, I got time out error,
SqlitePCLRawStorageEngine
ExecSQL timed out waiting for Task #12
AtomicAction: Error performing step #2: Couchbase.Lite.CouchbaseLiteException: ExecSQL timed out
at Couchbase.Lite.SqlitePCLRawStorageEngine.ExecSQL(String sql, sqlite3 db, Object[] paramArgs)
at Couchbase.Lite.SqlitePCLRawStorageEngine.ExecSQL(String sql, Object[] paramArgs)
at Couchbase.Lite.Store.SqliteCouchStore.c__AnonStoreyA.<>m__3()
at Couchbase.Lite.Util.AtomicAction.Perform()
SqlitePCLRawStorageEngine
ExecSQL timed out waiting for Task #13
AtomicAction: Error backing out step #1: Couchbase.Lite.CouchbaseLiteException: ExecSQL timed out
at Couchbase.Lite.SqlitePCLRawStorageEngine.ExecSQL(String sql, sqlite3 db, Object[] paramArgs)
at Couchbase.Lite.SqlitePCLRawStorageEngine.ExecSQL(String sql, Object[] paramArgs)
at Couchbase.Lite.Store.SqliteCouchStore.c__AnonStoreyA.<>m__2()
at Couchbase.Lite.Util.AtomicAction.BackOut()
Unhandled Exception: Couchbase.Lite.CouchbaseLiteException: Couchbase Lite error: Exception —> Couchbase.Lite.CouchbaseLiteException:
ExecSQL timed out
at Couchbase.Lite.SqlitePCLRawStorageEngine.ExecSQL(String sql, sqlite3 db, Object[] paramArgs)
at Couchbase.Lite.SqlitePCLRawStorageEngine.ExecSQL(String sql, Object[] paramArgs)
at Couchbase.Lite.Store.SqliteCouchStore.c__AnonStoreyA.<>m__2()
at Couchbase.Lite.Util.AtomicAction.BackOut()
— End of inner exception stack trace —
at Couchbase.Lite.Util.AtomicAction.BackOut()
at Couchbase.Lite.Util.AtomicAction.Perform()
at Couchbase.Lite.Database.ChangeEncryptionKey(SymmetricKey newKey)
What platform? I can’t tell from the stack if this is Java or C#…
It sounds like some layer of the code has a timeout for SQL calls, and the rekeying is taking longer than that because the database is so huge. The solution is probably to increase or eliminate the timeout.
I suggest filing a bug on Github against the appropriate couchbase-lite component.
It is C# 1.2.1.1, I checked the code and saw a fix me:
try
{ //FIXME.JHB: This wait should be optional (API change)
t.Wait(30000, _cts.Token);
}
in github.
The timeout is still there, but there have been some optimizations put into place since 1.2 so try again with build 143 from http://latestbuilds.hq.couchbase.com/couchbase-lite-net and see if it still happens. If it does, then file an issue on the repo.
After package with build 143, I still get time out message, however, the error message is not same as 1.2.1.1:
database count 93293
Unhandled Exception: Couchbase.Lite.CouchbaseLiteException: Couchbase Lite error: Exception —> Couchbase.Lite.CouchbaseLiteException: ExecSQL t
imed out waiting for Task #21
at Couchbase.Lite.Storage.SQLCipher.SqlitePCLRawStorageEngine.ExecSQL(String sql, sqlite3 db, Object[] paramArgs)
at Couchbase.Lite.Storage.SQLCipher.SqlitePCLRawStorageEngine.ExecSQL(String sql, Object[] paramArgs)
at Couchbase.Lite.Storage.SQLCipher.SqliteCouchStore.c__AnonStoreyA.<>m__3()
at Couchbase.Lite.Util.AtomicAction.Perform()
— End of inner exception stack trace —
at Couchbase.Lite.Util.AtomicAction.Perform()
at Couchbase.Lite.Database.ChangeEncryptionKey(SymmetricKey newKey)
at fullmaptiles_copy.Program.Main(String[] args) in D:\GitHub\mobent\fullmaptiles-copy\fullmaptiles-copy\Source\Program.cs:line 267
That’s still the same issue. Well, even copying 12 GB of data straight up is bound to take more than 30 seconds so I can’t imagine how long it’s going to take to decrypt and reencrypt the entire thing. I’ll probably get rid of that timeout since it’s usefulness is questionable at this point (It is supposed to prevent deadlocks but I haven’t seen it deadlock there in an extremely long time). File an issue about it on the github repo.
Jim, my gut feeling is there shouldn’t be any timeout on SQL operations, because you can reasonably expect them to complete (unlike network operations that might never get a reply.)
On the other hand, a 12GB database is awfully large for Couchbase Lite; we didn’t design it for that sort of scale. @zheng.gong, what’s making it that big? Huge numbers of documents, or very large documents? Can you move some of the large data into attachments? That would make the SQLite database smaller.
We have the data for map like data, 12G is just a modest dataset for us. We subdivided the data to separate docs so we only need access couple of them at time. For off line situation, we do need all the data on the device. We have about 100K documents for 12G cb lite, it is 5G in couchbase server. Document size varies up to 15mb. I read that attachment has size limit of 10mb for sync gateway, so attachment seems more hard to handle. It will be good if your company can provide a guideline how and limitations for using Couchbase Lite, like size etc. For 100K documents, we expect that up to 1% will be changed daily and sync through sync gateway, we will test that performance later.
The attachment size limit on Sync Gateway is 20MB, not 10MB. (This is actually a Couchbase Server limitation.)
15MB is pretty large for a document, since documents are read from and written to the database in their entirety, and the replicator sends the entire document as an update (not a delta.) Consider putting just the data that needs to be queried or accessed quickly in the document, and the rest in attachments. Also consider breaking up documents into smaller documents.
In addition, a document can contain attachments, named binary blobs that are useful for storing large media files or other non-textual data. Couchbase Lite supports attachments of unlimited size, although the Sync Gateway currently imposes a 10MB limit for attachments synced to it.