ForestDB, Background Fetch, Crashes when compacting

Mobile Couchbase Lite
1

Using:

  • couchbase-lite-ios V1.4 via podfile with ForestDB
    or
  • couchbase-lite-ios self-built master(1d4b2c28) with forestdb spock(9c4bf80)

In my app i’m experiencing frequent crashes while compaction when running or started in background (e.g. via background fetch).

Stacktrace:

Crashed: Thread
0  PurpleAnt                      0x1007cfe38 std::__1::__hash_const_iterator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool>, void*>*> std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool> > >::find<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const (__hash_table:2206)
1  PurpleAnt                      0x1007ce8b0 cbforest::Database::getKeyStore(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) const (unordered_map:1121)
2  PurpleAnt                      0x1007cefa4 cbforest::Database::updatePurgeCount() (Database.cc:227)
3  PurpleAnt                      0x1007ce624 cbforest::Database::compactionCallback(_fdb_file_handle*, unsigned int, char const*, fdb_doc_struct*, unsigned long long, unsigned long long, void*) (Database.cc:406)
4  PurpleAnt                      0x100813458 _fdb_compact_file(_fdb_kvs_handle*, filemgr*, btreeblk_handle*, docio_handle*, hbtrie*, hbtrie*, btree*, btree*, unsigned long long, bool) + 6636
5  PurpleAnt                      0x100812438 fdb_compact_file + 2508
6  PurpleAnt                      0x1007fe8cc compactor_thread(void*) + 21856
7  libsystem_pthread.dylib        0x18a3e568c _pthread_body + 240
8  libsystem_pthread.dylib        0x18a3e559c _pthread_body + 282
9  libsystem_pthread.dylib        0x18a3e2cb4 thread_start + 4

I also tried with a self-built version using couchbase-lite-ios master with a forestdb from “spock” branch to include latest fixes, especially this one: https://github.com/couchbase/forestdb/commit/5827abea5028bf05e2f00b14b7d22087213dbb27
but everything that changed was the line number in which the error happens: previously 2228, now 2206

Is it somehow possible to disable automatic compaction? As a workaround i would disable compaction when app is going into background. The error doesn’t seem to occur when compaction is running while the app is active.

Where to file a bug report for this issue? ForestDB issues seem not to be tracked in GitHub.

2

I’d recommend not using forestdb - IIRC recent releases of Couchbase Lite are not using it.

3

File an issue against the couchbase/cbforest repo, please. This isn’t a bug in ForestDB itself, rather in cbforest, a C++ wrapper library for it that’s part of Couchbase Lite.

Also, as Dave said, we’ve decided not to use ForestDB going forwards; we’ll support it in Couchbase Lite 1.4 until that release’s support period ends (a year after its release), but we’re not using it in 2.0.

4

Thanks for your hint about couchbase/cbforest. Will file an issue there in a few moments.

AFAIK you’re not only dropping ForestDB but also compatibility with the “old” sync protocol with Cloudant which is one of my main requirements. So i’m stuck on Version 1.4 and i’m also facing loss of support soon :frowning: . (Btw when was version 1.4 officially released?)

If 2.0 was supporting the old sync protocol i virtually would start migration right now.

5

I’d recommend switching to SQLite-based databases in 1.4, if you’re having trouble with ForestDB.

6

Thanks. Could you just give me some more background for this recommendation?
What is the reason (besides “avoiding problems with ForestDB”)?

I’m just asking because i am working on a quite complex app (nearly 35,000 lines of Swift code according to cloc, excluding third-party frameworks) managing multiple cbl databases. I already finished development stage of the app some weeks ago and i am currently (friendly-user) testing and bugfixing. Currently my only problem with ForestDB are those crashes while compaction. In the last about 2 weeks of testing there have been no other crashes and there are only some little bugs regarding functionality left.
I am using ForestDB because there was no encryption with SQLite when development of the app began (cbl version 1.3) and the documentation stated:

"Benefits of using ForestDB

  • Faster (2x to 5x as fast, depending on the operation and data set)
  • Better concurrency (writers never block readers)
  • Lower RAM footprint (data caches are shared between threads)
  • Database compaction is automatic and runs periodically in the background"

(Which is still online, btw)

I hope you understand that some further explanation would really be helpful for me to assess the situation.
Are the above statements regarding performance, concurrency, footprint still true?
What (other) sort of problems would you expect when switching to SQLite (except migrating already existing databases from forestdb to sqlite)?

Do you have any assumptions regarding the cause of the compaction bug? I’m not a C++ developer and i find C++ code and stacktraces really nasty to read but with a little help i maybe would be able to fix the problem by myself.

7

We’ve always supported SQLite encryption; you just have to link SQLCipher into your app instead of using the OS’s SQLite library. (SQLCipher is a 3rd party version of SQLite with encryption support.)

I suggested switching to SQLite to work around this crash. Also, in the long run we’re staying with SQLite instead of ForestDB (2.0’s use of SQLite is much faster than 1.x.)

The crash looks like a thread-safety problem, unfortunately. The ForestDB compactor is running on a background thread and notifying our code from there, and our notification handler is unfortunately calling another function that’s not thread-safe. This won’t be a simple fix.

8

For future reference, I see that the issue you filed is CBForest#119. Thanks for the report.

9

OK, thanks for your explanation. I think, i’ll give it a try with SQLite (can’t remember why i was thinking encryption was not available with SQLite storage in 1.3, must be mixing something up).
I hope the impact on performance will not be too bad. But it sounds like SQLite is more reliable and getting solutions/bugfixes would be much quicker and easier when using SQLite.

Just saw your suggestion on GitHub, thanks. Will try that if SQLite doesn’t work for me for some reason!

10

Yeah, sorry about the bug. If you don’t use document purging, and you’re OK with building your own Couchbase Lite, I think commenting out that one line in CBForest would be the least destabilizing change right now.