[db023/SqLite error] - Replication errors

thomas1983 · March 16, 2018, 1:16pm

CBL: db023
SGW: 2.0 Beta2
CB: 5.5

Prob: Replication errors (SqLite) for bucket with 200000+, cannot open file at line xxxx
Situation: Sequential non-continuous push and pull replication of 9 buckets
Remarks: smaller buckets (1000-) seem to run fine.

Opened issue for this one: https://github.com/couchbase/couchbase-lite-net/issues/996

But I was hoping to get some input here. Does anyone else has this problem or can give a hint at the possible cause?

00:45:01
.036068| WARNING: SQLite error (code 14): 
cannot open file at line 36229 of [1a584e4999]
00:45:01.037571| WARNING: SQLite error (code 
14): os_unix.c:36229: (24) open(/data/data/com.Xxxxx.XxxxxXxxxx.XxxxxXxxxxandroid/files/item.cblite2/db.sqlite3) - 

00:45:04
.825769| WARNING: 
SQLite error (code 14): 
cannot open file at line 36229 of [1a584e4999]
00:45:04
.830157| 
WARNING: 
SQLite error (code 
14
): 
os_unix.c:36229: (24) open(/data/data/com.Xxxxx.XxxxxXxxxx.XxxxxXxxxxandroid/files/item.cblite2/db.sqlite3) - 
Thread finished:  #6390
Thread finished:  #6392
Thread finished:  #6391
00:45:05
.872359| WARNING: SQLite error (code 14): cannot open file at line 36229 of [1a584e4999]
00:45:05.872869| WARNING: SQLite error (code 14): os_unix.c:36229: (24) open(/data/data/com.Xxxxx.XxxxxXxxxx.XxxxxXxxxxandroid/files/item.cblite2/db.sqlite3) - 
00:45:05.880681| WARNING: SQLite error (code 14): cannot open file at line 36229 of [1a584e4999]
00:45:05.881024| WARNING: SQLite error (code 14): os_unix.c:36229: (24) open(/data/data/com.Xxxxx.XxxxxXxxxx.XxxxxXxxxxandroid/files/item.cblite2/db.sqlite3) - 
00:45:05.884022| WARNING: SQLite error (code 14): cannot open file at line 36229 of [1a584e4999]
00:45:05.884356| WARNING: SQLite error (code 14): os_unix.c:36229: (24) open(/data/data/com.Xxxxx.XxxxxXxxxx.XxxxxXxxxxandroid/files/item.cblite2/db.sqlite3) - 
00:45:05.887380| WARNING: SQLite error (code 14): cannot open file at line 36229 of [1a584e4999]
00:45:05.887712| WARNING: SQLite error (code 14): os_unix.c:36229: (24) open(/data/data/com.Xxxxx.XxxxxXxxxx.XxxxxXxxxxandroid/files/item.cblite2/db.sqlite3) - 
00:45:05.890891| WARNING: SQLite error (code 14): cannot open file at line 36229 of [1a584e4999]
00:45:05.891212| WARNING: SQLite error (code 14): os_unix.c:36229: (24) open(/data/data/com.Xxxxx.XxxxxXxxxx.XxxxxXxxxxandroid/files/item.cblite2/db.sqlite3) - 
00:45:05.894602| WARNING: SQLite error (code 14): cannot open file at line 36229 of [1a584e4999]
00:45:05.894952| WARNING: SQLite error (code 14): os_unix.c:36229: (24) open(/data/data/com.Xxxxx.XxxxxXxxxx.XxxxxXxxxxandroid/files/item.cblite2/db.sqlite3) - 
00:45:05
.898199| WARNING: SQLite error (code 14): cannot open file at line 36229 of [1a584e4999]
00:45:05.898591| WARNING: SQLite error (code 14): os_unix.c:36229: (24) open(/data/data/com.Xxxxx.XxxxxXxxxx.XxxxxXxxxxandroid/files/item.cblite2/db.sqlite3) - 03-16 11:08:35.890 W/Adreno-EGLSUB(15994): <DequeueBuffer:700>: dequeue native buffer fail: Unknown error 2147483646, buffer=0x0, handle=0x0

03-16 11:08:35.890 W/Adreno-EGLSUB(15994): <DequeueBuffer:700>: dequeue native buffer fail: Invalid argument, buffer=0x0, handle=0x0
03-16 11:08:35.890 W/Adreno-EGL(15994): <qeglDrvAPI_eglSwapBuffers:3730>: EGL_BAD_SURFACE
00:45:05.902891| WARNING: SQLite error (code 14): cannot open file at line 36229 of [1a584e4999]
00:45:05.903228| WARNING: SQLite error (code 14): os_unix.c:36229: (24) open(/data/data/com.Xxxxx.XxxxxXxxxx.XxxxxXxxxxandroid/files/item.cblite2/db.sqlite3) - 
03-16 11:08:35.890 W/HardwareRenderer(15994): EGL error: EGL_BAD_SURFACE
03-16 11:08:35.900 W/HardwareRenderer(15994): Mountain View, we've had a problem here. Switching back to software rendering.
03-16 11:08:35.920 D/qdgralloc(15994): Invalid gralloc handle (at 0x0): ver(-1/12) ints(-1/12) fds(-1/2)magic(????/gmsm)
03-16 11:08:35.920 W/GraphicBufferMapper(15994): lock(...) failed -22 (Invalid argument)
03-16 11:08:35.920 W/Surface (15994): failed locking buffer (handle = 0x0)
03-16 11:08:35.920 E/ViewRootImpl(15994): Could not lock surface
03-16 11:08:35.920 E/ViewRootImpl(15994): java.lang.IllegalArgumentException
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.Surface.nativeLockCanvas(Native Method)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.Surface.lockCanvas(Surface.java:254)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.ViewRootImpl.drawSoftware(ViewRootImpl.java:2431)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.ViewRootImpl.draw(ViewRootImpl.java:2405)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.ViewRootImpl.performDraw(ViewRootImpl.java:2249)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:1879)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:996)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:5608)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.Choreographer$CallbackRecord.run(Choreographer.java:761)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.Choreographer.doCallbacks(Choreographer.java:574)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.Choreographer.doFrame(Choreographer.java:544)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:747)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.os.Handler.handleCallback(Handler.java:733)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.os.Handler.dispatchMessage(Handler.java:95)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.os.Looper.loop(Looper.java:136)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at android.app.ActivityThread.main(ActivityThread.java:5095)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at java.lang.reflect.Method.invokeNative(Native Method)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at java.lang.reflect.Method.invoke(Method.java:515)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:786)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:602)
03-16 11:08:35.920 E/ViewRootImpl(15994): 	at dalvik.system.NativeStart.main(Native Method)

jens · March 16, 2018, 4:33pm

SQLite error 14 is SQLITE_CANTOPEN, which is pretty generic … fortunately the errno value is hidden in the message: ~~14, EFAULT (“Bad address”). Wow. I don’t think I’ve ever seen this errno before…~~

Something weird is going on. We’ve replicated databases with a million documents, so it’s not something to do with size.

jens · March 16, 2018, 5:13pm

Whoops, the errno is actually 24, EMFILE, “Too many open files”. Your process has opened too many file descriptors, for some reason. Fix: don’t open so many files

thomas1983 · March 19, 2018, 7:08am

Hi Jens,

thank you for your reply. But how do I do that? I can’t control how many files the replication opens no? I don’t perform any other actions while it completes this process…

jens · March 20, 2018, 9:50pm

Sorry, I was being a bit flip. I’ve since figured out this probably is CBL’s fault; I’ve checked in a commit today that limits the number of blobs being downloaded simultaneously, which should help.