java.lang.IllegalArgumentException: handle must not be 0 when iterating through Array

Hello,
I am running CBLite 2.8.5 on Android. I am seeing an exception being thrown on rare occasions when iterating through an Array and attempting to filter and map it is a Dictionary. Specifically my code is array.filterIsInstance<Dictionary>() (from the Kotlin collections extensions in kotlin-stdlib-common). On rare occasions I can see a stack trace like this:

java.lang.IllegalArgumentException: handle must not be 0
	at com.couchbase.lite.internal.utils.Preconditions.assertNotZero(Preconditions.java:1)
	at com.couchbase.lite.internal.fleece.FLDict.<init>(FLDict.java:2)
	at com.couchbase.lite.internal.fleece.FLValue.asFLDict(FLValue.java:1)
	at com.couchbase.lite.internal.fleece.MDict.initInSlot(MDict.java:5)
	at com.couchbase.lite.internal.fleece.MDict.initInSlot(MDict.java:1)
	at com.couchbase.lite.Dictionary.<init>(Dictionary.java:6)
	at com.couchbase.lite.MValueDelegate.mValueToDictionary(MValueDelegate.java:8)
	at com.couchbase.lite.MValueDelegate.toNative(MValueDelegate.java:5)
	at com.couchbase.lite.internal.fleece.MValue.toNative(MValue.java:2)
	at com.couchbase.lite.internal.fleece.MValue.asNative(MValue.java:3)
	at com.couchbase.lite.Array.getValue(Array.java:1)
	at com.couchbase.lite.Array$ArrayIterator.next(Array.java:1)
    ...

The kotlin stdlib code as my Android Studio defines it:

public inline fun <reified R> Iterable<*>.filterIsInstance(): List<@kotlin.internal.NoInfer R> {
    return filterIsInstanceTo(ArrayList<R>())
}

/**
 * Appends all elements that are instances of specified type parameter R to the given [destination].
 */
public inline fun <reified R, C : MutableCollection<in R>> Iterable<*>.filterIsInstanceTo(destination: C): C {
    for (element in this) if (element is R) destination.add(element)
    return destination
}

I cannot supply the raw JSON for this but I do not think that is the problem anyway as this problem does not happen consistently even when the underlying data in the database has not changed. I should also mention that I have only seen this happen on Android 6 devices, but I am not sure if it is due to that or some other hardware difference. Am I doing something wrong here? Are the kotlin Collection helpers not compatible with some underlying mechanism in the couchbase Array class?

In case it is also helpful/relevant, I got the Array instance by using a QueryBuilder.select() call with a WHERE clause equaling a single document ID, got the query result by executing query.execute().allResults().firstOrNull(), got the nested properties by calling result.getDictionary("DB_NAME") (because the select query was a SelectResult.all()), and then finally invoking dictionary.getValue("key") as? Array to get the array.

Thank you,
Cole

This is probably a bug. Can you reproduce it reliably? That would be a big help.

Also, the line numbers in your stack trace are very very wrong. Are you using an obfuscator? If so, could you try this again, without it?

@blake.meike I have been able to reproduce it locally. I have been able to determine that it only happens on release builds with android.buildTypes.release.minifyEnabled set to true. Yes, I am using a proguard as an obfuscator, but unfortunately because this only reproduces with minifyEnabled set to true I don’t have a de-obfuscated stack trace. I do have -keep class com.couchbase.lite.** { *; } in my proguard-rules.pro file; are there other recommended entries to put in there?

Edit: I edited my proguard-rules.pro to follow the more narrow recommendations documented here Couchbase Lite Java (Android) framework | Couchbase Docs and it still happens. I think it’s also worth noting that as I have debugged this it doesn’t always result in the above exception being thrown - sometimes I get a Dictionary object with the correct keys, but all the values are 0-state.

In fact -keep class com.couchbase.lite.** { *; } is WAY overkill. You should be able to get away with the rules here:
Build and Run | Couchbase Docs.

Whoops. I see you did that. Excellent.

What is the actual (Iterable) receiver for filterIsInstance and filterIsInstanceTo that drives this?

… and, btw… that is N^2 :stuck_out_tongue_winking_eye:

One more thing: You are right about the causes of this. What is happening is that a CBL Array is a list of native objects. Once of them is being de-allocated before you are done using it. That has nearly nothing to do with specific content of the JSON document.

The receiver for filterIsInstance is a com.couchbase.lite.Array object which looks like it is instantiating an Array.ArrayIterator object to implement the Iterator interface. I did try rewriting it to use a for loop with index-based access instead of filterIsInstance() and it still had the problem:

fun Array.asListOfObjects(): List<Dictionary> {
   val list = mutableListOf<Dictionary>()
   for (i in 0 until this.count()) {
      array.getDictionary(i)?.let { list.add(it) }
   }
   return list
}

Uhhh… that doesn’t seem right. There is no iterator being used in the procedural version…

Confirm that it is the call to array.getDictionary(i) that is failing?

I phrased that poorly - the direct receiver for filterIsInstance() is the array, but I believe it uses the ArrayIterator to be able to do the for loop.

Yes, in some scenarios it is the getDictionary() call that fails and results in that stack trace. Other times it succeeds in creating the Dictionary but then my parser fails more softly down the line because the values are not present in the dictionary it is expecting.

Is it possible that you are trying to use the iterator more than once? That won’t work…

I still don’t quite understand the procedural version. Is filterIsInstance calling Array.asListOfObjects

Again, in the procedural version that you show above, using indexing, there is no use of an iterator that I can see…

I would expect the crash to happen every time if I was reusing the iterator. I have contrived a test scenario to reproduce the problem where the document in the database does not change. I have a timer firing every 10ms that runs the query for the document and then feeds the result into my parser, which reproduces this error every 5-10s or so, but most of the time it works.

Confirming that what you mean by “procedural version” is the Kotlin filterIsInstance() which then calls filterIsInstanceTo(). Assuming yes, I believe yes, it is indirectly instantiating the ArrayIterator, since Kotlin/Java implement for-each loops (the for (element in this) loop) by calling the iterator() function on an Iterable class, which Array instantiates an ArrayIterator to implement. You can also see in the bottom line of the first stack trace I posted that is calling ArrayIterator.next():

at com.couchbase.lite.Array$ArrayIterator.next(Array.java:1)

Edit: re-reading your post I see you said “using indexing”, so you probably meant my for (i in 0 until array.count()) example, in which case yes that would not use an iterator. I wrote that specifically to see if there was an iterator problem I could work around.

Sorry… By “procedural” I meant this:

fun Array.asListOfObjects(): List<Dictionary> {
   val list = mutableListOf<Dictionary>()
   for (i in 0 until this.count()) {
      array.getDictionary(i)?.let { list.add(it) }
   }
   return list
}

Using that method reduces the number of free variables by one: there is no iterator. If that version fails it is not the iterator that is to blame.

Is there any chance that you can get me failing code. I’d really like to track this one down.

Yes, the failure still occurs with that version. I will work on getting you a minimal code/JSON sample. It seems mostly prevalent with large JSON documents - the one I am working with is 166216 characters long, with something of a recursive structure like this:

{
  "root": {
    "listOfObjects": [
      {
        "id": "1",
        "name": "name1",
        "listOfObjects": [
          {
            "id": "1-child",
            "name": "child-of-1",
            "listOfObjects": []
          }  
        ]
      },
      {
        "id": "2",
        "name": "name2",
        "listOfObjects": []
      }
    ]
  }
}

A possible code sample in Kotlin (assumes couchbaseDatabase is a Database object already opened and an object matching the above JSON contract is already seeded with id 1234):

data class TestObject(val id: String, val name: String)

fun parseObjectDictionary(dictionary: Dictionary): TestObject? {
    val id = dictionary.getString("id") ?: return null
    val name = dictionary.getString("name") ?: return null
    return TestObject(id, name)
}

val testId = "1234"
val testQuery = QueryBuilder
    .select(
        SelectResult.expression(Meta.id).`as`("_id"),
        SelectResult.all()
    )
    .from(DataSource.database(couchbaseDatabase))
    .where(Expression.property("_id").equalTo(Expression.string(testId)))
testQuery.execute().allResults().firstOrNull()?.getDictionary("databaseFilename")?.let { dictionary ->
    val root = dictionary.getDictionary("root") ?: return@let null
    val objects = (root.getValue("listOfObjects") as? Array)?.filterIsInstance<Dictionary>()?.mapNotNull {
        parseObjectDictionary(it)
    }
    println("got objects=$objects")
}

You would need to make the JSON object in the DB larger than what I have provided above, preferably a similar size to my real document, and call the query code in some sort of timer or loop where it will iterate very rapidly. I would expect objects to start having less objects in it than are actually in the database document if you have successfully reproduced.

Another interesting thing to note - if I call toMap() on the top-level dictionary (root in my code sample above) and keep that, and then when parsing the object if I encounter an unexpected null (as in, id is null in the above sample) and compare it to the same property in the root map, the root map has the non-null property value I am expecting. This would again indicate the native objects are getting released before they should.

I would much prefer to not have to do this as a workaround for two reasons - it seems inefficient to use toMap() as it requires converting all the native objects to Java objects, and I would need to rewrite my parser to use Map APIs instead of directly interfacing with the Couchbase objects (Dictionary, Array, etc.). It almost seems as though the native objects are “use once”, even if it’s just reading, and then they are available for garbage collection, and in the case of large documents it is possible that they get garbage collected before the code is done parsing it. Is there anything I can do to get Couchbase to keep these objects around?

Update on this: I ran a quick test where I rewrote my parser to use Map<String, Any> and occasionally would still see the toMap() call fail with java.lang.IllegalArgumentException: handle must not be 0 stacktrace.

Update: If I use database.getDocument() instead of the QueryBuilder, I do not see this problem. I can use this as a limited workaround in some cases, but I do use Live Queries elsewhere which I cannot switch out for getDocument() and I sometimes select specific fields instead of all fields to optimize the query, so this is not a good long term solution.

@cole.s : I have fixed a bug that had to do with LiveQuery that might be relevant. Were you using LiveQueries at all? Do you have any updates?