Handling BigInteger Type?

monti.mauro · September 24, 2016, 7:23am

How do I handle a BigInteger types using Java SDK?
As example, I have the following problem:

User POST {"big":12345678901234567890} ==> Rest API receives a Map where “big” field is converted to BigInteger object.

When trying to insert the map:

JsonObject object = JsonObject.empty();
object.put("body", map); ----> Throws an Exception.

JsonDocument document = JsonDocument.create("1234", object);
bucket.insert(document);

Any workaround for this scenario?
Thanks!

egrep · September 24, 2016, 8:48am

@monti.mauro,

JsonObject acts (considering key-value storage implementation) like a wrapper of HashMap<String,Object>, see https://github.com/couchbase/couchbase-java-client/blob/master/src/main/java/com/couchbase/client/java/document/json/JsonObject.java for details.

Supported types for storing are:

github.com

couchbase/couchbase-java-client/blob/master/src/main/java/com/couchbase/client/java/document/json/JsonValue.java#L55


 *
 * @return an empty {@link JsonArray}.
 */
public static JsonArray ja() {
    return JsonArray.create();
}


/**
 * Helper method to check if the given item is a supported JSON item.
 *
 * @param item the value to check.
 * @return true if supported, false otherwise.
 */
public static boolean checkType(Object item) {
    return item == null
        || item instanceof String
        || item instanceof Integer
        || item instanceof Long
        || item instanceof Double
        || item instanceof Boolean
        || item instanceof BigInteger

Use JsonObject.from(Map) https://github.com/couchbase/couchbase-java-client/blob/master/src/main/java/com/couchbase/client/java/document/json/JsonObject.java#L106

or simply insert big integers as strings (i.e. toString() or toString(Character.MAX_RADIX) to save space).

But of course there is a much easier way to get all answers you need: http://docs.couchbase.com/sdk-api/couchbase-java-client-2.3.3/

In short, “Use the docs, Luke!”

monti.mauro · September 25, 2016, 9:42pm

Ey “luke”,

I posted the question cause I tested/read the docs/review the SDK code without finding a satisfying solution.

Using JsonObject.from(Map) will simply NOT work for the same reason you posted.

You can see that here:

github.com

couchbase/couchbase-java-client/blob/master/src/main/java/com/couchbase/client/java/document/json/JsonObject.java#L145


                c.initCause(e);
                throw c;
            }
        } else if (value instanceof List) {
            try {
                JsonArray sub = JsonArray.from((List<?>) value);
                result.put(key, sub);
            } catch (Exception e) {
                //no risk of a direct ClassCastException here
                ClassCastException c = new ClassCastException("Couldn't convert sub-List " + key + " to JsonArray");
                c.initCause(e);
                throw c;
            }
        } else if (!checkType(value)) {
            throw new IllegalArgumentException("Unsupported type for JsonObject: " + value.getClass());
        } else {
            result.put(key, value);
        }
    }
    return result;
}

Also, storing as a String is not an option. In my case I have an entity with a Map<String, Object> that can hold “big numbers” inside, and if I convert to String I lose the type information and I cannot convert back to the correct type.

Finally, Couchbase can store “big numbers” right, just because it stores a JSON and JSON Spec does not impose any constraints on range or precision for numbers, so this is a limitation only in the Java Client SDK.

I checked the Node SDK and the Go SDK and they don’t have that limitation.

On the other hand, using Jackson, you can map a BigInteger type without any problem.
Consider the following example:

String json = "{\"bignumber\": 12345678901234567890}";
ObjectMapper om = new ObjectMapper();
Map<String, Object> map = om.readValue(json, Map.class);
System.out.println(map.get("bignumber") instanceof BigInteger); // True

So I don’t understand why this limitation exist in the Java Client.

Sometimes is a bit more than just point to a JavaDoc method signature…
M.-

egrep · September 25, 2016, 11:47pm

@monti.mauro,
damn, i like this kinda “Ey, lulke” challenges!
ok, first of all seems like you’ve dived down to JsonObject.fromJson => CouchbaseAsyncBucket.JSON_OBJECT_TRANSCODER.stringToJsonObject(s) => JacksonTransformers.MAPPER.readValue(input, JsonObject.class) )
As i understand from static initialization part, for “real parsing” following one method is used:

github.com

couchbase/couchbase-java-client/blob/master/src/main/java/com/couchbase/client/java/transcoder/JacksonTransformers.java#L59


}


static class JsonArraySerializer extends JsonSerializer<JsonArray> {
    @Override
    public void serialize(JsonArray value, JsonGenerator jgen,
                          SerializerProvider provider) throws IOException {
        jgen.writeObject(value.toList());
    }
}


static abstract class AbstractJsonValueDeserializer<T> extends JsonDeserializer<T> {


    private final boolean decimalForFloat;


    public AbstractJsonValueDeserializer() {
        decimalForFloat = Boolean.parseBoolean(
            System.getProperty("com.couchbase.json.decimalForFloat", "false")
        );
    }


    protected JsonObject decodeObject(final JsonParser parser, final JsonObject target) throws IOException {

So, what about incoming numbers ? JsonParser.getValueAsNumber(), that returns java.lang.Number.
And what do we have ? No method (even abstract) for Number-> BigIntgeger conversation (https://docs.oracle.com/javase/8/docs/api/java/lang/Number.html). Of course, there is always a BigInteger(String, radix), but radix can vary from 2 to 36, so how could we guess what radix this partucular BigInteger is? Are there any reasonable method to clarify, for example, what is the radix [2…36] for BigInteger that is the sequence of 1024 “1”-es (ie “111111 … 11111”) ? But formally it’s a Number.

About your example: i suppose that default implementation of om.readValue(json, Map.class) uses implicit consideration that incoming numbers > Long.MAX_VALUE are BigIntegers with radix = 10.

“What about all this, Luke” ?

P.S. +more, for example, 100% valid BigInteger “aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa” (possible radix = [11…36]) is not even a Number.

P.P.S. Hmm, seems like it’s no too hard to add something kinda getBigInteger(radix) to JsonObject (also implying fixed radix for inner-parsing-needs, or setting it via bucket-connection-configuration-param). @daschl, @simonbasle, any comments ? @monti.mauro https://github.com/couchbase/couchbase-java-client/blob/master/CONTRIBUTING.md ?

monti.mauro · October 4, 2016, 7:27am

hi @egrep/@daschl/@simonbasle ,

Sorry for the late response.

Per my understanding the cb-java-client has Jackson as dependency and is using it to parse and serialize values, so why limit the types we can parse and serialize if Jackson supports a broader range of types.

I think the only change we need here is to add the Number class as valid type (or BigInteger or BigDecimal) in the checkType() method of the JsonValue class.

github.com

couchbase/couchbase-java-client/blob/master/src/main/java/com/couchbase/client/java/document/json/JsonValue.java#L61




/**
 * Helper method to check if the given item is a supported JSON item.
 *
 * @param item the value to check.
 * @return true if supported, false otherwise.
 */
public static boolean checkType(Object item) {
    return item == null
        || item instanceof String
        || item instanceof Integer
        || item instanceof Long
        || item instanceof Double
        || item instanceof Boolean
        || item instanceof BigInteger
        || item instanceof BigDecimal
        || item instanceof JsonObject
        || item instanceof JsonArray;
}


}

If we want to limit to some specific types:

|| item instanceof BigInteger || item instanceof BigDecimal

If not we can just check for Number

|| item instanceof Number

I tested this replacing the current JsonValue class for a custom one that accept these types in the checkType() method and storing and retrieving “Big Numbers” works as expected.

Unless I am missing something here, I think that’s the only change that need to be applied.

What do you guys think?

PS: I would be happy to contribute if needed
M.-

daschl · October 4, 2016, 9:04am

@monti.mauro I think trying to support BigInteger out of the box if we can do that without much hassle is a worthwhile goal - and we’d love a contribution!

You can find information about it here: https://github.com/couchbase/couchbase-java-client/blob/master/CONTRIBUTING.md and I’m happy to help if you get stuck along the way. If you put up a changeset please reference the issue https://issues.couchbase.com/browse/JCBC-1012 which I just created.

Thanks!
Michael

egrep · October 4, 2016, 10:49am

@monti.mauro,

well, most likely, i was wrong and it could be done even easier than i thought.
Within your modification tests did you use (BigInteger)JsonObject’s get(“name”) method to fetch and put(“name”, bi) to set ?
It’s even funny: toString() for BigInteger returns 10-based (absolutely inefficient from the storing size viewpoint) string representation (but formally it’s a Number) and seems like this allows such an easy way to use it
Probably, you should also add a method getBigInteger(), so looks like https://github.com/couchbase/couchbase-java-client/blob/master/CONTRIBUTING.md awaits you

P.S. In other hand, i think, these should be told:

10-radix is an inefficient way to store big integers (especially when we are talking not about “12345678901234567890” but about hundreds/thousands-chars in string-form-representation for cases like DH/SRP-like authentication procedures)
no radix except 10 for BigIntegers could be used with other API/REST calls for database if you use JavaSDK with this improvement
documentation should also explicitly say that “we store big integers in form of string with radix 10”

But all these is more like “lyrics”, so why don’t you just try ?

monti.mauro · October 7, 2016, 6:11pm

Hi @daschl, I started to follow the contribution process and in the document it says that I should get an email.

4 - If you do not receive an email, please contact us

I didnt receive the email. Who should I contact?
Thanks!

daschl · October 14, 2016, 11:38am

@monti.mauro hmm that is weird. As a PN here in the forums can you mail me your email address please? So I can check what happened with our folks thanks!