Deserialize issue with JSON property

Hi All,

I am reading Couchbase document using below code,

QueryResult result = cluster.query(cbStatement);
List<CBDocument> cbDock = result.rowsAs(CBDocument.class);

Json String :

{
  "KEY": "",
  "METADATA": {
   
  },
  "PAYLOAD": {
    
  }
}
public class CBDocument implements Serializable {
   private String id;
 
  @JsonProperty("KEY")
  @JsonAlias({"KEY"})
  private String key;

  @JsonProperty("METADATA")
  @JsonAlias({"METADATA"})
  private JsonObject metadata;

  @JsonProperty("PAYLOAD")
  @JsonAlias({"PAYLOAD"})
  private JsonObject payload;
}

Deserialization is not happen correctly … all property showing null value …

Thanks
Bhushan

Hi @bphalak
This document is a useful read, especially the ‘Using Custom Jackson’ section:

Which package are you importing @JsonProperty from - com.couchbase.client.core.deps.com.fasterxml.jackson.annotation or com.fasterxml.jackson.annotation? And is Jackson explicitly on your classpath or are you only using the shaded version bundled with the SDK?
My current guess is there’s a mismatched there, e.g. maybe you have Jackson explicitly on the classpath but you’re using the com.couchbase.client.core.deps.com.fasterxml.jackson.annotation namespace.

1 Like

hi @bphalak

First investigate what graham said, but …
What is cbStatement you are using?
If you have “select my_bucket.* from my_bucket…” then the results will match your object:

  {
    "KEY": "",
    "METADATA": {},
    "PAYLOAD": {}
  }

But if you have “select * from my_bucket” , they will not.

  {
    "my_bucket": {
      "KEY": "",
      "METADATA": {},
      "PAYLOAD": {}
    }
  }

i am using
package com.fasterxml.jackson.annotation;

@Data
@Builder
@ToString
@NoArgsConstructor
@AllArgsConstructor
public class CBDocument implements Serializable {
  
  private String id;

  @JsonProperty("METADATA")
  @JsonAlias({"METADATA"})
  private JsonObject metadata;

  @JsonProperty("PAYLOAD")
  @JsonAlias({"PAYLOAD"})
  private JsonObject payload;
}

HI @mreiche

yes if i am using all property in caps like below

private JsonObject METADATA;
private JsonObject PAYLOAD;

but i have to use camel case variable names only.

Thanks

Please show the n1ql statement.

select meta().id, METADATA, PAYLOAD from $BUCKET_NAME$.$SCOPE_NAME$.$COLLECTION_NAME$ as cfg

edit: you’ve edited the post and changed metadata and payload to METADATA and PAYLOAD. So now they match the data. The original n1ql statement - which had lower-case metadata and payload - would not match and result in the fields being null (or an exception, if FAIL_ON_UNKNOWN_PROPERTIES is set ‘true’ on the objectmapper). Keep in mind that json property names are case-sensitive.

And what is the document stored in couchbase? The annotations indicates they are to be stored as uppercase METADATA and PAYLOAD, so the n1ql must reference them as METADATA and PAYLOAD.

Sry Corrected :

N1QL : select meta().id, KEY, METADATA, PAYLOAD from $BUCKET_NAME$.$SCOPE_NAME$.$COLLECTION_NAME$ as cfg

Json : {

 "KEY": "",
  "METADATA": {
   
  },
  "PAYLOAD": {
    
  }
}

Java CLASS :

public class CBDocument implements Serializable {
   private String id;
 
  @JsonProperty("KEY")
  @JsonAlias({"KEY"})
  private String key;

  @JsonProperty("METADATA")
  @JsonAlias({"METADATA"})
  private JsonObject metadata;

  @JsonProperty("PAYLOAD")
  @JsonAlias({"PAYLOAD"})
  private JsonObject payload;
}

Does it work now after correcting it?

Also, please show the complete entity class so we can see which annotations are being used. And how you are determining that metadata and payload are null - since they are both private and there is no method that accesses them, it’s not possible to determine that they are null.

And show the code that is reading the document and determining that the fields are null.

When I make my own test which has the correct annotations and a method to access the payload property, and n1ql that projects the correct properties - it succeeds (when the name is specified in JsonProperty, then specifying JsonAlias is redundant).

	@Test
	void myTest() {
		String json = "{ \"KEY\": \"\", \"METADATA\": { }, \"PAYLOAD\": { } }";
		JsonObject jo = JsonObject.fromJson(json);
		System.out.println(jo);

		Authenticator authenticator = PasswordAuthenticator.create("Administrator", "password");
		ClusterOptions clusterOptions = clusterOptions(authenticator);

		Cluster cluster = Cluster.connect("10.144.220.101", clusterOptions);
		cluster.bucket("my_bucket").defaultCollection().upsert("1", jo);
		try {
			Thread.sleep(1000);
		} catch (InterruptedException ie) {}
		String cbStatement = "select my_bucket.* from my_bucket where PAYLOAD is not missing";
		QueryResult result = cluster.query(cbStatement);
		List<CBDocument> cbDock = result.rowsAs(CBDocument.class);
		assertNotNull(cbDock.get(0).metadata);
		System.out.println(cbDock.get(0));
	}
package org.springframework.data.couchbase.domain;

import com.fasterxml.jackson.annotation.JsonProperty;
import com.couchbase.client.java.json.JsonObject;

import java.io.Serializable;

public class CBDocument implements Serializable {
  //private String id;

  @JsonProperty("KEY")
  private String key;

  @JsonProperty("METADATA")
  private JsonObject metadata;

  @JsonProperty("PAYLOAD")
  private JsonObject payload;

  public String toString(){
    StringBuffer sb= new StringBuffer();
    sb.append(" key: ");
    sb.append(key);
    sb.append(" metadata: ");
    sb.append(metadata);
    sb.append(" payload: ");
    sb.append(payload);
    return sb.toString();
  }

  public Object getPayload() {
    return payload;
  }
}

output:

{"METADATA":{},"PAYLOAD":{},"KEY":""}
 key:  metadata: {} payload: {}

Data in couchbase:

{
    "KEY": "key_text",
  "METADATA": {
      "IS_TDGRAIL": 0,
    .....
  },
  "PAYLOAD": {
    "IS_TDGRAIL": 0,
    "MM_DAYS_TO_SPOT": 2,
 ...
 }
}

Test Class :

package XXX;

import com.couchbase.client.java.Bucket;
import com.couchbase.client.java.Cluster;
import com.couchbase.client.java.query.QueryResult;

import java.time.Duration;
import java.util.List;

public class TestClass {

    public static void main(String[] args) {

        String query = "select meta().id, METADATA, PAYLOAD from `BUCKET`.`SCOPE`.`COLLECTION` as cfg ";
        Cluster cluster = Cluster.connect("XXX", "Administrator", "XXXXXXX");
        Bucket bucket = cluster.bucket("CONFIG");
        bucket.waitUntilReady(Duration.parse("PT10S"));
        QueryResult result = cluster.query(query);
        List<CBDocument> cb = result.rowsAs(CBDocument.class);

        cb.forEach(cbDock -> cbDock.toString());
    }
}

package XXX;

import com.couchbase.client.java.json.JsonObject;
import java.io.Serializable;

import com.fasterxml.jackson.annotation.JsonAlias;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonKey;
import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.ToString;


@Data
@Builder
@ToString
@NoArgsConstructor
@AllArgsConstructor
public class CBDocument implements Serializable {
  @JsonProperty("KEY")
  @JsonAlias({"KEY"})
  private String key;

  private String id;

  @JsonProperty("METADATA")
  @JsonAlias({"METADATA"})
  private JsonObject metadata;

  @JsonProperty("PAYLOAD")
  @JsonAlias({"PAYLOAD"})
  private JsonObject payload;
}

Output :

Caused by: com.couchbase.client.core.deps.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "METADATA" (class com.citi.fx.stirt.couchbase.config.dcp.client.service.CBDocument), not marked as ignorable (4 known properties: "id", "cbEventType", "payload", "metadata"])

Expected:
Data should be correctly bind to all property because i provided required annotation …

At this point you should open a customer support case to get help. Provide them with a reproducer that they can run - that inserts the data and reads it back and demonstrates the issue. It’s not possible for me to troubleshoot when the code that is being run differs from the code provided. It would be best to provide them a reproducer as a github project.

A couple things -

The error message is showing a known property named “cbEventType” in CBDocument. But your CBDocument class does not have any such property.

Somehow you have the (unshaded) com.faster.jackson.* in your classpath (and are using it for the annotations), but the serializer being used is the (shaded) com.couchbase.client.core.deps.com.fasterxml. I don’t know how that happens as you are not manually configuring that on the ObjectMapper, and when couchbase does it, it checks for the existence of the unshaded (user-supplied) com.faster.jackson, and if provided uses that serializer. It’s likely that this is not happening - as your CBDocument class using the unshaded is not that one that is generating the error (see above comment about cbEventType).

The ObjectMapper can be manually configured with the code below.

import com.fasterxml.jackson.databind.ObjectMapper;

		ClusterEnvironment.Builder builder = ClusterEnvironment.builder();
		ObjectMapper couchbaseObjectMapper = new ObjectMapper();
		couchbaseObjectMapper.configure(com.fasterxml.jackson.databind.DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES,
				true);
		couchbaseObjectMapper.registerModule(new JsonValueModule());

		builder.jsonSerializer(JacksonJsonSerializer.create(couchbaseObjectMapper));
		ClusterEnvironment env = builder.build();
		Authenticator authenticator = PasswordAuthenticator.create("Administrator", "password");
		ClusterOptions clusterOptions = clusterOptions(authenticator).environment(env);
		Cluster cluster = Cluster.connect("10.144.220.101", clusterOptions);

You can also check if the unshaded jackson is available by writing a method like this:

import com.couchbase.client.java.codec.JacksonJsonSerializer;

	private boolean nonShadowedJacksonPresent() {
		try {
			JacksonJsonSerializer.preflightCheck();
			return true;
		} catch (Throwable t) {
			return false;
		}
	}