.Net SDK Serialization Inconsistency

[Karl_Speer]

I am using the .Net SDK (version 2.7.8 as of the time of this posting) to interact with a Couchbase data store. I have noticed an inconsistency with how the SDK serializes objects. If I were to have an object that I wrote as a document to couchbase (such as using _bucket.ReplaceAsync('my:object', myObject), then the object is serialized a certain way. Property names are converted to camel case. Enums are translated as numbers.

But then, I have the following method:

public async Task<(bool Success, string ErrorMessage)> UpdateRoleAsync(long orgId, Role role)
{
    var orgKey = GetKey(orgId);

    var query = new QueryRequest()
        .Statement(@"UPDATE users AS org  USE KEYS $orgKey
             SET org.roles = ARRAY_REPLACE(org.roles, r, $role)
             FOR r IN org.roles WHEN r.id = $roleId END")
        .AddNamedParameter("$roleId", role.Id)
        .AddNamedParameter("$role", role)
        .AddNamedParameter("$orgKey", orgKey);

    var result = await _usersBucket.QueryAsync<string>(query);

    if (!result.Success)
    {
        return (false, result.ErrorMessages());
    }

    return (true, null);
}

When this query executes, the role object is NOT serialized the same way. Instead of camel casing property names, they are left in the original C# casing. Instead of numbers, enums are translated as strings. This leads to this awkward result:

{
  "id": 100,
  "name": "Sample Organization",
  "roles": [
    {
      "description": "The owner of a collection",
      "id": 5,
      "name": "Collection Owner",
      "permissions": [
        "collection:general:add",
        "collection:general:rearrange",
        "collection:general:remove",
        "collection:general:userManage",
        "collection:view"
      ],
      "roleType": 0,
      "scope": 4,
      "type": "role"
    },
    {
      "Type": "role",
      "Id": 11963,
      "Name": "New Role",
      "Description": "This is my new description, get with it loser - take 2",
      "Scope": "workspace",
      "Permissions": [
        "asset:upload",
        "asset:download",
        "asset:ratingColor:update",
        "asset:labelSelect:update",
        "asset:labelApprove:update",
        "ws:update",
        "ws:roleAssignment:manage",
        "ws:view"
      ],
      "RoleType": "userDefined"
    }
  ],
  "type": "organization"
}

For this example, role 11963 was the role I updated via the query, while role 5 was written via document write.

In reality, I don’t care if enums are strings or numbers, whether properties are camel cased or not. - I just want it consistent. Is there a setting I can set/override somewhere to do this, or is this a bug in the SDK?

[jmorris]

@Karl_Speer -

The QueryRequest has a DataMapper property where you can provide a custom implementation of IDataMapper which takes an ITypeSerializer which you can customize by using the DefaultSerializerand then providing JsonSerializerSettings in the constructor that are equivalent to what K/V is using.

All that being said, the settings should be the same between the two different services. Are you overriding the default serializer in your config or programmatically?

-Jeff

[Karl_Speer]

@jmorris -

We are not overriding the serializer in the Couchbase config:

var configuration = new ClientConfiguration();
configuration.Servers = GetServerUris().ToList();
ClusterHelper.Initialize(configuration);

We are setting these settings on our JsonConvert instance:

// Setup default JSON serialization options
// Camel case property names and enum values
// Ignore null valued properties
JsonConvert.DefaultSettings = () =>
{
    var settings = new JsonSerializerSettings();

    settings.ContractResolver = new CamelCasePropertyNamesContractResolver();
    settings.Converters.Add(new StringEnumConverter(camelCaseText: true));
    settings.NullValueHandling = NullValueHandling.Ignore;

    return settings;
};

[jmorris]

@Karl_Speer -

That is likely the issue, but I am also assuming you are not using JsonProperty attributes which can affect the JSON output as well. This forum post explains an easy way to control the JSON serialization and deserialization.

-Jeff

[Karl_Speer]

@jmorris -

We have been using JsonProperty attributes as a stop-gap. We have set json properties to force proper casing, and we have had to make a custom JsonConverter to handle forcing enums as numbers.

Again, the main concern is not whether enums are numbers or not, or how the properties are capitalized once Couchbase stored the document - the concern is that whether I pass in an object as a parameter or write a document directly, that it is the SAME. While overriding configurations and jsonproperties can help, they are just patches over the real problem that the SDK is not serializing objects consistently.

Another thing to note is that JsonConvert is pulling double-duty for us - Couchbase uses it to serialize documents, but it is also used to serialize/deserialize web requests. To that end, we can’t change the conversion settings of JsonConvert due to frontend requirements.

[jmorris]

Hi @Karl_Speer -

I created a standalone project which includes uses your code above (I also creates Roles and Organization POCOs to match your JSON) and inserted two Organizations each with two Roles. I then updated one of the Roles in the Organization POCO using your code (slightly modified - removed GetKey()) and then ran a direct query and captured the HTTP response.

Here is the initial JSON after inserting:

Here is it after running your UpdateRoleAsync function:

Here is the HTTP response:

So with the default settings, the JSON parsing is consistent between K/V, query UPDATE and query SELECT in the SDK itself; any inconsistency woud be caused by settings outside of the SDK which then effect the SDK. There is no way for us to control that!

That being said, two things I also noticed:

1. The parameters passed in are internally turned to JSON using JsonConvert; this could/should be done using the IDataMapper instance associated with SDK and was discussed earlier with you. This can be changed and then your external changes to JsonConvert should not effect this JSON serialization. I opened a ticket for changing that to DataMapper.

2. What you are doing with Query can be done much more efficiently using the Sub-Document API - you should probably look into using that API for updating a JSON field in a doc by key.

Thanks,
Jeff