Need help with CustomTranscoder

Hi all. I don’t use default transcoder, but get such error “Unexpected character encountered while parsing value: q. Path ‘’, line 0, position 0.” at

   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at Couchbase.Core.IO.Serializers.DefaultSerializer.Deserialize[T](ReadOnlyMemory`1 buffer)
   at Couchbase.KeyValue.GetResult.ContentAs[T]()
   at BinaryCouchbase.Program.<Read1>d__1.MoveNext() in C:\Repositories\ToolsTestings\BinaryCouchbase\Program.cs:line 49
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ValueTaskAwaiter.GetResult()
   at BinaryCouchbase.Program.<Main>d__0.MoveNext() in C:\Repositories\ToolsTestings\BinaryCouchbase\Program.cs:line 32

My transcoder.

public sealed class Transcoder : ITypeTranscoder
    public ITypeSerializer? Serializer { get; set; }

    public Flags GetFormat<T>(T value)
        return new Flags
            Compression = Compression.None,
            DataFormat = DataFormat.Private,
            TypeCode = TypeCode

    public void Encode<T>(Stream stream, T value, Flags flags, OpCode opcode)
        var instance = value as TestType;
        var bytes = Encoding.Unicode.GetBytes(instance!.Property1);
        stream.Write(bytes, 0, bytes.Length);

    public T Decode<T>(ReadOnlyMemory<byte> buffer, Flags flags, OpCode opcode)
        var bytes = Encoding.Unicode.GetString(buffer.Span);
        return (T) (object) new TestType
            Property1 = bytes

    private static readonly TypeCode TypeCode = Type.GetTypeCode(typeof(TestType));

And simple using.

var insertResult = await collection.InsertAsync("testId", new TestType { Property1 = "qwerty" }, options => options
    .Transcoder(new Transcoder()));

var getResult = await collection.GetAsync("testId", options => options
    .Transcoder(new Transcoder())

var type = getResult.ContentAs<TestType>();

Is it my mistake somewhere or bug?

Yes, it’s bug. It happens only with .Expiry()


First of all, if your goal is pure binary storage/retrieval, then I recommend using RawBinaryTranscoder. This transcoder is designed for this purpose and is built in. Just use <byte[]> directly when setting or getting documents along with this transcoder (without the wrapping class).

As to your problem, I believe that it is in unsupported operation. When you use .Expiry() you are effectively using a subdoc multi-get operation. This operation gets the expiry as one part of the response and the content as another. The content portion is then assumed to be JSON and deserialization is attempted, rather than running through the transcoder.

However, I’m not sure if that’s a requirement of the design or an oversight. Hopefully @jmorris may know more.

It’s quite strange behaviour. It makes all custom transcoders are useless. And looks like a corruption of all SOLID principles.

@Ramirag -

The actual error is not from the transcoder, but I believe from the serializer component as it expects JSON here and your supplying unicode encoded bytes.

I am curious, why do you need a custom transcoder? The use-case for custom transcoders is very small - really the only one I know of is an update of a very old version of Couchbase Server (2.0 and before) to a later version and documents stored as binary blobs need to be written back as JSON documents.

Furthermore, if your not storing you documents as JSON documents, your not going to be able to use server features like N1QL, FTS, etc that depend upon a JSON format. Once you change the format you are limiting your available server feature set.



I’ve discussed your concerns with @jmorris, and we agree that your use case is applicable. The internals of how we get the expiry should be disguised from you, and a custom transcoder should work. I have filed a bug for tracking:

That said, we do feel like a custom transcoder may not be the best choice for your use case, RawBinaryTranscoder and ContentAs<byte[]>() is probably a better option. Please let me know if you have any questions about that, we’d be happy to help. Note, however, that the bug we filed does also affect RawBinaryTranscoder (for now).

@jmorris @btburnett3

I don’t need any abilities for working with JSON. It’s just a cache, read and write, nothing else. Working with binary data is faster, plus reduces more than 2x size.

  1. The custom transcoder is cheaper for memory and CPU. RawBinaryTranscoder makes a copy of internal bytes.
  2. Flexibility. I can change transcoder in the IOC and start using another format.
  3. Code style. It looks better than a calling some serializer after reading bytes from Couchbase. Plus I haven’t to add an additional interface for this serializer.

And why are u using JSON.NET? It works slower than a new one .NET serializer. Probably it is based on legacy, but can u add such custom transcoder?

@Ramirag -

Indeed, a cache scenario is a valid use-case for a binary or custom transcoder.

There is a Jira ticket for moving towards System.Text.Json, but there is a lot of work to do get there as we use JSON.NET in a lot of places outside K/V - atm its not a priority, but we’ll definitely get there.


1 Like


You are right, the RawBinaryTranscoder does make a copy of the bytes, though this is very intentional. Internally, the SDK is passing around copies of byte arrays from the ArrayPool to reduce such copies. A final copy is performing in RawBinaryTranscoder because we’re passing the byte[] out of the SDK. Once the IGetResult is disposed or GC’d the byte array is returned to the pool, leaving a risk that a non-copy is reused for something else and the data corrupted.

However, looking more closely at your use case, I would recommend the RawStringTranscoder. This should align better for you and avoids the copy much like your code does by encoding/decoding directly to/from the string.