Can the Couchbase NodeJS SDK return a JSON document as a raw JSON string, without parsing it first?

Hello,

  • Couchbase stores the documents as JSON, and the Couchbase NodeJS SDK retrieves the documents from the couchbase server as JSON strings. ( correct me if this is wrong)
  • when I use the method “couchbase.Scope.query” from the NodeJS SDK, the JSON is automatically parsed by the NodeJS SDK and returned as JavaScript objects.
  • when my REST API sends back a response to the browser client, the documents need to be re-stringified as JSON strings.
  • often, I make no changes to documents before returning them, and so it is useless to parse the JSON document and then re-stringify it, and this is a performance cost that could be avoided
  • so my question is : can the Couchbase NodeJS SDK return a json document that it has fetched from the Couchbase server as a json string, without parsing it first ? In particular the “couchbase.Scope.query” method ? In this case I could return the raw JSON string returned by Couchbase Server directly, and avoid a useless JSON.parse operation followed by a useless JSON.stringify operation.
    Thanks for any insight

It would help to see an example of the document and the query. If the document is not json, how is it being queried?
I think that with query - there is no option to get anything other json returned QueryOptions | couchbase. It might make sense to have a query that returns documentIds, then use kv get() with the RawStringTranscoder to retrieve the documents.

  • Couchbase stores the documents as JSON, and the Couchbase NodeJS SDK retrieves the documents from the couchbase server as JSON strings. ( correct me if this is wrong)

With the kv api, it can store anything that can be serialized. It can retrieve anything that can be deserialized.

1 Like

Hello. so I’m returning a stream of user’s projects from a NestJS route (example simplified for clarity):

@Get('/user/projects')
  async streamProjects(
    @Req() req: FastifyRequest,
    @Res() res: FastifyReply
  ) {
    const readableStream = new Readable({
      read() {},
    });

    const query = `SELECT a.*
            FROM default:db.scope.projects a 
            WHERE a.userId='${req.userId}' 
            AND a.archived=false
            ORDER BY a.updated DESC;`;

    const queryStream = CouchbaseService.scope.query<ProjectType>(query);
    queryStream.on('row', (projectRow: ProjectType) => {
        readableStream.push(JSON.stringify(projectRow) + '\r\n');
      });
    queryStream.on('end', () => {
        readableStream.push(null);
      });

    setStreamHeaders(res);
    readableStream.pipe(res.raw);
  }

as you can see, I have to call JSON.stringify on every document that I am returning to the user, which to me seemed inefficient because the documents are already stored as JSON in Couchbase.

I think you answered accurately that currently QueryOptions does not allow to specify a transcoder. So this should be a feature request : allow QueryOptions to specify to return the documents as strings, or allow it to specify a transcoder.

If you store and retrieve as a string, you would not be able to reference userId, archived and updated in n1ql - so storing as json is unavoidable for this case. For retrieving each row as a string - this is a bit of a guess, but I’m wondering if this would work ( I’m on my phone where I cannot test it )

Edit: use this:
SELECT RAW ENCODE_JSON(a)

1 Like

This seems to work

const couchbase = require('couchbase');
const { Readable, PassThrough } = require("stream");

async function main() {

  const cluster = await couchbase.connect('couchbase://localhost', {
    username: 'Administrator',
    password: 'password',
  });

  const readableStream = new Readable({
    read() {},
  });

  const query = "SELECT RAW ENCODE_JSON(a)   FROM `travel-sample` a limit 2";

  const queryStream = cluster.query(query);
  queryStream.on('row', (row) => {
      console.log('row: ',row, typeof(row));  // to show that the row is a string
      readableStream.push(row + '\r\n');  // adding \r\n shouldn't be necessary
    });
  queryStream.on('end', () => {
      readableStream.push(null);
    });

   //setStreamHeaders(res);
   //readableStream.pipe(res.raw);

    readableStream.pipe(process.stdout)
}

main().catch(e => console.log(e));  // this catch doesn't seem to work as I hoped.
row:  {"callsign":"MILE-AIR","country":"United States","iata":"Q5","icao":"MLA","id":10,"name":"40-Mile Air","type":"airline"} string
{"callsign":"MILE-AIR","country":"United States","iata":"Q5","icao":"MLA","id":10,"name":"40-Mile Air","type":"airline"}
row:  {"callsign":"TXW","country":"United States","iata":"TQ","icao":"TXW","id":10123,"name":"Texas Wings","type":"airline"} string
{"callsign":"TXW","country":"United States","iata":"TQ","icao":"TXW","id":10123,"name":"Texas Wings","type":"airline"}

Later I found there is somewhat of an answer here that also points to ENCODE_JSON. How convert json object to string?

Thanks I’ll give it a try