Can the Couchbase NodeJS SDK return a JSON document as a raw JSON string, without parsing it first?

Node.js SDK
,
1

Hello,

  • Couchbase stores the documents as JSON, and the Couchbase NodeJS SDK retrieves the documents from the couchbase server as JSON strings. ( correct me if this is wrong)
  • when I use the method “couchbase.Scope.query” from the NodeJS SDK, the JSON is automatically parsed by the NodeJS SDK and returned as JavaScript objects.
  • when my REST API sends back a response to the browser client, the documents need to be re-stringified as JSON strings.
  • often, I make no changes to documents before returning them, and so it is useless to parse the JSON document and then re-stringify it, and this is a performance cost that could be avoided
  • so my question is : can the Couchbase NodeJS SDK return a json document that it has fetched from the Couchbase server as a json string, without parsing it first ? In particular the “couchbase.Scope.query” method ? In this case I could return the raw JSON string returned by Couchbase Server directly, and avoid a useless JSON.parse operation followed by a useless JSON.stringify operation.
    Thanks for any insight
2

It would help to see an example of the document and the query. If the document is not json, how is it being queried?
I think that with query - there is no option to get anything other json returned QueryOptions | couchbase. It might make sense to have a query that returns documentIds, then use kv get() with the RawStringTranscoder to retrieve the documents.

  • Couchbase stores the documents as JSON, and the Couchbase NodeJS SDK retrieves the documents from the couchbase server as JSON strings. ( correct me if this is wrong)

With the kv api, it can store anything that can be serialized. It can retrieve anything that can be deserialized.

1 Like
3

Hello. so I’m returning a stream of user’s projects from a NestJS route (example simplified for clarity):

@Get('/user/projects')
  async streamProjects(
    @Req() req: FastifyRequest,
    @Res() res: FastifyReply
  ) {
    const readableStream = new Readable({
      read() {},
    });

    const query = `SELECT a.*
            FROM default:db.scope.projects a 
            WHERE a.userId='${req.userId}' 
            AND a.archived=false
            ORDER BY a.updated DESC;`;

    const queryStream = CouchbaseService.scope.query<ProjectType>(query);
    queryStream.on('row', (projectRow: ProjectType) => {
        readableStream.push(JSON.stringify(projectRow) + '\r\n');
      });
    queryStream.on('end', () => {
        readableStream.push(null);
      });

    setStreamHeaders(res);
    readableStream.pipe(res.raw);
  }

as you can see, I have to call JSON.stringify on every document that I am returning to the user, which to me seemed inefficient because the documents are already stored as JSON in Couchbase.

I think you answered accurately that currently QueryOptions does not allow to specify a transcoder. So this should be a feature request : allow QueryOptions to specify to return the documents as strings, or allow it to specify a transcoder.

4

If you store and retrieve as a string, you would not be able to reference userId, archived and updated in n1ql - so storing as json is unavoidable for this case. For retrieving each row as a string - this is a bit of a guess, but I’m wondering if this would work ( I’m on my phone where I cannot test it )

Edit: use this:
SELECT RAW ENCODE_JSON(a)

1 Like
5

This seems to work

const couchbase = require('couchbase');
const { Readable, PassThrough } = require("stream");

async function main() {

  const cluster = await couchbase.connect('couchbase://localhost', {
    username: 'Administrator',
    password: 'password',
  });

  const readableStream = new Readable({
    read() {},
  });

  const query = "SELECT RAW ENCODE_JSON(a)   FROM `travel-sample` a limit 2";

  const queryStream = cluster.query(query);
  queryStream.on('row', (row) => {
      console.log('row: ',row, typeof(row));  // to show that the row is a string
      readableStream.push(row + '\r\n');  // adding \r\n shouldn't be necessary
    });
  queryStream.on('end', () => {
      readableStream.push(null);
    });

   //setStreamHeaders(res);
   //readableStream.pipe(res.raw);

    readableStream.pipe(process.stdout)
}

main().catch(e => console.log(e));  // this catch doesn't seem to work as I hoped.

row:  {"callsign":"MILE-AIR","country":"United States","iata":"Q5","icao":"MLA","id":10,"name":"40-Mile Air","type":"airline"} string
{"callsign":"MILE-AIR","country":"United States","iata":"Q5","icao":"MLA","id":10,"name":"40-Mile Air","type":"airline"}
row:  {"callsign":"TXW","country":"United States","iata":"TQ","icao":"TXW","id":10123,"name":"Texas Wings","type":"airline"} string
{"callsign":"TXW","country":"United States","iata":"TQ","icao":"TXW","id":10123,"name":"Texas Wings","type":"airline"}

Later I found there is somewhat of an answer here that also points to ENCODE_JSON. How convert json object to string?

6

Thanks I’ll give it a try

7

This works, thanks. Marked it as solved. Best regards

1 Like