Couchbase does not load all data for 300k records

SQL++
1

Hi,

I’m using nodejs and N1QL query to load all records to an array. When do counting, there are 300k records, but when loading using NodeJS, I can’t load it all, some time got 201744, sometime got 200128. This is my query:

select t.* from default t where type='Place'
var query = N1qlQuery.fromString(sql);
bucket.query(query, (err, list) => {
   console.log("Size:", list.length);
});

It seem the timeout of query is about 80seconds, the query seem always stop after 80s then sometime the length is 201744, sometime is 200128.

Can you advice on how can I load all 300k records ?

2

Are you also receiving an error? If it’s timing out and you’re receiving a subset without receiving an error, that sounds like an issue. Just to verify this, you might set the n1qlTimeout on the bucket to a higher value to see if that enables you to get all of your items.

If, with a higher timeout set, you are still getting a subset of what you expect it could be either related to the consistency options or your query is selecting different items than you think it might be.

A related note, even if you set that higher timeout, the node.js client can react to items as they are returned via the underlying streaming parser. The docs have an example of how this works in the Streaming Rows section.

3

Try using curl or cbq shell. This will isolate the error and tell you if it is a server or client error.

4

Instead of select *, I specify the query projection, then I created covering index, so far, seem no more issue

1 Like