CBQ Out of key validation space Error

[patrizok]

Hello I am trying to run this query to move some information from one bucket to another.
/opt/couchbase/bin/cbq -u Administrator -p xxx --pretty=false --file=ss.txt -o found.txt
ss.txt content is:

\SET -max_parallelism 512;
\SET -timeout 1440m;
\SET -txtimeout 1440m;
\SET -scan_wait 0;
\SET -kvtimeout 1440s;
\SET -scan_consistency not_bounded;
\SET -pipeline_batch 20000;
\SET -pipeline_cap 20000;

UPSERT INTO YYY.YYY.YYY(
KEY prop,
VALUE {“property”: TRUE}
)
SELECT DISTINCT prop
FROM XXX.XXX.XXX
WHERE property IS NOT MISSING

But I got error

{
“requestID”: “6cc9b9f0-7507-4edb-bbbd-xxx”,
“signature”: null,
“results”: [
],
“errors”: [{“code”:5006,“msg”:“Out of key validation space.”},{“code”:5006,“msg”:“Out of key validation space.”},{“code”:5006,“msg”:“Out of key validation space.”}],
“status”: “fatal”,
“metrics”: {“elapsedTime”: “3h51m48.175057357s”,“executionTime”: “3h51m48.174928867s”,“resultCount”: 0,“resultSize”: 0,“serviceLoad”: 0,“mutationCount”: 540629506,“errorCount”: 3}
}

I already increase scan_timeout and request-size-cap but it does not effect.
I don’t want to chuck or split query with where condition.

How can I increase key validation space or fix problem?

Thanks.

[mreiche]

There is an internal mechanism that tracks the keys to prevent a document from being modified more than once in a single statement. The size of the structure used to maintain the list of keys is being exceeded (the size is at least 128MB). Would it be possible for you to perform those updates with multiple executions of say - 100 million documents each?
You could sort by ‘prop’ and use the “returning” clause of upsert to return the last ‘prop’ upserted, and then the next execution would start from > the last prop (and limit 100,000,000). In other words - keyset pagination.

Something like this :

UPSERT INTO  YYY.YYY.YYY ( KEY prop, VALUE {"property": TRUE})
SELECT DISTINCT  prop
  FROM XXX.XXX.XXX
  WHERE property is not MISSING and  prop > $last_prop order by prop limit 10000000 
  returning raw prop

p.s. While it may be possible to increase the 128MB with memory_quota, it is not possible to make it unlimited.