Parallel N1ql execution support?

Hi,

I’ve been working on an ETL script for couchbase to use at work. I’ve tried to use SDK, but then sync query was just too slow. I ended up using N1QL rest api instead since PHP can send multiple curl requests in parallel.

I think this was brought up before, but I’m not sure if it was planned or not.

Are there any plans for parallel n1ql execution?

It might be useful. Technically it is not complex, the problem is the API for that.

What do you think about this one?

$query1 = CouchbaseN1qlQuery::fromString('SELECT * FROM `travel-sample` WHERE type="airport"');
$query2 = CouchbaseN1qlQuery::fromString('SELECT * FROM `travel-sample` WHERE type="hotel"');

$multi = CouchbaseN1qlQuery::multi($query1, $query2);
$result = $multi->execute();

// printout airports
if ($result[0]->error == NULL) {
  for ($result[0]->rows as $row) {
    var_dump($row);
  }
}

Note that in this case it will not throw exceptions, but rather return them in error attribute of result.

1 Like

Hi! Thank you for getting back to me.

That’s how Guzzle has been doing so far. http://docs.guzzlephp.org/en/latest/quickstart.html#concurrent-requests. I think it works well.

It would be nice to name the key though.

You can do UNION ALL of all queries, Query engine will run each arm of the UNION in parallel.

1 Like

Also, max_parallelism controls the parallelism of certain stages (like join, sort) within the query. Query’s data fetch is always done in parallel.

1 Like

That is interesting. I can probably just create a class that takes a query then runs them with union all query.

$queryService->addQuery(N1qlQuery::fromString("select * from ..."));
$queryService->addQuery(N1qlQuery::fromString("select * from ..."));

$result = $queryService->run();

UNION ALL will be executed on single Query Service in parallel. Client side can explore further possibility of parallelism if there are multiple query services they can distribute across query services.