Index Scan timesout on limiting condition

Query

SELECT did, ARRAY_AGG(DISTINCT SUBSTR(t, 0, 10)) AS days, {
    "sums" : COUNT(*),
    "null" : COUNT(*) - COUNT(NULLIF(ARRAY_LENGTH(bcn), 0)),
    "full" : COUNT(NULLIF(ARRAY_LENGTH(bcn), 0))
} AS entries
FROM agora_bucket
WHERE type="bcn_scan"
GROUP BY did
ORDER BY did ASC;

returns

],
"status": "success",
"metrics": {
    "elapsedTime": "18m21.700100793s",
    "executionTime": "18m21.700054826s",
    "resultCount": 17,
    "resultSize": 4864,
    "sortCount": 17
}

while

SELECT did, ARRAY_AGG(DISTINCT SUBSTR(t, 0, 10)) AS days, {
    "sums" : COUNT(*),
    "null" : COUNT(*) - COUNT(NULLIF(ARRAY_LENGTH(bcn), 0)),
    "full" : COUNT(NULLIF(ARRAY_LENGTH(bcn), 0))
} AS entries
FROM agora_bucket
WHERE type="bcn_scan" AND SUBSTR(t, 0, 10) >= "2017-03-03"
GROUP BY did
ORDER BY did ASC;

returns

],
    "errors": [
        {
            "code": 12015,
            "msg": "Index scan timed out - cause: Index scan timed out"
        }
    ],
    "status": "errors",
    "metrics": {
        "elapsedTime": "4m28.956528875s",
        "executionTime": "4m28.956473392s",
        "resultCount": 9,
        "resultSize": 2140,
        "sortCount": 9,
        "errorCount": 1
    }

I was constantly running the second query (SUBSTR(t, 0, 10) >= "2017-03-03"), then I randomly run the first query, then again the second.

Server resources are limited (and I completely understand it being a limiting factor); moreover data from 9 devices are constantly streamed in the database 24/7.

However, it’s weird to me why would that happen.

can you check/post the EXPLAIN output for the queries. Also post the index definitions and server version and cluster setup details. Note that, the second query may be picking multiple indexes, in which case, post all index defs. thanks.

Version: 4.5.1-2844 Enterprise Edition (build-2844)
Cluster: Single Node / Master / No Replication

agora_bucket	127.0.0.1:8091	#primary	Standard Global Secondary Indexes	Ready	100 %
Definition: CREATE PRIMARY INDEX `#primary` ON `agora_bucket`
agora_bucket	127.0.0.1:8091	bcn_scan-date-did	Standard Global Secondary Indexes	Ready	100 %
Definition: CREATE INDEX `bcn_scan-date-did` ON `agora_bucket`(substr(`t`, 0, 10),`did`) WHERE (`type` = "bcn_scan")

queries-EXPLAIN.zip (2.5 KB)

There you go

Hi @sntentos, plans look normal to me. Can you check query.log and indexer.log for any warning/errors etc.

Can you try these:

  1. use MOI indexes

  2. change max-parallelism query setting to number of cores in your box. For ex:
    curl -u Administrator:password http://localhost:8093/admin/settings -XPOST -d ‘{“max-parallelism”:8}’

  3. increase time-out setting. For ex: this will set timeout to 300sec.
    curl -u Administrator:password http://localhost:8093/admin/settings -XPOST -d ‘{“timeout”:300}’

  4. Use second query with #primary index.
    SELECT did, ARRAY_AGG(DISTINCT SUBSTR(t, 0, 10)) AS days, {
    “sums” : COUNT(),
    “null” : COUNT(
    ) - COUNT(NULLIF(ARRAY_LENGTH(bcn), 0)),
    “full” : COUNT(NULLIF(ARRAY_LENGTH(bcn), 0))
    } AS entries
    FROM agora_bucket
    USE INDEX (`#primary`)
    WHERE type=“bcn_scan” AND SUBSTR(t, 0, 10) >= "2017-03-03"
    GROUP BY did
    ORDER BY did ASC;

Here are my logs query.log from the day - indexer unfortunately does not go that back.
They are also too massive, maybe I could somehow make them more meaningful?

query_2017-03-08.zip (62.1 KB)

I’d have to destroy all the DB indexes for now, so I tried to change the other parameters.
Also, with regards to this, I get that I will actually need more memory, and if that runs out, indexes are dead in the water. Since I already mentioned

This option, while more efficient, doesn’t strike me as a real alternative. I mean my indexes (for some unknown reason I don’t know) seem to be already half-way dead - but higher RAM requirements are a no-go.

I’d be interested if there was a way to force the indexes to sync with current data, even if that means that I will have to temporarily deny any entries coming my way.

I also have no idea how the internals of the DB work - but it seems counter-intuitive to me that a ‘limiting’ clause actually has an adverse effect on returning results.


When I changed the max-parallelism, I noticed:

{“completed-limit”:4000,“completed-threshold”:1000,“cpuprofile”:“”,“debug”:false,“keep-alive-length”:16384,“loglevel”:“INFO”,“max-parallelism”:8,“memprofile”:“”,“pipeline-batch”:16,“pipeline-cap”:512,“pretty”:true,“request-size-cap”:67108864,“scan-cap”:0,“servicers”:16,“timeout”:0}

What does that mean?

With that setting, my query timeouts at 5m34s.

Increasing the timeouts to 1200, 1200000000000, 1200000000000000 I get respective timeouts at 1.2 μs, 6m21s, 5m57s.

Only USE INDEX (#primary) seems to be working - with a whomping execution time of 39m40.559550172s!

Hi @sntentos,

The timeout in the query service is the total query execution time.
Index service has a separate scan timeout, default to 2 minutes.
@deepkaran.salooja can help you with setting the scan timeout for indices.

Index scan timeout can be configured using the below:

curl -X POST -u <user:pass> http://<ip>:9102/settings --data '{"indexer.settings.scan_timeout": <timeout_in_milliseconds>}'

@deepkaran.salooja - I am hitting same issue : Can you please help what is this 9102 port ?

@eldorado, 9102 is the indexer http port.

I see that is listed in CB documents but where do I have this information available from running nodes in K8s .
I have 4 active nodes running all services and kubectl get services not showing me any port called 9102 . Can this be accessible otherway ? I can’t use this port as its not open but I have index services running and I can see 8091 … 8096 …

You can use curl get current port numbers for each service

https://docs.couchbase.com/server/6.5/rest-api/rest-list-node-services.html#curl-syntax

I am not expert in k8. You can login individual k8 and use the command so that port is accessible. cc @anil
Sill need help in k8 you can post in CouchbaseServer/kubernetes category for faster response.

thanks @vsr1 … I will wait for Anil Gupta to respond … I do see below in my curl request but how indexHttp parameter config is corresponds to any K8s open port to access those settings.

Also below throwing errors for me : curl -u Administrator:password -XGET http://10.30.xxx.xxx:9102/settings

curl -u Administrator:password -XGET http://10.30.xxx.xxx:32161/pools/default/nodeServices | jq

"{
  "rev": 92,
  "nodesExt": [
    {
      "services": {
        "mgmt": 8091,
        "mgmtSSL": 18091,
        "cbas": 8095,
        "cbasCc": 9111,
        "cbasAdmin": 9110,
        "cbasSSL": 18095,
        "eventingAdminPort": 8096,
        "eventingDebug": 9140,
        "eventingSSL": 18096,
        "fts": 8094,
        "ftsSSL": 18094,
        "indexAdmin": 9100,
        "indexScan": 9101,
        "indexHttp": 9102,
        "indexStreamInit": 9103,
        "indexStreamCatchup": 9104,
        "indexStreamMaint": 9105,
        "indexHttps": 19102,
        "capiSSL": 18092,
        "capi": 8092,
        "kvSSL": 11207,
        "projector": 9999,
        "kv": 11210,
        "moxi": 11211,
        "n1ql": 8093,
        "n1qlSSL": 18093
      },
      "hostname": "cluster-0000.dc-cluster.bi-cb.svc",
      "alternateAddresses": {
        "external": {
          "hostname": "10.30.xxx.xxx",
          "ports": {
            "cbas": 30065,
            "cbasSSL": 30334,
            "fts": 30625,
            "ftsSSL": 31725,
            "kv": 31744,
            "kvSSL": 31091,
            "capi": 31988,
            "capiSSL": 31128,
            "n1ql": 30854,
            "n1qlSSL": 32480
          }
        }
      }
    },
    {
      "services": {
        "mgmt": 8091,
        "mgmtSSL": 18091,
        "cbas": 8095,
        "cbasCc": 9111,
        "cbasAdmin": 9110,
        "cbasSSL": 18095,
        "eventingAdminPort": 8096,
        "eventingDebug": 9140,
        "eventingSSL": 18096,
        "fts": 8094,
        "ftsSSL": 18094,
        "indexAdmin": 9100,
        "indexScan": 9101,
        "indexHttp": 9102,
        "indexStreamInit": 9103,
        "indexStreamCatchup": 9104,
        "indexStreamMaint": 9105,
        "indexHttps": 19102,
        "capiSSL": 18092,
        "capi": 8092,
        "kvSSL": 11207,
        "projector": 9999,
        "kv": 11210,
        "moxi": 11211,
        "n1ql": 8093,
        "n1qlSSL": 18093
      },
      "thisNode": true,