Api/v1/stats is returning only the replica index

http://XXXXX:9102/api/v1/stats/BUCKET/idx_BUCKET_ACCOUNT_BALANCE?pretty=true&partition=true

is returning only the replica index, although the master index is also there. Am I doing anything wrong?

{

    "LIQUIDITY:idx_BUCKET_ACCOUNT_BALANCE (replica 1)": {

        "avg_drain_rate": 0,

        "avg_item_size": 0,

        "avg_scan_latency": 0,

        "cache_hit_percent": 0,

        "cache_hits": 0,

        "cache_misses": 0,

        "data_size": 65536,

        "disk_size": 32768,

        "frag_percent": 0,

        "initial_build_progress": 100,

        "items_count": 0,

        "last_known_scan_time": 1632960566835276957,

        "memory_used": 0,

        "num_docs_indexed": 0,

        "num_docs_pending": 0,

        "num_docs_queued": 0,

        "num_items_flushed": 0,

        "num_pending_requests": 0,

        "num_requests": 0,

        "num_rows_returned": 0,

        "num_scan_errors": 0,

        "num_scan_timeouts": 0,

        "recs_in_mem": 0,

        "recs_on_disk": 0,

        "resident_percent": 0,

        "scan_bytes_read": 0,

        "total_scan_duration": 0

    },

    "Partition-1": {

        "avg_drain_rate": 0,

        "avg_item_size": 0,

        "avg_scan_latency": 0,

        "cache_hit_percent": 0,

        "cache_hits": 0,

        "cache_misses": 0,

        "data_size": 16384,

        "disk_size": 8192,

        "frag_percent": 0,

        "initial_build_progress": 0,

        "items_count": 0,

        "last_known_scan_time": 0,

        "memory_used": 0,

        "num_docs_indexed": 0,

        "num_docs_pending": 0,

        "num_docs_queued": 0,

        "num_items_flushed": 0,

        "num_pending_requests": 0,

        "num_requests": 0,

        "num_rows_returned": 0,

        "num_scan_errors": 0,

        "num_scan_timeouts": 0,

        "recs_in_mem": 0,

        "recs_on_disk": 0,

        "resident_percent": 0,

        "scan_bytes_read": 0,

        "total_scan_duration": 0

    },

    "Partition-2": {

        "avg_drain_rate": 0,

        "avg_item_size": 0,

        "avg_scan_latency": 0,

        "cache_hit_percent": 0,

        "cache_hits": 0,

        "cache_misses": 0,

        "data_size": 16384,

        "disk_size": 8192,

        "frag_percent": 0,

        "initial_build_progress": 0,

        "items_count": 0,

        "last_known_scan_time": 0,

        "memory_used": 0,

        "num_docs_indexed": 0,

        "num_docs_pending": 0,

        "num_docs_queued": 0,

        "num_items_flushed": 0,

        "num_pending_requests": 0,

        "num_requests": 0,

        "num_rows_returned": 0,

        "num_scan_errors": 0,

        "num_scan_timeouts": 0,

        "recs_in_mem": 0,

        "recs_on_disk": 0,

        "resident_percent": 0,

        "scan_bytes_read": 0,

        "total_scan_duration": 0

    },

    "Partition-4": {

        "avg_drain_rate": 0,

        "avg_item_size": 0,

        "avg_scan_latency": 0,

        "cache_hit_percent": 0,

        "cache_hits": 0,

        "cache_misses": 0,

        "data_size": 16384,

        "disk_size": 8192,

        "frag_percent": 0,

        "initial_build_progress": 0,

        "items_count": 0,

        "last_known_scan_time": 0,

        "memory_used": 0,

        "num_docs_indexed": 0,

        "num_docs_pending": 0,

        "num_docs_queued": 0,

        "num_items_flushed": 0,

        "num_pending_requests": 0,

        "num_requests": 0,

        "num_rows_returned": 0,

        "num_scan_errors": 0,

        "num_scan_timeouts": 0,

        "recs_in_mem": 0,

        "recs_on_disk": 0,

        "resident_percent": 0,

        "scan_bytes_read": 0,

        "total_scan_duration": 0

    },

    "Partition-7": {

        "avg_drain_rate": 0,

        "avg_item_size": 0,

        "avg_scan_latency": 0,

        "cache_hit_percent": 0,

        "cache_hits": 0,

        "cache_misses": 0,

        "data_size": 16384,

        "disk_size": 8192,

        "frag_percent": 0,

        "initial_build_progress": 0,

        "items_count": 0,

        "last_known_scan_time": 0,

        "memory_used": 0,

        "num_docs_indexed": 0,

        "num_docs_pending": 0,

        "num_docs_queued": 0,

        "num_items_flushed": 0,

        "num_pending_requests": 0,

        "num_requests": 0,

        "num_rows_returned": 0,

        "num_scan_errors": 0,

        "num_scan_timeouts": 0,

        "recs_in_mem": 0,

        "recs_on_disk": 0,

        "resident_percent": 0,

        "scan_bytes_read": 0,

        "total_scan_duration": 0

    }

}

@rajib761 The stats API only returns local stats from the Index node you send the REST call to – it does not do a scatter-gather to all Index nodes. To see the stats from another Index node you can issue the call again to the desired Index hostname. The master and replica will always be on different nodes – unless you have set up the cluster with multiple virtual nodes all on the same physical machine. In that case the hostname will be the same but the port numbers of the virtual nodes will be different, normally separated by +6 (9102, 9108, 9114, 9120…), so you can contact each virtual Index node by sending the REST call to its associated port. As far as the Index service is concerned, virtual nodes are considered different nodes, so master and replica will be on different virtual nodes even when all virtual nodes are sharing a single physical machine.

From the doc page

https://docs.couchbase.com/server/current/rest-api/rest-index-stats.html

Description

Returns statistics for an index node, and for all indexes on that node.

@Kevin.Cherkauer
Hi Kevin
I am posting the image of the index. It is a partition index. Both the master and replica index are on this node(different partitions). let me explain what I am trying to solve. I wanted to see if partition elimination is happening, for that I am writing a python code with below logic

first I do a /getIndexStatus to get the partition map for the index
Then using the partition map, I look at /api/v1/stats/ to see the num_requests value in each partition to check if partition elimination is happing

The problem I am facing is that /getindexStatus is giving me the correct partiton number which is Partition-3 for the main index, but when i do the next api call (/api/v1/stats), Partition-3 is not there because it is returning the partitions for the replica index

api/v1/stats/

get_index
getIndexStatus

api_v1_stats

Additional finding
------------------------
If I hit the /api/v1/stats//?partition=true multiple times, sometimes it is bringing back the main index sometimes the replica, looks like it is load balancing between the two. Should this behave like this?

@rajib761 /getIndexStatus API does a scatter-gather and returns information about all indexes and partitions on all Index nodes, i.e. global results. By default it also consolidates all results for a given index definition (all partitions of all replicas of the index across all nodes holding any of them) into a single entry, but you can also get the unconsolidated information by adding the ?getAll=true flag (/getIndexStatus?getAll=true).

/v1/stats does not do scatter-gather; it only returns the stats about indexes and partitions on the node the call is sent to.

Example
An index has 3 partitions and 1 replica, for a total of 6 objects. (The partitions will be numbered 1, 2, 3. Note that non-partitioned indexes are treated as having a single partition numbered 0, while partitioned indexes never have a partition 0.) The Index service will ensure that the master and replica of each individual partition are never on the same node, but the nodes hosting each individual partition can vary. If you have four Index nodes the index topology might be

Host 1 partitions

  • master [1 3]
  • replica [2]

Host 2 partitions

  • master [2]

Host 3 partitions

  • replica [1 3]

Host 4 partitions

  • (none)

/getIndexStatus will return information about all of these, consolidated or not based on the getAll flag. The Hosts array field of consolidated results lists all the nodes holding any pieces of the index (in this case Host 1, Host 2, Host 3). In unconsolidated results there will be separate entries for Host 1, Host 2, and Host 3 for this index each of whose Hosts array will have only one entry.

/v1/stats will return only information about the partitions on the node servicing this REST call, so if it comes to Host 1 you will get info only about master [1 3] and replica [2] partitions, whereas if it comes to Host 4 you will get no stats for this index at all because this node has none of its partitions either for the master or replica. To get the full stats information for this index you need to call /v1/stats on each node that contains any piece of the index (i.e. Hosts 1, 2, and 3, but not Host 4).

If there is some front-end load balancer that is directing your /v1/stats call to an arbitrary Index node, then you will get back the stats for whichever random node it sent the call to, which can make the results appear nondeterministic.

Hi Kevin, thanks for the detailed response. But I am still not able to understand why the api/v1/stats is showing master[13] sometimes and replica[2] other times. i am not using any load balancer. I am calling this rest endpoint from POSTMAN and also through PYTHON. In both cases it is toggling between the two

@rajib761 Is it possible you have configured two virtual nodes on a single host that are using the same port, or maybe a prior cluster did not shut down before a new one was started so there are two independent clusters listening to the same port?

No Kevin, i have three separate nodes on Azure

Hi @rajib761 , which Couchbase server version are you using?

There was a bug in api/v1/stats for indexes, MB-43438. This bug is fixed in version 6.6.3.

Another forum post had reported it:

Are you observing the same behaviour ?

Thanks.

1 Like

Yes Amit, this is the same behavior. I am using 6.6.2