FTS reindex while rebalance

Full Text Search
1

We have a cluster with:

5 x data
2 x index/query
2 x fts

We needed to change data directory, so we decided to remove each node, rebalance, setup new data directory, attach to cluster, rebalance.

1.) First we removed a data node (there are 2 replicas on the bucket) and rebalanced. FTS started to reindex everything while doing that. After we re-attached the node to the cluster and rebalanced again , FTS started to reindex everything again. Is that how it should work? And why? While we have replicas, the data should not change for FTS. Indexing ~ 4M documents in fts needs some time, so it seems that failover/rebalance on data causes a interrupt on fts service until everything is indexed again, right?

2.) We did the same on fts nodes (remove - rebalance - reattach to cluster - rebalance), then we got:

Rebalance exited with reason {service_rebalance_failed,fts,
{linked_process_died,<21046.1283.3143>,
{badmatch,
{false,
{topology,,
[<<“8ba82fa472ec490ad26936ac3b6ccfa6”>>,
<<“9b0cf23579009d052e3ec67c1e3a3dbd”>>],
false,
[<<“warning: resource: “ftsde_v1” – could not meet replication constraints”>>,
<<“warning: resource: “ftsfr_v1” – could not meet replication constraints”>>]},
{topology,,
[<<“8ba82fa472ec490ad26936ac3b6ccfa6”>>,
<<“9b0cf23579009d052e3ec67c1e3a3dbd”>>],
false,
[<<“warning: resource: “ftsfr_v1” – could not meet replication constraints”>>,
<<“warning: resource: “ftsde_v1” – could not meet replication constraints”>>]}}}}}

Reblanced completed then after some retries.

2

hi @gizmo74,

Can you please reach out to customer support team for taking the first issue further. (if you have license contact)

For the second issue you posted - we have observed this problem in 6.0.x releases.
Those “could not meet replication constraints” warning comes when the index has a replica count configured, but there aren’t enough nodes to support that many replicas.

Such rebalance failures are ought to get solved if you update the index definitions with zero replica count or with right replica count. eg - (replica count = no of nodes -1 )

Beware that updating the index definitions with replica count changes would result in the index rebuild.

Also NOTE - both these issues (index rebuild on replica count update and the rebalance failures) are fixed in the latest/upcoming 6.5.0 release.

Cheers!