We have a cluster with:
5 x data
2 x index/query
2 x fts
We needed to change data directory, so we decided to remove each node, rebalance, setup new data directory, attach to cluster, rebalance.
1.) First we removed a data node (there are 2 replicas on the bucket) and rebalanced. FTS started to reindex everything while doing that. After we re-attached the node to the cluster and rebalanced again , FTS started to reindex everything again. Is that how it should work? And why? While we have replicas, the data should not change for FTS. Indexing ~ 4M documents in fts needs some time, so it seems that failover/rebalance on data causes a interrupt on fts service until everything is indexed again, right?
2.) We did the same on fts nodes (remove - rebalance - reattach to cluster - rebalance), then we got:
Rebalance exited with reason {service_rebalance_failed,fts,
{linked_process_died,<21046.1283.3143>,
{badmatch,
{false,
{topology,,
[<<“8ba82fa472ec490ad26936ac3b6ccfa6”>>,
<<“9b0cf23579009d052e3ec67c1e3a3dbd”>>],
false,
[<<“warning: resource: “ftsde_v1” – could not meet replication constraints”>>,
<<“warning: resource: “ftsfr_v1” – could not meet replication constraints”>>]},
{topology,,
[<<“8ba82fa472ec490ad26936ac3b6ccfa6”>>,
<<“9b0cf23579009d052e3ec67c1e3a3dbd”>>],
false,
[<<“warning: resource: “ftsfr_v1” – could not meet replication constraints”>>,
<<“warning: resource: “ftsde_v1” – could not meet replication constraints”>>]}}}}}
Reblanced completed then after some retries.