Index initial build times taking forever since cluster rebalance

I have a small bucket with a little over 6000 documents in it. Index creation used to take no more than 5-10 seconds.

This afternoon, I removed each node from the cluster (2 total), one at a time. Doing a rebalance each time. Ive done this many times before.

However this time, once both nodes were connected again, any index creation query takes ages to complete. Several minutes. I tried restarting the nodes, but nothing has fixed the issue. I tried creating indexes with each node isolated from each other and still same issue. I compacted the bucket, in case that would solve it, but still, same problem.

Im running 4.5.1 CE. Any ideas? Nothing in the logs is hinting to an issue.

Just an update on this. I had to reinstall couchbase on both nodes for the slowness issues to go away. Any insight on the matter would be useful, as this seems to be a major detriment to removing and re-adding nodes to a cluster. In a production environment, it would be unacceptable.

@kev, without looking at the logs it is difficult to say what went wrong. Usually you would see logs complaining about port connectivity issues or lack of file handles.

I assume the indexes in question are GSI indexes. Rebalance is done by production customers all the time without any downtime or slowness issue you mentioned.

I agree with you. I can only imagine something went wrong that normally does not happen. All I know is that a full re-install solved the issue right away.