LOOOOONG rebalance - w/errors

From your explanation of the symptoms, it appears
- periodically one (or more of the nodes) which happen to be replication destination, is hitting a 1Million item mark in the Disk Write Queue. when this happens the node sends a backoff to the rest of the cluster, so that it can drain the queue before resuming its duties of participating in a rebalance operation
- your resident ratio (items in memory v/s disk storage) is low. Doing a Rebalance operation in a relatively high (disk greater than memory) scenario will impede the rebalance, as items from disk have to progressively be moved up to memory before being shuffled around for replication / rebalance. Curious to see if you can track the resident ratio (at the cluster level as well as individual nodes) during a rebalance and see if that number rapidly drops down.

At the root of all this could also be the problem with fragmentation at the disk persistence layer(sqlite files) that usually results in slow disk writes / reads and consequently a loooooooooong rebalance.

'm sure you are asking, so what's the way out? it depends-

- If you are able to tolerate a full downtime, then bringing all the nodes down + performing a vacuum of the sqllite files and then bring 'em back up. Then adding the addl nodes and Rebalance should yield the best result
- If you can only manage a partial downtime, then taking a single node or couple of nodes out, then vacuuming the files and then adding them back to the cluster should help. In both this and the above scenario, the affected data will not be available for the clients during the downtime.
- a 3rd option is to wait for the next release of Couchbase which has a capability to do a contained rebalance operation

In the longer run, you may want to look at improving the resident ratio. Hope this helps..

Thanks for the response...

I don't see any "Disk Write Queue" in the GUI

Add more nodes and therefore memory should improve the resident ratio - right?

How do I perform a vacuum of the sqllite files??

Just to be sure.

I should stop this rebalance
Remove the 4 new nodes (there isn't much on them)
Rebalance
Shut down the existing nodes
SQLlite vaccuum
Bring up the nodes (I won't lose any data right :-) )
Add the new nodes
Re-balance

Look good?

Monitor->ServerNodes Cick on the node in question and then Expand the "Summary" category. Disk write queue will be one of the counters towards the lower right hand corner. You may navigate your cursor to the counter and then click on the blue arrow to view the counter by server

Yes, adding more nodes that increases the overall RAM capacity of the cluster will definitely improve your Resident ratio.

The data files are normally stored under the /var/lib/Couchbase/data/-data/
where is the name of the bucket that you are going to vacuum. The files under this directory will be called -0.mb, -1.mb, -2.mb, -3.mb

To vacuum the files, run the following commands as the 'Couchbase' user to defragment each of the sqlite data files:

<Couchbase_install_location>/bin/sqlite3 <couchbase dir>/var/lib/Couchbase/data/<bucket_name>-data/<bucket_name>-0.mb 'VACUUM;'
<Couchbase_install_location>/bin/sqlite3 <couchbase dir>/var/lib/Couchbase/data/<bucket_name>-data/<bucket_name>-1.mb 'VACUUM;'
<Couchbase_install_location>/bin/sqlite3 <couchbase dir>/var/lib/Couchbase/data/<bucket_name>-data/<bucket_name>-2.mb 'VACUUM;'
<Couchbase_install_location>/bin/sqlite3 <couchbase dir>/var/lib/Couchbase/data/<bucket_name>-data/<bucket_name>-3.mb 'VACUUM;'

In step 2 where you say Remove the 4 new nodes, I assume you are referring to Doing a "Remove Server" from the console. That'd be a preferred method. The next best would be "Fail over". Just shutting down the node or performing a kill -9 may not be a good idea, however small the data they hold.

Just a caution note, the vacuum process does need 2x diskspace to create temp files etc. So you may want to make sure you have enough disk space to perform the vacuum.