During index creation (I build 39 differed index - 3*13 index - as I use CE and cannot set replica) , All server fulled disk space , and now I cannot access couchbase web console and couchbase-cli
What must I do?
@vsr1 , @graham.pople , @ingenthr any idea?
Oh! fortunately I have this issue in development , But I’m very worry about production , What happens of I have full disk , lost whole service and after 3 days I have no response , 3 day down time? with couchbase! it’s crazy
What do you mean by:
All your nodes have full disk? Not just index nodes?
I am developing (and experiencing) my app with couchbase , Currently I have 3 nodes in development mode and each node has index and data services
I create 39 (3*13) deferred indexes , As I use CE , and I cannot see replica, I create double index, when I built indexes , my disk fulled in 3 nodes
I test with 20 million data
After that, I lost access to couchbase, In production , what must I do to?
Is it safe to delete index folder? To have some free space? What is the best way to fix it?
Yes, I believe it should be safe to delete the “@2i” folder on all the nodes if that is what is actually consuming the disk space.
You can then restart Couchbase Server on all of the nodes and then redefine the indexes.
As you mentioned this is a test cluster, so I assume you are okay with doing the above.
Sounds like an undersized cluster, especially for a 20 million items.
It might be better if you use separate partitions for the index vs. data directory.
Yes, I believe it should be safe to delete the “@2i” folder on all the nodes if that is what is actually consuming the disk space.
is it official couchbase recommendation?
As you mentioned this is a test cluster, so I assume you are okay with doing the above.
What about production?
For production setup, if all servers do not run into disk full problem at the same time, you can failover the one with problem and rebalance it in again. For the situation like above, the only option would be to cleanup the @2i folder.
Couchbase has an alert mechanism to warn users when disk space reaches 90%(I guess it works for CE as well).
It would be better to switch to using circular mode for index compaction to have better managed disk space usage (https://docs.couchbase.com/server/5.1/settings/configure-compact-settings.html).
For the next upcoming release(6.5), you’ll have the ability to drop index even when the indexer is crashing(https://issues.couchbase.com/browse/MB-32755). This will allow cleanup of a subset of indexes to get the system back online.
1 Like