Server Crash - {write_loop_died,{badmatch,{error,enospc}}}

Aleksey Kondratenko added a comment - 29/Feb/12 11:01 AM no space means no space. There's not much you can do when you exhaust FS space

Tommie McAfee added a comment - 29/Feb/12 11:14 AM did compaction fail? The cluster has 120gb space.

Tommie McAfee added a comment - 29/Feb/12 12:27 PM Spoke with Filipe about this....who explained that compaction doesn't start until indexing finishes. So what happens is my couch data disk size is 3.5 GB and Couchbase is going to try and create main and replica index files for 20 view, and worst case(if I re-emit the entire db), my cluster would have to reserve an extra 140 GB (3.5gb*40) for queries. Filipe says it's possible to implement some sort of incremental compaction or possibly giving compaction threads priority when necessary.

Tommie McAfee added a comment - 29/Feb/12 12:28 PM UI view of disk usage overhead

Damien Katz added a comment - 13/Apr/12 3:15 PM Filipe, can you look at this. If this bug is a invalid or limitation, just mark as Won't fix with a small explanation.

Filipe Manana added a comment - 16/Apr/12 9:38 AM Unfortunately once we get out of disk space, we can't have query views with ?stale=ok or ?stale=update_after (default). We get a file_error from within ns_server (somewhere in the HTTP handlers / ALE logger): $ curl ' http://localhost:9500/default/_design/test/_view/view1?limit=10&#39; {"error":"badmatch","reason":"{error,{file_error,\"logs/n_0/log\",enospc}}"} The relevant full stack trace: [menelaus:warn] [2012-04-16 15:10:51] [ n_0@192.168.1.80 :<0.29010.0>:menelaus_web:loop:358] Server error during processing: ["web request failed",                                  {path,"/pools/default"},                                  {type,error},                                  {what,function_clause},                                  {trace,                                   [{menelaus_stats,                                     '-invoke_archiver/3-lc$^0/1-0-',                                     [{'EXIT',                                       {{badmatch,                                         {error,                                          {file_error,"logs/n_0/log",enospc}}},                                        [{'ale_logger-stats',error,5},                                         {stats_reader,latest,4},                                         {menelaus_stats,invoke_archiver,3},                                         {menelaus_stats,last_membase_sample,2},                                         {menelaus_stats,last_bucket_stats,3},                                         {menelaus_stats,basic_stats,3},                                         {ns_storage_conf,                                          '-do_cluster_storage_info/1-fun-2-',                                          3},                                         {lists,foldl,3}]}}]},                                    {menelaus_stats,last_membase_sample,2},                                    {menelaus_stats,last_bucket_stats,3},                                    {menelaus_stats,basic_stats,3},                                    {ns_storage_conf,                                     '-do_cluster_storage_info/1-fun-2-',3},                                    {lists,foldl,3},                                    {ns_storage_conf,do_cluster_storage_info,1},                                    {menelaus_web,build_pool_info,4}]}] Technically the view engine is capable of serving queries with stale=ok|update_after if there's no space left on disk, as long as the logger doesn't crash when there's no disk space left. Queries with ?stale=false will always get an error mentioning the posix error code 'enospc'.

Filipe Manana added a comment - 16/Apr/12 9:40 AM Damien, what's your call?

Damien Katz added a comment - 17/Apr/12 3:27 PM Running out of disk space shouldn't cause corruptions, but other than that we cannot do anything. A possible future feature is to have an admin function to purge all indexes, which will have them rebuilt from scratch, but will take a lot of disk IO and potential application downtime, that might be easily resolved in another way by the administrator.