Server run out of space and now is crashing with the below error

is there a way to repair it, or the only option is to lose all the bucket data?

[error_logger:error,2024-09-01T03:15:46.088Z,ns_1@img01.cbi:logger_proxy<0.69.0>:ale_error_logger_handler:do_log:101]
=========================CRASH REPORT=========================
  crasher:
    initial call: misc:turn_into_gen_server/4
    pid: <16652.29282.0>
    registered_name: 'capi_set_view_manager-images'
    exception exit: {error,
                        {select_bucket_failed,
                            {memcached_error,key_enoent,undefined}}}
      in function  capi_set_view_manager:wait_for_bucket_to_start/2 (src/capi_set_view_manager.erl, line 193)
      in call from capi_set_view_manager:init/1 (src/capi_set_view_manager.erl, line 154)
      in call from misc:turn_into_gen_server/4 (src/misc.erl, line 503)
    ancestors: [<0.903.0>,'single_bucket_kv_sup-images',ns_bucket_sup,
                  ns_bucket_worker_sup,ns_server_sup,ns_server_nodes_sup,
                  <0.299.0>,ns_server_cluster_sup,root_sup,<0.145.0>]
    message_queue_len: 0
    messages: []
    links: [<0.903.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 2586
    stack_size: 29
    reductions: 48785
  neighbours:

it looks like all the data is there but the server doesn’t want to come up

also, I am seeing these error messages:

[ns_server:info,2024-09-01T03:15:45.738Z,babysitter_of_ns_1@cb.local:<0.134.0>:ns_port_server:log:226]memcached<0.134.0>: 2024-09-01T03:15:45.538439+00:00 ERROR 186: Create bucket [images] failed - Already exists

why would it try to create a bucket with the same name?

:ns_memcached:handle_info:744]ensure_bucket failed as bucket "images" has not completed coming online

another error that I see is that is saying on another bucket that there is no more space available. When I do df -h, there are 28GB of available disk

compaction_daemon:do_chain_compactors:654]Compactor for vbucket `images_fts/12` (pid [{type,vbucket},
                                            {name,<<"images_fts/12">>},
                                            {important,false},
                                            {fa,
                                             {#Fun<compaction_daemon.13.82522328>,
                                              [<<"images_fts">>,
                                               {12,<<"images_fts/12">>},
                                               {config,
                                                {undefined,409600},
                                                {20,undefined},
                                                undefined,false,false,
                                                {daemon_config,30,131072,
                                                 20971520}},
                                               false,
                                               {1725155561,0,false}]}}]) terminated unexpectedly (ignoring this): {not_enough_space,
                                                                                                                   <<"images_fts/12">>,
                                                                                                                   441656646,
                                                                                                               0}

To provide any help with an issue like this we’d generally need more info - it’s near impossible to give any valid/safe guidance from the small log snippets provided - e.g. maybe you’ve hit a known bug in some version (what version are you).

In general you would just fail this node out of your cluster, can you do that?

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.