Home | Forums | Couchbase | Couchbase Server 1.8.x

Deletion of default bucket in a 4 node cluster caused auto-failover

1 reply [Last post]
Thu, 11/08/2012 - 16:48
owentran
Offline
Joined: 10/18/2012
Groups: None

We have a 4x96 Gb couchbase cluster running 1.8.1 with two buckets. While removing the default bucket through the web interface it took over 30 seconds and then timed out. Then all the nodes tried to auto-failover (since we have a 30 second value). Fortunately only one node went down and the other three remained up because it saw that one node was already down.

A few questions:

  1. Is there something special about deleting the default bucket?
  2. Why would it block for 30 seconds? We didn't have this issue removing named buckets.
  3. The default bucket wasn't used in over a week. We don't understand why it would take so long to remove/delete the default bucket since all the objects would have been expired.
  4. Is it safe to bring back the failed node back?

Raw logs:

Failed over 'ns_1@10.33.5.4': ok <span class="Apple-tab-span" style="white-space: pre; "> </span>ns_orchestrator006 <span class="Apple-tab-span" style="white-space: pre; "> </span>ns_1@10.33.5.6 <span class="Apple-tab-span" style="white-space: pre; "> </span>22:53:51 - Thu Nov 8, 2012
 
Shutting down bucket "release-123" on 'ns_1@10.33.5.4' for deletion <span class="Apple-tab-span" style="white-space:pre"> </span>ns_memcached002 <span class="Apple-tab-span" style="white-space:pre"> </span>ns_1@10.33.5.4 <span class="Apple-tab-span" style="white-space:pre"> </span>22:53:51 - Thu Nov 8, 2012
 
Starting failing over 'ns_1@10.33.5.4' <span class="Apple-tab-span" style="white-space:pre"> </span>ns_orchestrator000 <span class="Apple-tab-span" style="white-space:pre"> </span>ns_1@10.33.5.6 <span class="Apple-tab-span" style="white-space:pre"> </span>22:53:51 - Thu Nov 8, 2012
 
Could not auto-failover node ('ns_1@10.33.5.5'). There was at least another node down.
 
<span class="Apple-tab-span" style="white-space:pre"> </span>auto_failover003 <span class="Apple-tab-span" style="white-space:pre"> </span>ns_1@10.33.5.6 <span class="Apple-tab-span" style="white-space:pre"> </span>22:53:31 - Thu Nov 8, 2012
 
Updated bucket release-123 (of type membase) properties:
 
[{ram_quota,89128960000},{auth_type,sasl}] <span class="Apple-tab-span" style="white-space:pre"> </span>menelaus_web_buckets000 <span class="Apple-tab-span" style="white-space:pre"> </span>ns_1@10.33.5.3 <span class="Apple-tab-span" style="white-space:pre"> </span>22:53:28 - Thu Nov 8, 2012
 
Could not auto-failover node ('ns_1@10.33.5.6'). There was at least another node down.
 
<span class="Apple-tab-span" style="white-space:pre"> </span>auto_failover003 <span class="Apple-tab-span" style="white-space:pre"> </span>ns_1@10.33.5.6 <span class="Apple-tab-span" style="white-space:pre"> </span>22:53:27 - Thu Nov 8, 2012
 
Deleted bucket "default"
 
<span class="Apple-tab-span" style="white-space:pre"> </span>menelaus_web011 <span class="Apple-tab-span" style="white-space:pre"> </span>ns_1@10.33.5.3 <span class="Apple-tab-span" style="white-space:pre"> </span>22:52:35 - Thu Nov 8, 2012
 
Shutting down bucket "default" on 'ns_1@10.33.5.3' for deletion <span class="Apple-tab-span" style="white-space:pre"> </span>ns_memcached002 <span class="Apple-tab-span" style="white-space:pre"> </span>ns_1@10.33.5.3 <span class="Apple-tab-span" style="white-space:pre"> </span>22:52:05 - Thu Nov 8, 2012
 
Shutting down bucket "default" on 'ns_1@10.33.5.4' for deletion <span class="Apple-tab-span" style="white-space:pre"> </span>ns_memcached002 <span class="Apple-tab-span" style="white-space:pre"> </span>ns_1@10.33.5.4 <span class="Apple-tab-span" style="white-space:pre"> </span>22:52:05 - Thu Nov 8, 2012
 
Shutting down bucket "default" on 'ns_1@10.33.5.5' for deletion <span class="Apple-tab-span" style="white-space:pre"> </span>ns_memcached002 <span class="Apple-tab-span" style="white-space:pre"> </span>ns_1@10.33.5.5 <span class="Apple-tab-span" style="white-space:pre"> </span>22:52:05 - Thu Nov 8, 2012

 

Top
Mon, 11/12/2012 - 04:57
Neo-matrix
Offline
Joined: 10/15/2012
Groups: None

Hi,
"since buckets could be renamed and there is no way to determine the name for the default
bucket for a cluster, the system will attempt to connect non-SASL, non-proxied to a bucket clients to
a bucket named "default". If it does not exist, Couchbase Server will drop the connection."

Reference "8.14. Viewing Cluster Details" in the manual 1.8.1.

Let me know if it helps/answers your question.

Thanks
Neo

Top