Hi there,
Today we faced some issues with connection query nodes. So we removed query and data pods for auto re-creation by the operator. But, it couldn’t be successful. So, I want to remove the couchbase cluster for full re-creation by the k8s couchbase operator. So I needed a get a full backup but I can’t backup because of cbbackupmgr trying to connect the query node because it doesn’t exist. So, what do suggest for me? I don’t want to lose data, but I can’t recover the system Some logs are below from CB UI:
Resetting rebalance status since rebalance stop was requested but rebalance isn’t orchestrated on our node
ns_orchestrator 000
ns_1@couchbase-0000.couchbase.database.svc
10:17:30 AM 4 Sep, 2023
Resetting rebalance status since rebalance stop was requested but rebalance isn’t orchestrated on our node (repeated 3 times, last seen 0.320712 secs ago)
ns_orchestrator 000
ns_1@couchbase-0000.couchbase.database.svc
10:17:15 AM 4 Sep, 2023
Resetting rebalance status since rebalance stop was requested but rebalance isn’t orchestrated on our node
ns_orchestrator 000
ns_1@couchbase-0000.couchbase.database.svc
10:16:29 AM 4 Sep, 2023
Could not auto-failover node (‘ns_1@couchbase-0003.couchbase.database.svc’). There was at least another node down.
auto_failover 000
ns_1@couchbase-0000.couchbase.database.svc
10:16:18 AM 4 Sep, 2023
Could not auto-failover node (‘ns_1@couchbase-0002.couchbase.database.svc’). There was at least another node down.
auto_failover 000
ns_1@couchbase-0000.couchbase.database.svc
10:16:18 AM 4 Sep, 2023
Could not auto-failover node (‘ns_1@couchbase-0001.couchbase.database.svc’). There was at least another node down.
auto_failover 000
ns_1@couchbase-0000.couchbase.database.svc
10:16:18 AM 4 Sep, 2023
Could not auto-failover node (‘ns_1@couchbase-0000.couchbase.database.svc’). There was at least another node down.
auto_failover 000
ns_1@couchbase-0000.couchbase.database.svc
10:16:18 AM 4 Sep, 2023
Resetting rebalance status since rebalance stop was requested but rebalance isn’t orchestrated on our node (repeated 3 times, last seen 1.072448 secs ago)
ns_orchestrator 000
ns_1@couchbase-0000.couchbase.database.svc
10:16:15 AM 4 Sep, 2023
Enabled auto-failover with timeout 20 and max count 3 (repeated 1 times, last seen 16.420201 secs ago)
auto_failover 000
ns_1@couchbase-0000.couchbase.database.svc
10:16:15 AM 4 Sep, 2023
Could not auto-failover node (‘ns_1@couchbase-0003.couchbase.database.svc’). There was at least another node down.
auto_failover 000
ns_1@couchbase-0000.couchbase.database.svc
10:15:32 AM 4 Sep, 2023
Could not auto-failover node (‘ns_1@couchbase-0002.couchbase.database.svc’). There was at least another node down.
auto_failover 000
ns_1@couchbase-0000.couchbase.database.svc
10:15:32 AM 4 Sep, 2023
Could not auto-failover node (‘ns_1@couchbase-0001.couchbase.database.svc’). There was at least another node down.
auto_failover 000
ns_1@couchbase-0000.couchbase.database.svc
10:15:32 AM 4 Sep, 2023
Could not auto-failover node (‘ns_1@couchbase-0000.couchbase.database.svc’). There was at least another node down.
auto_failover 000
ns_1@couchbase-0000.couchbase.database.svc
10:15:32 AM 4 Sep, 2023
Resetting rebalance status since rebalance stop was requested but rebalance isn’t orchestrated on our node
ns_orchestrator 000
ns_1@couchbase-0000.couchbase.database.svc
10:15:28 AM 4 Sep, 2023
Resetting rebalance status since rebalance stop was requested but rebalance isn’t orchestrated on our node (repeated 3 times, last seen 1.864152 secs ago)
ns_orchestrator 000
ns_1@couchbase-0000.couchbase.database.svc
10:15:15 AM 4 Sep, 2023
Enabled auto-failover with timeout 20 and max count 3
auto_failover 000
ns_1@couchbase-0000.couchbase.database.svc
10:15:13 AM 4 Sep, 2023