Running cbbackupmgr restore and getting this error - “Error restoring cluster: EOF”. Its CB enterprise server 6.5 in a docker container on Apple M2 Max processor.
Runs fine on an Intel-based laptop (same docker image, same scripts, same data).
Do any of these help you? Loading...
Can you share the command you are using and the complete error?
Is the data from a Mac or from the Intel-based laptop?
If you are using enable-bucket-config - don’t.
Failing is Macbook with Apple silicon. Apple intel works fine.
Both are using docker, same image. CB container on apple silicon machine uses rosetta emulation.
Command is:
/opt/couchbase/bin/cbbackupmgr restore \
-a "$(pwd)/opt/backup/$filename/" \
-c couchbase://127.0.0.1 \
-r cluster \
-u $CB_USER \
-p $CB_PASSWORD
Before that we do:
couchbase-cli cluster_init
couchbase_cli setting-compaction
couchbase_cli bucket-create
and all 3 of those cmds complete successfully.
Not sure what that error is telling me. I see all of my backup data, seems reasonable. Permissions are ok. Sounds like its reading something and prematurely hitting end of file, but what?
Also I notice these in the logs, looks like every 6 seconds or so:
Service 'indexer' exited with status 2. Restarting. Messages:
goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:262 +0x91 fp=0xc4239ab4d8 sp=0xc4239ab478
github.com/couchbase/nitro/skiplist.(*Skiplist).Insert(0xc4203e2680, 0xc423bae000, 0x12012d8, 0xc4239abaf8, 0xc4203e2698, 0x21)
goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:254 +0x68 fp=0xc4239ab530 sp=0xc4239ab4d8
github.com/couchbase/plasma.New(0xc8, 0x190, 0x19, 0x4, 0x1201320, 0x12012e0, 0x1201330, 0xc42398eca0, 0xc42398ecb0, 0x1201360, ...)
goproj/src/github.com/couchbase/plasma/plasma.go:637 +0x71c fp=0xc4239abd30 sp=0xc4239ab530
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0xc423b22600, 0xc423b2c000, 0xc423b22610, 0xc423a95000)
goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:334 +0x91 fp=0xc4239abf80 sp=0xc4239abd30
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4239abf88 sp=0xc4239abf80
created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores
goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:339 +0x2b69more from indexer.log. its crashing:
2023-05-26T19:17:21.134+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -nodeUUID=59dd5f5bc4c81eb522cd935a230a75d0 -ipv6=false -isEnterprise=true]
2023-05-26T19:17:21.158+00:00 [Info] Indexer::NewIndexer Status Warmup
2023-05-26T19:17:21.165+00:00 [Info] Setting buffer block size to 16384 bytes
2023-05-26T19:17:21.165+00:00 [Info] Setting maxcpus = 6
2023-05-26T19:17:21.165+00:00 [Info] Setting log level to Info
2023-05-26T19:17:21.169+00:00 [Info] SettingsMgr:: Dropped request /indexing/settings/config {"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.log_level":"info","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":268435456,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready.
2023-05-26T19:17:21.170+00:00 [Info] Recieve security change notification. code 3
2023-05-26T19:17:21.170+00:00 [Info] TLS config refreshed successfully
2023-05-26T19:17:21.179+00:00 [Info] Certificate refreshed successfully
2023-05-26T19:17:21.179+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false
2023-05-26T19:17:21.179+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.18.0.2:true]
2023-05-26T19:17:21.209+00:00 [Info] security context initialized
2023-05-26T19:17:21.209+00:00 [Info] security port mapping updated : map[9102:19102 9999:9999 11210:11207 9100:9100 9101:9101 9103:9103 9104:9104 9105:9105 8093:18093]
2023-05-26T19:17:21.210+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise
2023-05-26T19:17:21.210+00:00 [Info] Indexer::Cluster Storage Mode Set plasma
2023-05-26T19:17:21.210+00:00 [Info] memcachedTimeout set to 120
2023-05-26T19:17:21.211+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024
2023-05-26T19:17:21.213+00:00 [Info] OperationsMonitor(kvSender) starting monitor ...
2023-05-26T19:17:21.218+00:00 [Info] [Queryport ":9101"] started ...
2023-05-26T19:17:21.223+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap
2023-05-26T19:17:21.230+00:00 [Info] bootstrap storage mode plasma
2023-05-26T19:17:21.256+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 0 min file size 0
2023-05-26T19:17:21.257+00:00 [Info] servicesChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default
2023-05-26T19:17:21.261+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification
2023-05-26T19:17:21.262+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification
2023-05-26T19:17:21.265+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 1061 us
2023-05-26T19:17:21.272+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore
2023-05-26T19:17:21.819+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server
2023-05-26T19:17:21.854+00:00 [Info] indexer:: Staring http server : :9102
2023-05-26T19:17:21.854+00:00 [Info] HTTPS server created for :19102
2023-05-26T19:17:21.855+00:00 [Info] indexer:: SSL server started: :19102
2023-05-26T19:17:21.855+00:00 [Info] Indexer::indexer version 4
2023-05-26T19:17:21.855+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId
2023-05-26T19:17:21.856+00:00 [Info] Indexer Id 59dd5f5bc4c81eb522cd935a230a75d0
2023-05-26T19:17:21.867+00:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_59dd5f5bc4c81eb522cd935a230a75d0
2023-05-26T19:17:21.872+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_59dd5f5bc4c81eb522cd935a230a75d0. Err projector.topicMissing
2023-05-26T19:17:21.872+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
2023-05-26T19:17:21.872+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success
2023-05-26T19:17:21.881+00:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic INIT_STREAM_TOPIC_59dd5f5bc4c81eb522cd935a230a75d0
2023-05-26T19:17:21.882+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic INIT_STREAM_TOPIC_59dd5f5bc4c81eb522cd935a230a75d0. Err projector.topicMissing
2023-05-26T19:17:21.882+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
2023-05-26T19:17:21.882+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success
2023-05-26T19:17:21.882+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning
2023-05-26T19:17:21.882+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken
2023-05-26T19:17:21.882+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken <nil>
2023-05-26T19:17:21.882+00:00 [Info] ClustMgr:handleGetGlobalTopology &{map[]}
2023-05-26T19:17:21.885+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap
InstanceId: 13325764127247504452 Name: doc_tab_uuid_idx Bucket: union State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0
2023-05-26T19:17:21.896+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set
2023-05-26T19:17:21.896+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 59dd5f5bc4c81eb522cd935a230a75d0
2023-05-26T19:17:21.897+00:00 [Info] RebalanceMgr::NewRebalanceMgr false <nil>
2023-05-26T19:17:21.899+00:00 [Info] RebalanceMgr::initService Init
2023-05-26T19:17:21.899+00:00 [Info] Indexer::local storage mode plasma
2023-05-26T19:17:21.899+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 59dd5f5bc4c81eb522cd935a230a75d0
2023-05-26T19:17:21.904+00:00 [Info] ServiceMgr::GetTaskList []
2023-05-26T19:17:21.906+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 0] []}
2023-05-26T19:17:21.908+00:00 [Info] Indexer::initPartnInstance Initialized Partition:
Index: 13325764127247504452 Partition: PartitionId: 0 Endpoints: [:9105]
2023-05-26T19:17:21.909+00:00 [Info] ServiceMgr::GetCurrentTopology []
2023-05-26T19:17:21.909+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 0] [59dd5f5bc4c81eb522cd935a230a75d0] true []}
2023-05-26T19:17:21.911+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 0]
2023-05-26T19:17:21.911+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 0]
unexpected fault address 0xc42041225f
unexpected fault address 0xc42041225f
fatal error: fault
fatal error: fault
[signal SIGBUS: bus error code=0x1 addr=0xc42041225f pc=0x71248f]
Ticket opened. Thanks!
1 Like
Hi Steve, I was not able to reproduce your issue. Please could you upload the backup logs? You can create them by using “cbbackupmgr collect-logs -a <path-to-archive>”.
Thanks
Safian
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.