We not able to reproduce the issues while unit testing but these crashes are getting logged on Firebase Crashllytics.
Crash Item : _hidden#11233 line 1
0 libobjc.A.dylib 0x1528 objc_msgSend + 8
1 CouchbaseLiteSwift 0x1acca8 void std::__1::__invoke_void_return_wrapper::__call<std::__1::function<void (litecore::blip::MessageProgress)> litecore::actor::Actor::_asynchronizelitecore::blip::MessageProgress(std::__1::function<void (litecore::blip::MessageProgress)>)::‘lambda’(litecore::blip::MessageProgress)&, litecore::blip::MessageProgress>(litecore::blip::MessageProgress&&) + 1 (_hidden#11233:1)
2 CouchbaseLiteSwift 0x1ac974 void std::__1::__invoke_void_return_wrapper::__call<std::__1::function<void (litecore::blip::MessageProgress)>&, litecore::blip::MessageProgress const&>(std::__1::function<void (litecore::blip::MessageProgress)>&, litecore::blip::MessageProgress const&) + 96 (_hidden#4022:96)
3 CouchbaseLiteSwift 0x1ab9c4 litecore::blip::Message::sendProgress(litecore::blip::MessageProgress::State, unsigned long long, unsigned long long, litecore::blip::MessageIn*) + 49 (_hidden#5154:49)
4 CouchbaseLiteSwift 0x1aa410 litecore::blip::BLIPIO::writeToWebSocket() + 364 (_hidden#9094:364)
5 CouchbaseLiteSwift 0x1aa1a8 litecore::blip::BLIPIO::requeue(litecore::blip::MessageOut*, bool) + 305 (_hidden#9094:305)
6 CouchbaseLiteSwift 0x1a9d90 litecore::blip::BLIPIO::_queueMessage(fleece::Retainedlitecore::blip::MessageOut) + 278 (_hidden#9094:278)
7 CouchbaseLiteSwift 0x1a9efc hidden#9063 + 87 (_hidden#4022:87)
8 CouchbaseLiteSwift 0x161654 litecore::actor::GCDMailbox::safelyCall(void () block_pointer) const + 91 (_hidden#5692:91)
9 CouchbaseLiteSwift 0x161610 hidden#5680 + 102 (_hidden#5692:102)
10 libdispatch.dylib 0x2924 _dispatch_call_block_and_release + 32
11 libdispatch.dylib 0x4670 _dispatch_client_callout + 20
12 libdispatch.dylib 0xbdf4 _dispatch_lane_serial_drain + 672
13 libdispatch.dylib 0xc968 _dispatch_lane_invoke + 392
14 libdispatch.dylib 0x171b8 _dispatch_workloop_worker_thread + 656
15 libsystem_pthread.dylib 0x10f4 _pthread_wqthread + 288
16 libsystem_pthread.dylib 0xe94 start_wqthread + 8
Crash Events : 41 crash events affecting 40 users
Device For Max count : iPhone 12 Pro Max
OS for Max count : 15.3.1
@vargesh so the writeToWebSocket would make me believe this is replication. What kind of replication do you have configured (Remote or PtP)?
So logs would be greatly helpful in figuring out what’s going on. Not sure if you can enable them or not but this is how to turn them on in our docs:
Are you registering any Change Listeners such as Live Query, Document Change, etc? Any more details can help pinpoint this.
Thanks
Aaron
Hi @biozal ,
This is a remote replication and you are right, we are listening to document changes that happening in the database and few live queries are also configured.
Unfortunately we are unable to provide you the logs as the crash is happening on the Production builds and we have disabled the logs for same.
As @vargesh mentioned, we are unable to reproduce the same locally.
Adding code snippet below for reference. Additionally do you thing upgrading to latest version 3.0.1 will help?
private func startPushPullReplicationForUser(profileDetail: ProfileDetails) {
guard let remoteUrl = URL(string: remoteSyncURL) else {
return
}
guard let userName = profileDetail.syncGatewayId, let password = profileDetail.syncGatewayPassword else {
return
}
Utility.printDebug(msg: "Starting database connection for \(userName)")
let dbUrl = remoteUrl.appendingPathComponent(dataBaseName)
let config = ReplicatorConfiguration(database: dataBase!, target: URLEndpoint(url: dbUrl))
config.replicatorType = .pushAndPull
config.conflictResolver = MergeConflictResolver()
config.continuous = true
config.authenticator = BasicAuthenticator(username: userName, password: password)
pushPullReplicator = Replicator(config: config)
pushPullReplicatorListener = pushPullReplicator?.addChangeListener { [weak self] change in
let status = change.status
switch status.activity {
case .busy:
Utility.printDebug(msg: "Busy transferring data")
case .connecting:
Utility.printDebug(msg: "Connecting to Sync Gateway")
case .idle:
Utility.printDebug(msg: "Replicator in Idle state")
case .offline:
Utility.printDebug(msg: "Replicator in offline state")
case .stopped:
self?.deregisterForDatabaseChanges(deleteDatabase: Channel.shared.getLoggedInUser() == nil)
Utility.printDebug(msg: "Replicator Stopped")
@unknown default:
Utility.printDebug(msg: "Unknown state")
}
if status.progress.completed == status.progress.total {
Utility.printDebug(msg: "All documents synced")
} else {
Utility.printDebug(msg: "Documents \(status.progress.total - status.progress.completed) still pending sync")
}
}
pushPullReplicator?.start()
executeUnreadDocumentLiveQuery()
}
private func registerForDatabaseChanges() {
dbChangeListenerToken = dataBase?.addChangeListener { [weak self] change in
guard let weakSelf = self else {
return
}
for docID in change.documentIDs {
if let dataBase = weakSelf.dataBase, dataBase.document(withID: docID) != nil {
weakSelf.postDocumentUpdatedNotification(documentID: docID)
Utility.printDebug(msg: "Document with \(docID) was added/updated")
} else {
weakSelf.postDocumentDeletionNotification(documentID: docID)
Utility.printDebug(msg: "Document with \(docID) was deleted")
}
}
}
}
Hello,
So normally when the application goes to the background I would stop the replicator. Then when the app goes to the foreground I would start the replicator again. Since the replicator is not running, the document change or live query events that might be tied to UI components that would run on the main thread that you no longer have access to them the background won’t fire.
In the sceneDidEnterBackground I would stop replication and then start it again when you enter the foreground using the recommended lifecycle events.
https://developer.apple.com/documentation/uikit/app_and_environment/scenes/preparing_your_ui_to_run_in_the_background
My guess the reason why you can’t replicate it is replication is firing at the exact time the app goes to background and those events that are trying to possibly update the UI are triggering but your app has already started background state processing. iOS can be extremely tricky on the timing between entering background and foreground.
If you have Background fetch enabled, then you will have to stop the document change and live query events when in the background if they are tied to UI components since in the background you don’t have access to the UI components, and trying to update them will segfault the app because they have to be updated on the mainUIThread.
My guess is you aren’t seeing this crash very often because of the timing of it if this is in fact the issue.
-Aaron
Thanks @biozal for quick reply.
We have taken care of Application lifecycle and doing same when application is going in background and becoming active. PFB related code for same.
Also as I understand if crash was happening due to UI updation on main thread shouldn’t the main thread be mentioned somewhere in the crash logs like it happens.
Crashed: com.apple.main-thread
override private init() {
super.init()
NotificationCenter.default.addObserver(self, selector: #selector(self.applicationDidEnterBackground), name: UIApplication.didEnterBackgroundNotification, object: nil)
NotificationCenter.default.addObserver(self, selector: #selector(self.applicationDidBecomeActive), name: UIApplication.didBecomeActiveNotification, object: nil)
}
@objc private func applicationDidEnterBackground() {
self.closeDatabaseForCurrentUser()
}
@objc private func applicationDidBecomeActive() {
if pushPullReplicator == nil, let profileDetails = LocalUserDefaultStorage.getObject(key: LocalStorageKeys.ProfileDetails) as? ProfileDetails {
self.startPushPullReplicationForUser(profileDetail: profileDetails)
}
}
func closeDatabaseForCurrentUser() {
if let replicator = pushPullReplicator {
replicator.stop()
}
}
I think I was confused and thought this was happening when the application goes to the background, but you are saying this isn’t the case. If it was this would be something extremely weird because you are saying you have the code to stop the replicator when entering the background, so it shouldn’t be running in the background regardless and thus this crash log should never happen when the application is in the background.
At this point, I’m going to ping someone from the SDK team to see if they can look at the crash and see what they think. 3.01 has some bug fixes so it’s always a good idea to get bug fixes in - so if you can upgrade that’s always recommended. As to will it fix your problem, I can’t answer that just yet.
Let me talk to some extremely smart folks on our SDK team.
Thanks
Aaron
-Aaron
Thanks @biozal . Meanwhile we will try to upgrade the SDK and see if that resolves this issue or not. @vargesh Please proceed with implementing 3.0.1 SDK in our builds.
Crash log includes obfuscated info, could you try convert them?
https://firebase.google.com/docs/crashlytics/get-deobfuscated-reports?platform=ios
@vargesh @Suraj_Kumar
Also, are we making sure the replicator is completely stopped before starting it again. I mean, user can switch between foreground/background very quickly.
yes @jayahari.vavachan once replicator is stopped, we are removing all the changeListeners also and making our replicator object nil. We only start the replicator again if the previous object was null.
Regarding Obfuscation information, let us check and get back to you how we can provide the same details to you.
once the replicator is stopped
Curious, while the crash happens, is it possible to make sure replicator is successfully stopped or not?
Also when you are testing it locally, have you seen the replicator is getting stopped every time you switch background/foreground(while switching really fast)?
Trying to see, whether replicator is stopped, but some underlying instance is lying around which causes this issue.
Is there any other place you are deallocating the replicator instance, and deregistering the change listener other than the stopped replicator.status == stopped?
Normally there is limited application background time allowed, which should be enough to get it stopped. Do you know how much data is normally synced or how much time it takes to finish the sync?
Any other valuable information for the crashed users, like low network coverage/cellular connection? Time period between the app background, foreground and crash?
Also its recommend to always use the latest patch release, in this case 2.8.4.
I can see a fix for a similar crash when closing the connection while replicator is connecting. It can happen, when application tries to go background(replicator stops) while replicator tries to connect(may be it was just starting up). It was reported in Android/Java, and the fix is available in 2.8.4