It’s always 99%! This screenshot was taken during the initial sync of a Capella database down to the local device. Obviously, this isn’t very useful for showing progress to the user.
Improvement
Could the SDK be improved so that Progress is useful? I understand that it may be difficult to determine the true total number of changes in some cases, but at least for the case where the database is being synced down for the very first time, there must be some way to set an accurate total number so that the progress percentage is meaningful?
I believe that the claim that that number is not useful/meaningful, simply because it is approximate, is way overboard. It will always be, only approximate because, by the time the information gets to client code, it is almost certainly out of date.
In computing, the concept of “approximate” numbers is completely common. This number should be treated in exactly that same way. In particular the right way to check for equality is with some error bounds: maybe 1 or 2 percent.
FWIW, you can tell when a replication is complete by observing the replicator state. When it is IDLE or STOPPED, it is done.
Could the SDK be improved so that Progress is useful?
Only by reducing performance of the replicator, unfortunately, so it’s not viable.
A lot of data is being streamed concurrently during replication. That includes the negotiation of which revisions are “new” to the other side. If we turned off concurrency and worked out all the revisions (and their sizes) before transferring any, we could get an accurate denominator for the progress fraction. But it would waste time and probably increase memory usage and require extra queries.
(The same problem comes up in GUIs for copying files. If you use a Mac, you have probably noticed that when you copy a large folder the Finder will sit there for a while calculating sizes before it starts copying anything. That’s in part to give you an accurate progress indication.)
During the initial download/sync of a 2GB database, I would like to show my users a progress bar that doesn’t just sit at 99% for 5 minutes. Showing them that makes my app (and me) look dumb.
Once the database’s initial download/sync is complete and we’re then handling little changes rolling in, I’m much less concerned with progress reporting—the sync happens so fast, I don’t really need to show anything at all.
Perhaps we could explore improving Couchbase so that if the Replicator realizes, “Oh, there’s NOTHING in this local database, I’ve gotta sync the whole thing and that’s going to take a couple minutes, so let me spend a few seconds figuring out the approximate total size of the data that has to be transferred so that I can guess at how long this is going to take and present some meaningful progress report,” that would be fantastic.
I do that. In practice, the replicator’s progress will hit 1.0 and then it will sometimes sit for several minutes before finally flipping to .idle. Other times it will flip to .idle immediately after progress hits 1.0. There appears to be no deterministic relationship between progress and state.
Rephrased Question
With your expertise, perhaps you could show me how YOU would use the progress value from your Replicator to present a meaningful progress indicator to the user during long-running sync operations.
Yeah… all of that sound pretty much as designed.
I do have suggestions, depending on what it is you are trying to convey to your user.
Perhaps the simplest is that, when the percent progress gets up above 97%, you could take down the progress bar and call the download complete.
I don’t think it is possible for the Replicator to figure out how much data it is going to get, especially when that depends on the other side of the connection doing the calculation and forwarding it.
When the replicator completes syncing all of the things that it knows, at the start of replication, need to be synced, it has to check again, to see if anything has changed during the sync operation. It also delivers the IDLE/STOPPED notification asynchronously. All of this means that the notification may be delivered a while after the progress gets near 100% Until your code gets that notification, however, it really has to assume that the replication is not complete.
Again, though, the assertion that there is “no deterministic relationship between progress and state” seems extravagant. They are closely related.
I think that if I understood what you are trying to convey to your users, I might have better suggestion. If the problem is only displaying an approximate number to the customer, I would suggest you display max(100, (progress <= 0) ? 0 : progress + 2)
As shown in the screenshot, the progress is never below 99%. From the very first time it’s posted to the very last time (minutes later), the value is always 99%. If that’s intentional, I’m not sure why this API even exists at all.
It is ALWAYS0.999xxxx. I have never seen a progress percentage lower than that.
My guess was that the denominator (the total) is constantly updated as sync progresses and remains very close to “completed”, which yields a consistent 99%. This is why the API seems broken to me.
You definitely are not wrong about that. Here is a breakdown of what actually happens and why I never recommend using that. The very concept of progress when replicating in the pull direction is a nebulous one. In continuous it doesn’t really have a good definition since by definition it is never actually finished.
The biggest problem in calculating this is that this information is not calculated up front. If Sync Gateway had to calculate all changes for every connected client taking into account the filtering settings and everything it would slow down significantly. Not to mention that it would be immediately invalidated if more changes came in after that. Instead the changes are calculated on demand and streamed to the client. You can think of it as TCP stream in which you are simply receiving data and it will be “done” when it is closed.
So the long and short of it is that “total” here is pretty much a myth. It’s only going to be correct when it is ACTUALLY at 99%. So a progress bar is pretty much impossible to make unless you resort to some shenanigans like calculating how far it is between 99 and 100% and converting that 0.0 to 1.0 range to a percentage. The “completed” part is accurate though, so while not ideal it might be better than nothing to show a running total of completed changes.
