Ratelimit: No synchronization on 2020-08-01?

[r-r-liu]

I noticed around 14:40 today, Aug. 1, that ver. 1.07 had not synchronized since about 19:01 yesterday evening. A few minutes later, I discovered that 1.08 was available and updated. However, even after restarting the iPhone the situation has not changed. I understand that the app tries to synchronize at random times after 06:00 and after 18:00, so I might have to wait until after 18:00. But why didn’t 1.07 synchronize this morning?

By the way, there really ought to be a way to report bugs and problems to the BAG, who, after all, are the purveyors of the app, but its Swiss Covid App site seems not to have found its way into the second decade of the 21st Century. I apologize for reporting such things here.

[r-r-liu]

On Sunday Aug. 2 at 07:22 still no new sync since July 31 at 19:01. I'm worried.

[stmitt]

@r-r-liu Thank you for your feedback. This might be related to #163.
To Analyse your issue further we need a sysdiagnose of your phone. If you are willing to provide one can you please contact me by mail via [email protected]

[r-r-liu]

Just an update for the sake of completeness; I've already informed Stefan by email.

Yesterday, Aug. 2, SwissCovid 1.0.8 running on iOS 13.6 synchronized twice. This morning, Aug. 3, at 9:39, according to the message in the app, it had not yet synced successfully; however, in Settings > Privacy > Bluetooth > COVID-19 Exposure Logging > Exposure Checks two exposure checks from today at 6:04 appeared. Later at 9:54, the app apparently pulled 6 exposures from the backend and the message in the app has been changed accordingly to show the last successful sync to have taken place then. So the message is somewhat irritating. If the app knows as soon as it connects to the backend how many exposures it is supposed to download, then does not download all of them, a message from the app should read something like this:

Incomplete synchronisation. x of y expected exposures received. Will attempt to get y-x later.
dd.mm.yyyy hh:mm:ss

and should appear above the message about the last successful sync.

[r-r-liu]

Today this appeared in the Corona Ticker of the Basler Zeitung (https://www.bazonline.ch/bag-meldet-am-freitag-220-neuinfektionen-mit-dem-coronavirus-707221502952):

"Die automatisierte Aktualisierung der Gesamtzahl aller Swiss Covid-Apps sei über das Wochenende ausgesetzt worden, hiess es am Montag auf Anfrage beim BFS. Man habe damit «etwaige technische Fehler respektive deren Auswirkungen auf die publizierten Zahlen ausschliessen» wollen. Ab wann die Aktualisierung übers Wochenende wieder aufgenommen werde, könne derzeit noch nicht gesagt werden. Unter der Woche werde die Seite weiterhin täglich aktualisiert."

Even assuming, as would be par for this newspaper, that a detail or two isn't quite correct and others have been omitted, does anybody know what's behind this? If, as the blurb seems to imply, that no syncing on the weekend (or, maybe, only on holidays on weekends?) will become a new "feature", this is certainly disconcerting. What technical errors might "man" have been fearing. Due to the extreme privacy protection design of the app and its associated backend systems, wouldn't any such ultimately have to do with a rise in the number of users making use of their covid codes and uploading their encounters? And isn't that the whole point of the app?

Can somebody shed some light on this situation?

[r-r-liu]

I am finding this new (since 1.0.8) message about last successful sync infuriatingly ambiguous. It seems, once again, that the app received 7 exposure codes this morning around 6:00, and yet the message still says that the last successful sync took place yesterday at around 18:00. If things follow the present pattern, latter this morning it will get 2 more and then the message will change. What is the difference between what is evidently not a successful sync, but which nevertheless downloads something, and a successful one? Can someone please review these messages at make sure they reflect what the app actually is doing, not what developers think the public can handle? Thanks.

[stmitt]

The last successful sync timestamp shows the last time the app was able to successfully check against exposures for the last 10 days.
It seems like the app was not able to check all 10 days on your phone around 6:00 maybe because we did not get enough execution time from the operating system. Therefore on a second run, the app could complete the previous checks and update the last successful sync timestamp.
To verify this theory I will follow up on how you can send us a sysdiagnose by mail.

[r-r-liu]

Last ten days, i.e. ten records, on per day. I’ve been getting only nine recently. Will send sysdiadnose ASAP. Thanks.

…

______________________ Richard R. Liu [email protected]

Sent from my iPhone

On 5 Aug 2020, at 10:05, Stefan Mitterrutzner ***@***.***> wrote:  The last successful sync timestamp shows the last time the app was able to successfully check against exposures for the last 10 days. It seems like the app was not able to check all 10 days on your phone around 6:00 maybe because we did not get enough execution time from the operating system. Therefore on a second run, the app could complete the previous checks and update the last successful sync timestamp. To verify this theory I will follow up on how you can send us a sysdiagnose by mail. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[stmitt]

Unfortunately, the calls for detecting exposure to the ExposureNotification framework of iOS are rate limited. We can only make a certain amount of calls to the API before iOS blocks us, however this window is not clearly defined by Apple (calendar day, 24 rolling windows, 24h based on the first call, ...). You can read more about it here.
In your case, the app was running into the rate limit at 06:00 and could only continue the remaining calls to the framework later.
We only show a successful synchronization in the app if all calls have been successful.
Thank you for reporting this issue to us, we will look further into this.

[r-r-liu]

Does this also explain why there wasn't any synchronization at all on Aug. 1. There is no exposure list for that day. Also, should I be receiving just nine exposure checks per successful synchronization? Thanks.

By the way, could the fact that there was no successful synchronization at all on Aug. 1 have to do with my updating the app to 1.0.8 that same day after noting that there had been neither a successful sync nor any downloaded exposures by 14:00?

[r-r-liu]

Yesterday, 2020-08-15, I had two successful syncs in "quick" succession, one at 16:33 (8 records at 16:32, last at 16:33) and the other at 18:33 (9 records). Two questions to this:

1. What does each record represent, exposures for one of the last ten days for which the app checks? In that case shouldn't I be seeing ten records per successful sync, not nine?

2. I was away from home yesterday from around 05:00 to 16:15 and probably had no WIFI during that whole period. I have configured Mobile Data ON, Mobile Data Options ROAMING ON. Does exposure checking only work with WIFI? This is probably not a well-thought out restriction. If I am participating in a multi-day SAC tour and have at best 4G reception, then my exposure risk won't be calculated the whole time, and by the time a sync is made I might have infected many other guests and personnel of the huts that we visited and companions in the group.

Today, 2020-08-16 (Sunday), I see only 2 records from an incomplete sync at 07:50. It's now 13:48. We'll see whether this corrects itself in the course of today.

The iPhone 6S is running iOS 13.6.1 and SwissCovid 1.0.8.

[jonasschuetz]

Hej I have noticed that my app says that the last successful synchronization was on the 19.08.2020. After that no logs have been downloaded either (according to the view in the settings app).
But everything looks just fine in the app. Is there a way to force the update?

I have noticed, that I wasn't connected to wifi a lot in the last days. Does this prevent the background tasks from being triggered?

[r-r-liu]

@jonasschuetz,

Stefan Mitterrutzner, @stmitt, is the only developer on this thread. As you've seen from my second-to-last post, I, too, have doubts about whether the app attempts a synchronization if it has no WiFi connection, despite the fact that I have switched on Mobile Data.

Just out of curiosity, how many records has your app fetched when it claims to have synchronized successfully? I first installed the app on the day that it officially became available, and regularly received 10 records. Now, I only receive 9.

[r-r-liu]

@jonasschuetz,

By the way, I'm assuming you're running the latest versions of the app and iOS. Are you?

[UBaggeler]

Yesterday, 2020-08-15, I had two successful syncs in "quick" succession, one at 16:33 (8 records at 16:32, last at 16:33) and the other at 18:33 (9 records). Two questions to this:

1. What does each record represent, exposures for one of the last ten days for which the app checks? In that case shouldn't I be seeing ten records per successful sync, not nine?

Yes, in the current implementation (V1) of the iOS ExposureNotification API, SwissCovid downloads TEKs grouped by day for the past 10 days (including today). As neither Apple nor Google have activated current day TEK release yet (TEK of the current day is delayed until midnight UTC), the bucket with keys for the current day is empty. This explains why you only get 9 exposure checks for a full sync. When the pilot phase started, the DP3T backend added random keys to every bucket to make sure the whole system was fully running while there were no or not enough keys published yet. That's why you saw 10 checks per day in the past. This has been deactived since.

The background task to download keys gets scheduled around every 2h by iOS. SwissCovid performs a full sync twice a day (after 6am and after 6pm). When exactly the background task is getting scheduled is determined by iOS. This can lead to situations where you get 9 successful checks at 07:23 and 9 checks at 19:45 on one day (total of 18 checks). If iOS schedules the background task a bit earlier on the following day (let's say 06:32), SwissCovid downloads keys for all 10 days but after submitting 2 days to the ExposureNotification API, detectExposures() will fail because of the rolling rate-limit of 20 calls per day. The remaining days will then be retried the next time the background task gets scheduled. If the app gets scheduled at 09:13 the next time, it can perform the 7 remaining checks as the rolling rate-limit is not at its limit anymore (after 07:23). Until then, the app will show the date of the last successful sync (of all 10 days).

This will sync mechanism will change substantially after switching to V2 of the EN-API, which will allow to group exposures by day without grouping them on the backend but by supplying diffs every 4h to the EN API (and then grouping the results).

2. I was away from home yesterday from around 05:00 to 16:15 and probably had no WIFI during that whole period. I have configured Mobile Data ON, Mobile Data Options ROAMING ON. Does exposure checking only work with WIFI? This is probably not a well-thought out restriction. If I am participating in a multi-day SAC tour and have at best 4G reception, then my exposure risk won't be calculated the whole time, and by the time a sync is made I might have infected many other guests and personnel of the huts that we visited and companions in the group.

Not having Wi-Fi should not impact scheduling of the background task. So far our testing does not show such behaviour but if you suspect this is the case for you, we would appreciate a sysdiagnose (at most 1 day after you recognized this behaviour) to be able to share it with Apple.

Today, 2020-08-16 (Sunday), I see only 2 records from an incomplete sync at 07:50. It's now 13:48. We'll see whether this corrects itself in the course of today.

The iPhone 6S is running iOS 13.6.1 and SwissCovid 1.0.8.

I hope this explains most of the questions. If I missed one, let me know.

[r-r-liu]

Yes, immensely helpful. Thank-you. I will keep an keep my eyes open for evidence that the syncing might somehow need WiFi.

[r-r-liu]

The sync behavior has gone haywire once again (iOS 13.7, app 1.0.9, sysdiagnoses will be sent to @stmitt).

2020-09-09: 9 x 06:55, 6 x 20:21, 3 x 23:04
= 2 complete syncs this day, OK

2020-09-10: 9 x 08:26
= only sync this day

2020-09-11: 9 x 01:50, 2 x 08:42
= one complete sync.
Second not successful until next morning?

2020-09-12: 9 x 00:52, 6 x 08:53
= one complete sync.
Second not successful until next morning?

2020-09-13: 9 x 00:55, 6 x 09:05
I am assuming the above pattern will repeat, i.e., no more syncs today, next successful one early 2020-09-14

This is not in agreement with the documentation, which says several times a day.

[r-r-liu]

Should have known! On 2020-09-13 +3 x 16:28 to complete the second sync of the day. Nevertheless, three consecutive days of only a single sync according to the message in the app will probably drive even devotees of the app nuts and lead them to wonder whether the thing is working properly.

[stmitt]

Thank you @r-r-liu for your feedback.
After analyzing your sysdiagnose we found out that your device is again running into the rate limit by Apple.
We forwarded this issue to apple.

[BorntraegerMarc]

I wonder if this has something to do with #344 ?

[r-r-liu]

I have since update SwissCovid to 1.2. Today I see synchronizations at 00:46, 04:50, 09:45, 14:42 so far at 15:27. Let me be clear, at each time one record seems to have been fetched, and I did see messages from the app about last successful synchronizations took place at 00:46, 09:45 and 14:42. I don't know whether such a message was generated for 04:50, since after waking up this morning I didn't look until around 12 noon. I don't know whether any more synchronizations will be attempted today.

[UBaggeler]

SwissCovid 1.2 uses the new API version of the ExposureNotification.framework (ENv2). Using ENv2 we do not need to group TEKs by day anymore and can just sync/submit all new keys (diff) which were published since the last time we checked. The new rate-limit, using ENv2, is 6 calls per 24h (every 4h one exposure check). Compared to before, where we were able to check exposures twice per day (once in the morning and once in the evening), this now means notifications should be shown sooner and are spread more throughout the day.

[r-r-liu]

Yes, I apologize, I only consulted #344 after posting the message to which you responded. Had I read #344 first, I wouldn't have posted. Thanks for the explanation.