Broadcast push being throttled with very little traffic

Hi,

We've implemented the new Live Activity Broadcast push notifications (https://developer.apple.com/documentation/usernotifications/setting-up-broadcast-push-notifications) and are testing this out in sandbox/development.

Looking at our logs I can see that we tried to send 54 broadcast notifications the last hour. 16 of these failed with "429 Too Many Requests".

We probably have no devices registered on any of the channels at the moment. These are all channels from the "Development" environment.

Any pointers to what could be wrong?

Regards,

-Tommy

Answered by tno in 795707022

For others reading this post:

The error was due to too frequent token updates. We had 4 worker thread sharing the same HttpClient sending notifications to APNs. Instead of sharing the JWT Token they all maintained their own (refreshing every 20 mins), causing the 429 response code.

We only logged the status code so it looked like we were throttled.

https://developer.apple.com/documentation/usernotifications/handling-notification-responses-from-apns#Understand-error-codes

Thanks for the support @Argun.

We can only guess as to what app you are having this issue with, but if your app Bundle ID matches com.x_____g.M_________a.__v we have seen that 45 notifications were sent within 1 minute, which is very much above the allowed limit. You may want to check your bursts, which, more often than not, are what will cause the 429 error. While the limits are not published, I can say that broadcast notifications are not meant to send real-time updates every second, but to be used in a more relaxed fashion. In any case, the 429 error is not fatal, and you can implement a pull-back algorithm to slow down your requests once you start getting the error.

If you have questions about how to best make use of broadcast notifications for your specific use case, please do not hesitate to ask.


Argun Tekant /  DTS Engineer / Core Technologies

Hi Argun,

Our Bundle ID doesn't match the above. It's: com.m__________e.f_______o

This is a live score app for football (soccer).

I checked the logs from yesterday and I can't see that we're sending notifications too frequent.

We sent a total of 10 notifications for one of the games yesterday at these times:

2024-07-10T17:01:15.3213590Z
2024-07-10T17:18:32.9094852Z
2024-07-10T17:37:39.1622861Z
2024-07-10T17:52:00.8026556Z
2024-07-10T18:05:56.6542415Z
2024-07-10T18:15:50.5259535Z
2024-07-10T18:36:11.2406366Z
2024-07-10T18:46:46.0640576Z
2024-07-10T18:52:10.7170653Z
2024-07-10T19:01:33.9505385Z

There are several minutes between each push but we still got two throttles:

2024-07-10T18:52:10.8535674Z
2024-07-10T19:01:34.0837425Z

I'm assuming the rate limit is per channel.

Even looking across all channels we sent to at the time we got a 429 error I don't see why we would get it:

2024-07-10T18:55:46.449Z - TAJdlj2GEe8AAErsQhdY4w==
2024-07-10T19:00:28.570Z - TE6ByT2GEe8AAErsQhdY4w==
2024-07-10T19:00:48.331Z - S+hs9T2GEe8AAErsQhdY4w==
2024-07-10T19:01:29.158Z - TDUtaD2GEe8AAErsQhdY4w==
2024-07-10T19:01:33.950Z - znqayz1kEe8AANJjTmHR9Q==
2024-07-10T19:04:05.323Z - TAJdlj2GEe8AAErsQhdY4w==

Yesterday was a slow day for football/soccer, so this is a bit worrying.

Also, we have a very limited set of users in the "Development" environment.

We have a lot of users in production though, so we're afraid of rolling out this feature as it stands.

We're using Live Activities extensively and sent 534,818 Live Activity Start notifications for the Euro Semi-finale match between Netherlands - England yesterday.

Btw, will the regular Live Activity sending affect the rate limit of broadcast notifications in any way?

Today each device will register itself with our server after we send a Live Activity Start. The plan is to have them register on the broadcast channel instead.

Is there any rate limiting for clients registering on a topic/channel id?

The reason I'm asking is that the traffic will be very spiky. With the example above, we'll send 500K LA Start that will be followed by all the devices registering on the channel for the given match.

Best regards,

-Tommy

Accepted Answer

Hi Tommy,

We'll need some specific information here about your app and some details. Can you please head to http://developer.apple.com/contact/technical and submit a code-level support request, and refer to this thread in the request form. I will take it over from there.

In your request please provide the following info:

  • Bundle ID
  • Channel ID used
  • Time of the incident
  • Request ids of the pushes sent with the issue

Talk to you on the other side.


Argun Tekant /  DTS Engineer / Core Technologies

For others reading this post:

The error was due to too frequent token updates. We had 4 worker thread sharing the same HttpClient sending notifications to APNs. Instead of sharing the JWT Token they all maintained their own (refreshing every 20 mins), causing the 429 response code.

We only logged the status code so it looked like we were throttled.

https://developer.apple.com/documentation/usernotifications/handling-notification-responses-from-apns#Understand-error-codes

Thanks for the support @Argun.

Broadcast push being throttled with very little traffic
 
 
Q