Apple Push Notification service server certificate update

Question: Do I need to make any code changes on my server for this update?
Answer: No code changes are needed, and the use of the new certificate should be automatic once it is installed. But do keep in mind that on March 31st, the legacy binary protocol will no longer function, and if you are not using the new HTTP/2 protocol already, you will need code changes to support that change independent of the certificate update.

You can read about this protocol change here: https://developer.apple.com/news/?id=c88acm2b

Since yesterday my push notifications doesnt work anymore. without changing anythine.
I am using Parse Server with Push Adapter and p8 certificate.
It suddenly stopped working without change on my side.
its using node and https/2 to establish the connection.

Getting this error when sending:
parse-server-push-adapter APNS VError: stream ended unexpectedly

Thank you.

Question: I don't use HTTP/2. Do I still need to update the certificate on my server?
Answer: If you are not using HTTP/2 by March 29th, you will have bigger problems 2 days after on March 31st. The legacy binary protocol will stop working on March 31st, and you will no longer be able to send push notifications using the old protocol. So, you will need to start working on moving to the HTTP/2 protocol ASAP.

I also face the similar issue mentioned by Skyborg, push notifications were unable to be sent since yesterday.

I am using Node.js and received error "UNABLE_TO_GET_ISSUER_CERT_LOCALLY" every-time when I attempt to send notifications to both development and production environments.

Thank you.

Update : my issue fixed by updating to latest Node.js (v14.15.5)

Question: Is it sufficient to add the new root certificate AAACertificateServices 5/12/2020 or do I have to add the additional
certificates (USERTrustRSAAAACA 5/12/2020; COMODORSAAAACA 5/12/2020) which I found at https:// support.sectigo.com/Com_KnowledgeDetailPage?Id=kA03l00000117cL
as well?

Question: If we are using the non-expiring Apple Push Notification service (APNs) key, do we need to create a new one to put on our server?

Question: Just want to make sure I understand this message clearly,

Note that Apple Push Notification service SSL provider certificates issued to you by Apple DO NOT need to be updated at this time.

Does this mean that app developers do NOT need to re-create and re-upload these certificates using the latest Worldwide Developer Relations Certificate Authority (Expiring 02/20/2030), in order to continue serving push notifications from third party services? And that it is up to the third party service to update the certificate in their Trust Store? Example: Mixpanel push notifications

Hello!
We use curl on a Windows box to send low volume APN payloads to Apple's server. We DON'T have the GeoTrust Global CA or the new AAACertificateServices in that Windows box's trust store. Can we just add the AAACertificateServices to the curl option --cacerts CACertsFile?

I'm not clear why the AAACertificateServices is necessary. The api.push.apple.com SSL certificate is authorized by the Apple Root CA, not GeoTrust Global CA or AAACertificateServices Root CA. So why is it necessary to add it?

The curl command line we're using consists of the following options:

curl -x http://somePROXY -U X:X -v -d "{\"aps\":{\"alert\":\"Success!\",\"sound\":\"default\"}}" -k --cert <path to cert PEM>  --key <path to cert key> --pass "XXXX" --cacert <path to Apple Root CA> -H "apns-topic:
<APNS topic>" --http2 https://api.push.apple.com/XX/device/YYYYYYYYYYY

With the -v curl option we get the following output:
<HTTP/1.1 200 Connected
<

• Proxy replied OK to CONNECT request

• ALPN, offering h2

• ALPN, offering http/1.1

• Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH

• successfully set certificate verify locations:

•   CAfile: <path to CA.pem file>

CApath: none

• SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

• ALPN, server accepted to use h2

• Server certificate:

•  subject: CN=api.push.apple.com; OU=management:idms.group.533599; O=Apple Inc.; ST=California; C=US

• start date: Mar 14 17:50:10 2019 GMT

•  expire date: Apr 12 17:50:10 2021 GMT

•  issuer: CN=Apple IST CA 2 - G1; OU=Certification Authority; O=Apple Inc.; C=US

•  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

• Using HTTP2, server supports multi-use

• Connection state changed (HTTP/2 confirmed)

• Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

• Using Stream ID: 1 (easy handle 0x1eeb1e8)

POST /x/device/YYYYYYYYYYYYYYYYYYYY HTTP/1.1
Host: api.push.apple.com
User-Agent: <Our user Agent>
Accept: */*
apns-id: <AN APNS ID>
apns-topic: <OUR APNS Topic>
Content-Length: 208
Content-Type: application/x-www-form-urlencoded

• Connection state changed (MAX_CONCURRENT_STREAMS updated)!

• We are completely uploaded and fine

< HTTP/2 200

< apns-id: <some APNS ID>



Thanks

Question: I am currently using FireBase CloudMessaging(FCM) for ios notification, I just want to confirm if this update is something third party provider has to update (eg. update the certificate on their side) since my backend server does not send directly to the APNs? Or if this is something I need to change, FCM hasn't updated anything on their website regarding to whether they will be impacted by this issue.

@KJZZZ,
I would imagine that it is Firebase's responsibility to update their back end appropriately. Your server connects only to FireBase, so it's out of your domain. Have you considered getting in touch with FireBase directly?

Will you make connections to the Sandbox Server fail prior to the production server transition date for connections that do not include the new certificate? If yes, can you please publish the transition date for the Sandbox server?

Thank you!

I don't see anything concerning the laravel implementation. We have an API that sends the APNs and we are using the Laravel notification channels and the APNMessage class.
The config for the implementation at this moment looks like this:

'apn' => [
'environment' =>1,
'key_id' => env('APN_KEY_ID'),
'team_id' => env('APN_TEAM_ID'),
'app_bundle_id' => env('APN_APP_BUNDLE_ID'),
'private_key_path' => env('APN_PRIVATE_KEY', base_path('APN_cert.p8')),

where the private key is the old .p8 file.
Any hint to where the new AAACertificateServices.crt fits in?

Thanks!

@skyborg We had similar issues to what you described, with Parse Server notifications suddenly starting to fail last week. We updated both Parse Server, and Node, and everything seems to be working now. Still unclear if this was related to this new certificate requirement. Our versions now are:
Node v14.15.5
@parse/node-apn 4.0.0
parse-server 4.5.0

I'm using this package to send push notification with a .p8 file. As far as I know, I not using any certificates till now. So do I have to install the new AAA certificate in order to send push notifications?

"verify that both the old and new root certificates for the HTTP/2 interface are included in the Trust Store of each of your notification servers before March 29"

How can i verify that the required certificates (AAACertificateServices) are included in the Trust store of my backend server?

My backend server's OS is Ubuntu 18.04.3 LTS.

Question: I use Axway platform do send push notifications. The way to do it is to convert my Apple Push Notification certificate in p12 format on my Mac and then load it on the Axway online dashboard.
This morning I created a new Apple Push Notification certificate (the old one expires in a few days), I downloaded and installed it in my keychain, then I downloaded and installed the "AAA certificate services" and then I created the new P12 certificate for the Axway dashboard. Is it enough? How can I check if my new p12 certificate has or not the root "AAA certificate services"? Thank you in advance

It seems that I have the same situation as @ds_wrg: we are using npm package to connect to APN on our server-side with a .p8 file and not using any certificates.

The package we use:
https://github.com/node-apn/node-apn#readme

It is “Based on HTTP/2 based provider API”.
And we use, if I understand correctly, token-based authentication way. I.e to create a new connection we use such such kind of data only:

Code Block 
token: {
  key: "path/to/APNsAuthKey_XXXXXXXXXX.p8",
  keyId: "key-id",
  teamId: "developer-team-id"
},

So the question:
1) do these “root certificates” changes affect us in any way?
2) And if so, what and where should be changed for us ?"

Somewhat similar situation as @Nikitkirik here. Although we don't use npm, we use curl to send APN payloads to Apple servers, and it is absolutely not clear why this is necessary.

We are not using the GeoTrust root CA for anything, and can't locate it on our servers. Our push notifications have been working for about 2 years and we are using HTTP/2. Can someone from Apple tell me what the GeoTrust Root CA is being used for right now w.r.t APNS? Our SSL cert and key are signed with the Apple Root CA, and so are the Apple push servers (that certificate expires in April 2021 btw).

This thread is not very helpful. I had to file a TSI and hopefully someone from Apple will get back to me.

Hi @awaisfayyaz17.apple,
we have a similar situation. We looked for the GeoTrust Root CA in out server's trust store and it's not there, and it's also not being picked up by curl. Our server is a Windows 10 box (don't ask, we have very low volume push notifications and it was not my decision) so it was easy to find the trust store and list all of the root CA's. No GeoTrust Root Ca in it.

For Ubuntu 18.xx the trust store is accessible in /etc/ssl/certs/ or /etc/ssl/certs/ca-certificates.crt for the CA certs. What will you do if the GeoTrust Root CA is not in there?

For us we specify --cacerts file for the curl command, but it contains the Apple Root CA, which I believe is not derived from the GeoTrust root CA.

I have the same question others have asked - we are using the JWT token approach to authenticate when sending notifications. During that process we use a .p8 file to help generate the token. The information about the changes we received from Apple clearly indicates token connections are affected ("On March 29, 2021, token and certificate-based HTTP/2 connections to the Apple Push Notification service must incorporate the new root certificate ..."), but it's not clear how.

@sal_from_new_york

Thanks for replying. I will tell my server guy to check that the certificates in locations you mentioned on our ubuntu server.

/etc/ssl/certs/
 or /etc/ssl/certs/ca-certificates.crt

If it's not there, i think we are supposed to place the certificate there. the AAA cert can be downloaded from Sectigo knowledge base website as mentioned in the update. for some reason, i can't paste the link here. it says the URL can't be included.

Thanks

Hi @awaisfayyaz17.apple, you're quite welcome.

I was able to get detailed help from Apple as to what to do regarding the new vs. old root CAs. I'm sharing and hoping that you may find it useful.

If you are using a third party service to send push notifications, it is their responsibility to update their CAs. Therefore contact them to ensure a seamless transition. The rest of this post is for developers who maintain a backend server which directly communicates via HTTP/2 to Apple's APNS servers.

The GeoTrust Global CA is currently being used by api.push.apple.com:443. This is indirect via the intermediary certificate Apple IST CA 2 - G1 which is authorized via the GeoTrust Global CA. The site certificate is then authorized via the intermediate Apple certificate. My mistake was not looking all the way up the certificate path to the root, and if I caused any confusion in my earlier posts, I apologize.

Regardless of where your server's trust store is located you must make sure that A) it's the one used by server app and B) it contains both the GeoTrust Global CA and the derived intermediary Apple IST CA 2 - G1 certificate. In my case we are using curl and passing a file which contains the root certificate and its intermediary Apple IST CA 2 - G1.

In my case, we were using curl with a default where if it could not authenticate the certificate chain of api.push.apple.com it would continue anyway. This was being treated as a warning and not an error, making us potentially susceptible to MITM attacks, as well as other issues in the future.

This is easily testable since Apple is currently using that certificate chain until March 29th. Note that api.push.apple.com, api.development.apple.com and api.sandbox.push.apple.com are all using the same certificate chain.

As for the new AAACertificateServices Root CA, if you're using a Mac OS X machine to send APNS payloads to Apple, you most likely already have it in your keychain. Still, you should verify it. if it's a Linux or Windows box, you should verify that it's in the trust store being used by your server app to send the APN payload. Keep in mind that although your server has a default trust store, your app may not be using it and may be using a file or some other certificate repository. You need to verify which certificate trust store is being used by your server.

When you go to the Sectigo web site (the url is in the Apple announcement as well as in the email send to you by Apple) to obtain the AAACertificateServices, you'll notice that there are 3 certificates:

1. AAACertificateServices 5/12/2020

2. UserTrustRAAAACA 5/12/2020

3. COMODORSAAAACA 5/12/2020

The first is the root CA certificate, the last 2 are CA certificates authorized via #1. I assume that they are intermediary certificate, so I grabbed those as well. I placed all 3 in the same trust store as the GeoTrust Global root CA and its intermediary Apple certificate.

Now here is the issue. I cannot test the new root CA until Apple makes the switch to it. Furthermore, if the new site certificate for api.push.apple.com is derived from any of the 3 certificates, then everything will work fine after March 29th.

However if Apple creates a new Apple intermediary certificate derived from any of those 3 and then has the site certificate authorized by this new Apple intermediary certificate, Apple must share this certificate or the certificate chain will not be validated correctly by your software. I am really hoping that Apple is authorizing the site certificate with the AAACertificateServices root CA, and then everything will work. (It'll also work if Apple does it using the last 2 certificates as well.)

Thanks

I have an update, it turns out that there is a new Apple intermediate certificate: Apple Public Server RSA CA 12 - G1 that needs to also be added to the trust store. This is directly authorized by the AAACertificateServices Root CA. You can get it from:
www dot apple dot com / certificateauthority / public .

Click on the "valid" link under the Apple Public Server RSA CA 12 - G1 header. This certificate is in PEM format.

@sal_from_new_york where did you get this information? Im using ubuntu 20.04 LTS, should be pretty updated but can't find the AAACertificateServices Root CA, only the other two found in the sertigo link: USERTrustRSAAAACA 5/12/2020
COMODORSAAAACA 5/12/2020. I wonder if this is enough. I dont know how to test if this is enough