Apple Push Notification service server certificate update

"verify that both the old and new root certificates for the HTTP/2 interface are included in the Trust Store of each of your notification servers before March 29"

How can i verify that the required certificates (AAACertificateServices) are included in the Trust store of my backend server?

My backend server's OS is Ubuntu 18.04.3 LTS.

Question: I use Axway platform do send push notifications. The way to do it is to convert my Apple Push Notification certificate in p12 format on my Mac and then load it on the Axway online dashboard.
This morning I created a new Apple Push Notification certificate (the old one expires in a few days), I downloaded and installed it in my keychain, then I downloaded and installed the "AAA certificate services" and then I created the new P12 certificate for the Axway dashboard. Is it enough? How can I check if my new p12 certificate has or not the root "AAA certificate services"? Thank you in advance

It seems that I have the same situation as @ds_wrg: we are using npm package to connect to APN on our server-side with a .p8 file and not using any certificates.

The package we use:
https://github.com/node-apn/node-apn#readme

It is “Based on HTTP/2 based provider API”.
And we use, if I understand correctly, token-based authentication way. I.e to create a new connection we use such such kind of data only:

Code Block 
token: {
  key: "path/to/APNsAuthKey_XXXXXXXXXX.p8",
  keyId: "key-id",
  teamId: "developer-team-id"
},

So the question:
1) do these “root certificates” changes affect us in any way?
2) And if so, what and where should be changed for us ?"

Somewhat similar situation as @Nikitkirik here. Although we don't use npm, we use curl to send APN payloads to Apple servers, and it is absolutely not clear why this is necessary.

We are not using the GeoTrust root CA for anything, and can't locate it on our servers. Our push notifications have been working for about 2 years and we are using HTTP/2. Can someone from Apple tell me what the GeoTrust Root CA is being used for right now w.r.t APNS? Our SSL cert and key are signed with the Apple Root CA, and so are the Apple push servers (that certificate expires in April 2021 btw).

This thread is not very helpful. I had to file a TSI and hopefully someone from Apple will get back to me.

Hi @awaisfayyaz17.apple,
we have a similar situation. We looked for the GeoTrust Root CA in out server's trust store and it's not there, and it's also not being picked up by curl. Our server is a Windows 10 box (don't ask, we have very low volume push notifications and it was not my decision) so it was easy to find the trust store and list all of the root CA's. No GeoTrust Root Ca in it.

For Ubuntu 18.xx the trust store is accessible in /etc/ssl/certs/ or /etc/ssl/certs/ca-certificates.crt for the CA certs. What will you do if the GeoTrust Root CA is not in there?

For us we specify --cacerts file for the curl command, but it contains the Apple Root CA, which I believe is not derived from the GeoTrust root CA.

I have the same question others have asked - we are using the JWT token approach to authenticate when sending notifications. During that process we use a .p8 file to help generate the token. The information about the changes we received from Apple clearly indicates token connections are affected ("On March 29, 2021, token and certificate-based HTTP/2 connections to the Apple Push Notification service must incorporate the new root certificate ..."), but it's not clear how.

@sal_from_new_york

Thanks for replying. I will tell my server guy to check that the certificates in locations you mentioned on our ubuntu server.

/etc/ssl/certs/
 or /etc/ssl/certs/ca-certificates.crt

If it's not there, i think we are supposed to place the certificate there. the AAA cert can be downloaded from Sectigo knowledge base website as mentioned in the update. for some reason, i can't paste the link here. it says the URL can't be included.

Thanks

Hi @awaisfayyaz17.apple, you're quite welcome.

I was able to get detailed help from Apple as to what to do regarding the new vs. old root CAs. I'm sharing and hoping that you may find it useful.

If you are using a third party service to send push notifications, it is their responsibility to update their CAs. Therefore contact them to ensure a seamless transition. The rest of this post is for developers who maintain a backend server which directly communicates via HTTP/2 to Apple's APNS servers.

The GeoTrust Global CA is currently being used by api.push.apple.com:443. This is indirect via the intermediary certificate Apple IST CA 2 - G1 which is authorized via the GeoTrust Global CA. The site certificate is then authorized via the intermediate Apple certificate. My mistake was not looking all the way up the certificate path to the root, and if I caused any confusion in my earlier posts, I apologize.

Regardless of where your server's trust store is located you must make sure that A) it's the one used by server app and B) it contains both the GeoTrust Global CA and the derived intermediary Apple IST CA 2 - G1 certificate. In my case we are using curl and passing a file which contains the root certificate and its intermediary Apple IST CA 2 - G1.

In my case, we were using curl with a default where if it could not authenticate the certificate chain of api.push.apple.com it would continue anyway. This was being treated as a warning and not an error, making us potentially susceptible to MITM attacks, as well as other issues in the future.

This is easily testable since Apple is currently using that certificate chain until March 29th. Note that api.push.apple.com, api.development.apple.com and api.sandbox.push.apple.com are all using the same certificate chain.

As for the new AAACertificateServices Root CA, if you're using a Mac OS X machine to send APNS payloads to Apple, you most likely already have it in your keychain. Still, you should verify it. if it's a Linux or Windows box, you should verify that it's in the trust store being used by your server app to send the APN payload. Keep in mind that although your server has a default trust store, your app may not be using it and may be using a file or some other certificate repository. You need to verify which certificate trust store is being used by your server.

When you go to the Sectigo web site (the url is in the Apple announcement as well as in the email send to you by Apple) to obtain the AAACertificateServices, you'll notice that there are 3 certificates:

1. AAACertificateServices 5/12/2020

2. UserTrustRAAAACA 5/12/2020

3. COMODORSAAAACA 5/12/2020

The first is the root CA certificate, the last 2 are CA certificates authorized via #1. I assume that they are intermediary certificate, so I grabbed those as well. I placed all 3 in the same trust store as the GeoTrust Global root CA and its intermediary Apple certificate.

Now here is the issue. I cannot test the new root CA until Apple makes the switch to it. Furthermore, if the new site certificate for api.push.apple.com is derived from any of the 3 certificates, then everything will work fine after March 29th.

However if Apple creates a new Apple intermediary certificate derived from any of those 3 and then has the site certificate authorized by this new Apple intermediary certificate, Apple must share this certificate or the certificate chain will not be validated correctly by your software. I am really hoping that Apple is authorizing the site certificate with the AAACertificateServices root CA, and then everything will work. (It'll also work if Apple does it using the last 2 certificates as well.)

Thanks

I have an update, it turns out that there is a new Apple intermediate certificate: Apple Public Server RSA CA 12 - G1 that needs to also be added to the trust store. This is directly authorized by the AAACertificateServices Root CA. You can get it from:
www dot apple dot com / certificateauthority / public .

Click on the "valid" link under the Apple Public Server RSA CA 12 - G1 header. This certificate is in PEM format.

@sal_from_new_york where did you get this information? Im using ubuntu 20.04 LTS, should be pretty updated but can't find the AAACertificateServices Root CA, only the other two found in the sertigo link: USERTrustRSAAAACA 5/12/2020
COMODORSAAAACA 5/12/2020. I wonder if this is enough. I dont know how to test if this is enough

@sal

Thank you very much for posting the TSI and submitting a detailed response here.
Much appreciated!

We have the new cert installed on our server however when we try to connect using:
~$ openssl s_client -showcerts -connect api.sandbox.push.apple.com:443

We get the following

Code Block 
CONNECTED(00000003)
depth=1 CN = Apple Public Server RSA CA 12 - G1, O = Apple Inc., ST = California, C = US
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/CN=api.development.push.apple.com/OU=management:idms.group.533599/O=Apple Inc./ST=California/C=US
   i:/CN=Apple Public Server RSA CA 12 - G1/O=Apple Inc./ST=California/C=US
-----BEGIN CERTIFICATE-----
.........

This line:
unable to get local issuer certificate

is what concerns me.. What are we doing wrong here?

Also this line concerns me more and may have more clues as to the problem:

@@ X���HTTP/2 client preface string missing or corrupt. Hex dump for received bytes: 0aread:errno=0

Can anyone help direct me on this matter? I'm an iOS dev so I'm not used to handling these types of issues.

@sal_from_new_york I actually copied all the three new certs to /etc/ssl/certs/.

Do i have to run "update-ca-certificates" to make sure they are updated?

Thanks in advance