Decrypting Data Encrytped with kSecKeyAlgorithmRSAEncryptionOAEPSHA256AESGCM

iOS 10+ provides the crypto API SecKeyCreateEncryptedData to encrypt data. Using an algorithm of kSecKeyAlgorithmRSAEncryptionOAEPSHA256AESGCM, the data to be encrypted is encrypted using a randomly generated AES-GCM key, and a 16 byte AES-GCM tag is generated. The AES key is encrypted using the RSA public key.


We are encrypting data using this method and sending it to a non-macOS server, on which SecKeyCreateDecryptedData is not available. The server is using a recent version of OpenSSL.


As a simple test, we are using a 2048 bit RSA key and 16 bytes of data to encrypt. The result of the call to SecKeyCreateEncryptedData is a 288 byte Data.

After sending the data to the server, the server is able to successfully decrypt the first 256 bytes of the buffer as the AES key. But the AES decryption fails because the tag verification fails, and the message is not even close to being the same as the original.


Our question is, how to successfully decrypt a buffer encrypted with kSecKeyAlgorithmRSAEncryptionOAEPSHA256AESGCM? Given our test message, we assumed the format of the buffer returned from SecKeyCreateEncryptedData is:

bytes 0..<256 = RSA encrypted AES key

bytes 256..<272 = AES encrypted message

bytes 272..<288 = AES-GCM tag


But that may not be correct. The comment in the header for kSecKeyAlgorithmRSAEncryptionOAEPSHA256AESGCM says: “Raw public key data is used as authentication data for AES-GCM encryption”. We added the RSA public key data as authentication data to the AES decryption operation, but the tag verification still fails.


How does one successfully decrypt a buffer encrypted with kSecKeyAlgorithmRSAEncryptionOAEPSHA256AESGCM?


Thanks!

Accepted Reply

Given our test message, we assumed the format of the buffer returned from SecKeyCreateEncryptedData is …

Agreed.

The comment in the header for kSecKeyAlgorithmRSAEncryptionOAEPSHA256AESGCM says: “Raw public key data is used as authentication data for AES-GCM encryption”. We added the RSA public key data as authentication data to the AES decryption operation, but the tag verification still fails.

I suspect that’s where things are going wrong. There are many different ways to interpret “Raw public key data”. In this case the auth data input to the AES-GCM encryption is taken from the result of

SecKeyCopyExternalRepresentation
. IIRC this is a DER-encoded
RSAPublicKey
ASN.1 structure.

I recommend that, for testing purposes, you call

SecKeyCopyExternalRepresentation
on iOS, dump the results, and feed that into your OpenSSL AES-GCM decrypt.

Finally, be aware that much of the code for this stuff is in Darwin. In this case, check out

SecKeyRSAAESGCMCopyEncryptedData
in
Security/OSX/sec/Security/SecKeyAdaptors.c
.

Note This is the code that’s (roughly) equivalent to macOS 10.12, although the result format, which is what you care about, will be the same on iOS.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Replies

Given our test message, we assumed the format of the buffer returned from SecKeyCreateEncryptedData is …

Agreed.

The comment in the header for kSecKeyAlgorithmRSAEncryptionOAEPSHA256AESGCM says: “Raw public key data is used as authentication data for AES-GCM encryption”. We added the RSA public key data as authentication data to the AES decryption operation, but the tag verification still fails.

I suspect that’s where things are going wrong. There are many different ways to interpret “Raw public key data”. In this case the auth data input to the AES-GCM encryption is taken from the result of

SecKeyCopyExternalRepresentation
. IIRC this is a DER-encoded
RSAPublicKey
ASN.1 structure.

I recommend that, for testing purposes, you call

SecKeyCopyExternalRepresentation
on iOS, dump the results, and feed that into your OpenSSL AES-GCM decrypt.

Finally, be aware that much of the code for this stuff is in Darwin. In this case, check out

SecKeyRSAAESGCMCopyEncryptedData
in
Security/OSX/sec/Security/SecKeyAdaptors.c
.

Note This is the code that’s (roughly) equivalent to macOS 10.12, although the result format, which is what you care about, will be the same on iOS.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

That solved it. Thank you Quinn!