iOS 10+ provides the crypto API SecKeyCreateEncryptedData to encrypt data. Using an algorithm of kSecKeyAlgorithmRSAEncryptionOAEPSHA256AESGCM, the data to be encrypted is encrypted using a randomly generated AES-GCM key, and a 16 byte AES-GCM tag is generated. The AES key is encrypted using the RSA public key.
We are encrypting data using this method and sending it to a non-macOS server, on which SecKeyCreateDecryptedData is not available. The server is using a recent version of OpenSSL.
As a simple test, we are using a 2048 bit RSA key and 16 bytes of data to encrypt. The result of the call to SecKeyCreateEncryptedData is a 288 byte Data.
After sending the data to the server, the server is able to successfully decrypt the first 256 bytes of the buffer as the AES key. But the AES decryption fails because the tag verification fails, and the message is not even close to being the same as the original.
Our question is, how to successfully decrypt a buffer encrypted with kSecKeyAlgorithmRSAEncryptionOAEPSHA256AESGCM? Given our test message, we assumed the format of the buffer returned from SecKeyCreateEncryptedData is:
bytes 0..<256 = RSA encrypted AES key
bytes 256..<272 = AES encrypted message
bytes 272..<288 = AES-GCM tag
But that may not be correct. The comment in the header for kSecKeyAlgorithmRSAEncryptionOAEPSHA256AESGCM says: “Raw public key data is used as authentication data for AES-GCM encryption”. We added the RSA public key data as authentication data to the AES decryption operation, but the tag verification still fails.
How does one successfully decrypt a buffer encrypted with kSecKeyAlgorithmRSAEncryptionOAEPSHA256AESGCM?
Thanks!