First of all I know MD5 is "broken" and we shouldn't be using it but it's not my choice and there is legacy code I need to work with so for now at least I need to sign some data using MD5 and a RSA signing Key.
I think this is enough to explain the problem but I can share more code if needed.
I'm using this to sign:
let result = SecKeyRawSign(privateKey, SecPadding.PKCS1MD5, digestData.arrayOfBytes(), digestData.count, &signatureBytes, &signatureDataLength)
But that throws the error OSStatusCode -50 (parameter error from the Security framework)
If I change the padding to SecPadding.PKCS1SHA256 it works and after testing all the SecPadding options:
Fails with OSStatus -50: OAEP sigRaw PKCS1MD2 PKCS1MD5
Works: PKCS1 PKCS1SHA1 PKCS1SHA224 PKCS1SHA256 PKCS1SHA384 PKCS1SHA512
The digestData is created using CC_MD5 no matter which padding I've used so its stranger that it works with the SHA padding but not with MD5.
I'm getting a similar error when attempting to do this using the SwCrypt library, when using MD5 & PKCS1 padding I get the OSStatus -4300 which is a parameter error from CommonCrypto.
let result = try CC.RSA.sign(paramString.data(using: .utf8)!, derKey: der, padding: .pkcs15, digest: .md5, saltLen: 0)
However I don't get an error if using the pss padding option.
So my question/s then, why am I getting this parameter error when using MD5 and PKCS1? Is it because use of MD5 is deprecated or could there be some problem with another parameter?
Any pointers would be greatly appreciated, I've been stuck trying to implement this all week.
Fails with OSStatus -50: OAEP sigRaw PKCS1MD2 PKCS1MD5 Works: PKCS1 PKCS1SHA1 PKCS1SHA224 PKCS1SHA256 PKCS1SHA384 PKCS1SHA512
What looks like is happening here is that SecKeyRawSign tries to derive the signature algorithm from the key and SecPadding input and it cannot find matching results for anything MD related. This also matches the API definition.
The other APIs that you could look at here are SecKeyCreateSignature and CryptoKit, but CryptoKit does not support creating signatures with MD5. SecKeyCreateSignature skips right to passing in the SecKeyAlgorithm instead of deriving it from the padding, but there is not an option for MD5 here either. You could try rsaEncryptionRaw, but I am not confident that will work with MD5 either and your best bet may be just to advance to the SHA family of hashes here.
Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com