-34018 A required entitlement isn't present. from command line swift program and more

[Just when you thought that the keychain API couldn’t get any weirder…]

First up, some background info:

• TN3137 On Mac keychain APIs and implementations explains the difference between the data protection and file-based keychain.

• SecItem: Fundamentals and SecItem: Pitfalls and Best Practices provide lots of important backstory, including more details about the keychain shim on macOS.

• Sharing access to keychain items among a collection of apps explains how the data protection keychain uses entitlements for access control.

• TN3125 Inside Code Signing: Provisioning Profiles talks about restricted entitlements and provisioning profiles.

• SecItem attributes for keys has some info about the semantics of each of the label-ish keychain attributes.

You’ve described two issues here:

• When you add a certificate to the keychain with a custom label, you can’t then find that certificate using that label.

• When you add a keychain to the keychain from your command-line tool, that fails with -34018.

I’ll explain the cause of each in turn, and then offer advice as to how to proceed.

The first issue is one of the many incompatibilities caused by the keychain shim on macOS. When you add an object reference to the keychain using kSecValueRef, the system populates the keychain attributes from the attributes associated with that object. That raises the question of what happens when you also specify attributes via the dictionary. For example, if the object has an internal label and you specify a label yourself using kSecAttrLabel, what do you get?

Annoyingly, the results differ between the file-based keychain, through the shim, and the data protection keychain:

• The shim uses the label attached to the certificate object.

• The data protection keychain uses the label you supply via kSecAttrLabel.

)-:

Hence the behaviour you’re seeing. The certificate ends up in the file-based keychain with a kSecAttrLabel value of testdeletejune27.invisinet.com, so you can’t find it when you query for a certificate with the kSecAttrLabel of com.example.mycert.

The second issues seems to be related to a recent change in SecItemAdd when dealing with keys. In general, SecItemAdd no macOS goes through the shim and thus targets the file-based keychain by default. However, it seems that on modern systems (I’m testing on macOS 14.5) it’s targeting the data protection keychain. So, your key ends up in the data protection keychain, which is all well and good until you try to do this from a command-line tool.

The data protection keychain relies on keychain access groups, and the use of keychain access groups is mediated by entitlements. Those are restricted entitlements, which means that they must be authorised by a provisioning profile. That’s problematic for a command-line tool because there’s no place to store that profile (r. 125850707)

So, when you call SecItemAdd it tries to add the key to the data protection keychain, which fails because you don’t have the entitlements necessary to establish a default keychain access group, and hence the -34018 error, aka errSecMissingEntitlement.

So, what should you do? First up, stop using kSecAttrLabel. Regardless of what else is going on here, that attribute is meant for a user-visible label, not for you the app developer. The correct option is kSecAttrApplicationTag.

Now, that attribute is only supported on keys, not certificates. What to do with certificates is tricky. Most folks who import a certificate and a private key are trying to form a digital identity. Is that what you’re doing here? If so, you don’t need to label the certificate because you can search for a digital identity and then identify both components using the label from the private key component.

If you’re not trying to form a digital identity then things get trickier. In that case, please post back with more details about how you’re tracking certificates.

Regarding the errSecMissingEntitlement error, you need to make a decision about whether you want to target the data protection keychain or the file-based keychain. IMO it’s better to use the former rather than the latter. Quoting TN3137:

The file-based keychain is on the road to deprecation.

If you want to stick with the file-base keychain, the simple fix is to pass false to the kSecUseDataProtectionKeychain on SecItemAdd.

If you want to move forward to the data protection keychain, you need to sign your command-line tool with the necessary entitlements (specifically, the App ID entitlement). This is possible, but a little painful. I wrote up my recommendations in Signing a daemon with a restricted entitlement.

*phew*

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

First of all, thank you for the superbly detailed response! Quinn always delivers.

So, it sounds like the data protection keychain is the way to go and more portable between macOS and iOS and we are implementing on both. I created a test .app to call my code, added appropriate entitlement, and your suggestions work so far, I can add a key with a tag, find it, etc.

For tracking identities we just generate a GUID to store in core data with some other app specific items for each identity.

When we run our app of course we would like to find the cert/key pair using that ID.

Here is what I plan on doing.

1. Add the key with a tag (GUID)
2. Add the certificate
3. When I need to grab the key/cert I will

a) find the key using that tag b) find the cert associated with that key

Of course we would also like to delete the identities when we no longer need them.

I'm not clear on your statement: "identify both components using the label from the private key component."

You mentioned I should stop using labels, so I'm not sure what "label" means in this context.

How do I go about finding the cert once i have the key?

Is there an easy way to identify the key associated with a certificate in the keychain access app?

Is there any advantage to getting the certs to show up in the MyCertificates section I see in keychain app or to have our own keychain?

alf

I'm not sure what "label" means in this context.

Yeah, sorry, I was using that in the general sense, not in the sense of kSecAttrLabel.

How do I go about finding the cert once i have the key?

That ties into my question about about digital identities. Most folks who have a private key and a certificate are actually working with a digital identity. That is, the private key matches the public key in the certificate. Is that the case here? Or are the key and the certificate unrelated?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Sorry, yes, I am trying to create/manage identities.

On a side note I am concerned about the one to may thing with keys and certificates and how to manage that possibility. Multiple certs for a key may (hopefully) be an error condition for us but I still have to handle it in the code.

Back to the main problem. Using the data protection keychain I can add, find, delete keys no problem using kSecAttrApplicationTag. I was also able to add the cert previously discussed without specifying any tag or label but am not able to find it with the label testdeletejune27.invisinet.com, so maybe that label isn't assigned with the data security keychain? In addition, it seems that certs in this keychain aren't visible with the keychain access app or apparently security find-certificate -a, so it's disconcerting not having another way to verify the state of things. Also, now that my cert is added I have no way of deleting it and I know it's there as I get an already exists error if I try to add it again.

I think I understand now that I need to find the cert from the key and I have an example of that via the AI bots, so can move in that direction. Once I can find the cert, I am still not sure how to delete it, since SecItemDelete needs a query that works.

This example I found seems like it may be a good way to manage these identities. Is this the right approach? what about the one to many issue? I'm not sure we will have control as far as how the customer will generate their certs.:

  var items = [String: Any]()
  
  items[kSecClass as String] = kSecClassIdentity
  items[kSecAttrApplicationTag as String] = tag.data(using: .utf8)
  items[kSecImportExportPassphrase as String] = kCFNull // Optional passphrase
  items[kSecValueRef as String] = SecPKCS12Import(certificateData: certificateData as CFData, keyData: privateKeyData as CFData, options: [:])
  
  let status = SecItemAdd(items as CFDictionary, nil)
  return status
}

func findIdentity(forKeyTag tag: String) -> SecIdentity? {
  var query = [String: Any]()
  query[kSecClass as String] = kSecClassIdentity
  query[kSecAttrApplicationTag as String] = tag.data(using: .utf8)
  query[kSecReturnRef as String] = kCFBooleanTrue!
  query[kSecUseDataProtectionKeychain as String] = true
  
  var result: AnyObject?
  let status = SecItemCopyMatching(query as CFDictionary, &result)
  
  if status == errSecSuccess, let itemRef = result as? SecCFTypeRef {
    return itemRef as? SecIdentity
  } else {
    // Handle error (e.g., no item found)
    print("Error searching keychain: \(status)")
    return nil
  }
}

func deleteIdentity(forKeyTag tag: String) -> OSStatus {
  var query = [String: Any]()
  query[kSecClass as String] = kSecClassIdentity
  query[kSecAttrApplicationTag as String] = tag.data(using: .utf8)
  
  let status = SecItemDelete(query as CFDictionary)
  return status
}

Sorry, missed the first line of that first function: func addIdentityToKeychain(certificateData: Data, privateKeyData: Data, tag: String) -> OSStatus {

Also, would this code work the same on iOS?

I also now realize I didn't notice that example wasn't using the data protection keychain

When I was able to add a key to the data protection keychain, I then tried the following code to look for all certs, but I think it didn't like the kSecUseDataProtectionKeychain attribute there, as I got:

findCertificatesFromKey(key:): 358, certStatus: -25303 The specified attribute does not exist.

code:

func findCertificatesFromKey(key: SecKey) -> [SecCertificate]? {
	let certQuery: [String: Any] = [
		kSecUseDataProtectionKeychain as String: true,
		kSecClass as String: kSecClassCertificate,
		kSecAttrKeyType as String: kSecAttrKeyTypeRSA,  // or other key type as required
		kSecReturnRef as String: true,
		kSecMatchLimit as String: kSecMatchLimitAll
	]

	var certItems: CFTypeRef?
	let certStatus = SecItemCopyMatching(certQuery as CFDictionary, &certItems)

	if certStatus == errSecSuccess {
		if let certArray = certItems as? [SecCertificate] {
			for certificate in certArray {
				// Here you can check if the certificate is associated with the key
				print("Certificate found: \(certificate)")
				// You can use SecCertificateCopyValues to get more info about the certificate
			}
			return certArray
		} else {
			print("No certificates found")
		}
	} else {
		print("\(#function): \(#line), certStatus: \(certStatus) \(SecCopyErrorMessageString(certStatus, nil) as String? ?? "Unknown error")")
	}
	return nil
}

I am trying to create/manage identities.

Thanks for confirming that.

it seems that certs in this keychain aren't visible with the keychain access app or apparently security find-certificate

Yep. Both of those are a common source of confusion, which I call out in the User interface section of TN3137.

I usually work around this by adding a small test UI to my app that lets me do keychain-related stuff, like dump all the items in my keychain access group, delete those items, and so on.

In terms of how you handle multiple certificates for the same key, the trick here is to work in ‘identity space’. If there are multiple certificates for the same key, you’ll get an identity for each one. It’s trivial to get the certificate and key from each identity (SecIdentityCopyCertificate and SecIdentityCopyPrivateKey). You can then check for key equality using CFEquals [1].

Alternatively, if you want to prevent this from happening then you can do that on the import side. When you import your PKCS#12, you get an identity. Add its private key to the keychain first. If it’s a dup, that add will fail, and you know you shouldn’t be adding this digital identity.

-25303 The specified attribute does not exist.

Right. Look at the parameter block you’re passing in there:

let certQuery: [String: Any] = [
    kSecUseDataProtectionKeychain as String: true,
    kSecClass as String: kSecClassCertificate,
    kSecAttrKeyType as String: kSecAttrKeyTypeRSA,
    …
]

You’re querying for a certificate (kSecClass is kSecClassCertificate) and the certificate ‘table’ only has certificate ‘columns’. Thus, you can’t query for a key ‘column’ (kSecAttrKeyType).

Earlier I mentioned my two SecItem posts:

• SecItem: Fundamentals

• SecItem: Pitfalls and Best Practices

I recommend that you go back and have a read of those. The first explains how you can view the keychain as an SQL database, with a table for each item class (kSecClass), columns in that table representing the various attributes of such items, and a row in that table for each item. If you keep that model in your head it’s much easier to understand what’s going on.

Oh, and the second has a bunch of info about digital identity formation. It also explains how to avoid all those annoying as String casts (-:

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] I can’t remember whether the Swift == operator will route through to that. You’ll have to test it.

After reading your links and suggestions I decided to try working in the identity space as you suggested.

On iOS, I can convert my key/cert combo to PKCS12 and import it, but when I try to add the identity to the keychain I get: status: -50 One or more parameters passed to a function were not valid. Any idea what param may be wrong? Tried removing tag, password, data protection, still same error.

On macOS, the SecPKCS12Import fails with: status: -25264 MAC verification failed during PKCS12 import (wrong password?) I get the same if I pass "" for the password or just omit it.

func addIdentityToKeychain(certificatePEM: String, privateKeyPEM: String, tag: String) -> OSStatus {
	guard let certificateData = certificatePEM.data(using: .utf8) else {
		print("Error converting certificate PEM to data")
		return errSecParam
	}
	guard let privateKeyData = privateKeyPEM.data(using: .utf8) else {
		print("Error converting private key PEM to data")
		return errSecParam
	}
	guard let secCertificate = convertPEMToSecCertificate(certificatePEM) else {
		print("Error converting PEM TO SecCertificate")
		return errSecParam
	}
	guard let secKey = convertPEMToSecKey(privateKeyPEM) else {
		print("Error converting PEM TO SecKey")
		return errSecParam
	}

	let (pkcs12Data, err) = createPKCS12Data(certificate: secCertificate, key: secKey, p12Name: "p12Name", p12Password: "")
	
	guard let pkcs12Data = pkcs12Data else {
		return errSecParam
	}

	var items: CFArray?
	let status = SecPKCS12Import(pkcs12Data as NSData, [kSecImportExportPassphrase as String: ""] as CFDictionary, &items)
	if status != errSecSuccess {
		print("\(#function): \(#line), SecPKCS12Import failed, status: \(status) \(SecCopyErrorMessageString(status, nil) as String? ?? "Unknown error")")
		return status
	}

	if let importedItems = items as? Array<Any> {
		guard let identity = importedItems.first  else {
			return errSecItemNotFound
		}
		let attrs = [
			kSecClass: kSecClassIdentity,
			kSecAttrApplicationTag: tag,
			kSecImportExportPassphrase: kCFNull!, // Optional passphrase (or your CFString passphrase)
			kSecValueRef: identity,
			kSecUseDataProtectionKeychain: true
		] as NSDictionary
		
		let stat = SecItemAdd(attrs, nil)
		
		if stat != errSecSuccess {
			print("\(#function): \(#line), SecItemAdd of identity failed, status: \(stat) \(SecCopyErrorMessageString(stat, nil) as String? ?? "Unknown error")")
			return stat
		}
	}

	return errSecSuccess
}

Any ideas? alf

Progress. I noticed the debugger that the "identity" returned by SecPKCS12Import was actually an array where the first element was a dictionary with identity, trust and chain in there. So, I was able to pull out what I think is the real identity:

	let dics = items! as! Array<Dictionary<String, Any>>
	let firstItem = dics[0]
	
	let identity = firstItem[kSecImportItemIdentity as String] as! SecIdentity?

And as long as I don't use the: kSecAttrApplicationTag: tag, element in the query I can add the identity with SecItemAdd. If I try to add a tag to it, I get status: -25303 The specified attribute does not exist.

So I can add it but I won't be able to find/delete later if I can't put a tag on it.

Any suggestions?

The issue with kSecAttrApplicationTag is that it only applies to keys, but an identity is the combination of a private key and a certificate. The keychain can’t apply the tag to your certificate and that’s why the add fails.

The solution is to split your identity into its components (using SecIdentityCopyPrivateKey and SecIdentityCopyCertificate) and add each one individually. You can then apply the tag to the key.

Regarding your macOS issue, AFAIK our PKCS#12 parser doesn’t support PKCS#12 structures without a password. Honestly, I’m surprised that works on iOS. You should build your PKCS#12 with a password — it doesn’t have to be a good one! — and apply that when you import it.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Adding a password worked for the pkcs12import, thank you.

On iOS I was able to add/find/delete if I add the identity using kSecAttrLabel and search with that. On mac, add and delete work fine with the same code but find using kSecAttrLabel label fails to find it. I don't have to add key or cert individually, and I can extract them from the identity if I need them.

Reading your link about attributes, it says this is the user visible portion in the keychain access app, but it is not a component of key uniqueness. Is this true for identites? If I use a lable with a UUID in it, it makes it unique for our concern, but maybe what you're saying is that it's not enforced by the keychain? Also, the data protection keychain doesn't show up in the keychain access app so that part is moot I guess.

Previously you suggested not to use kSecAttrLabel, but it works except for the find on mac so far. Is that a "feature" of the shim on mac?

This seems like a clean sequence to me except for the find not working.

I'm not clear on the sequence you are suggesting. It sounds like you are suggesting I add the key with SecAttrApplicationTag and certificate with no tag since that attribute is not supported for certs to the keychain in addition to adding the identity.

Will I be able to search for the identity by the tag I use for the key?

Are you suggesting sequence of:

add identity with no tag or label add key with SecAttrApplicationTag add cert (maybe not necessary since I can extract from identity later)?

I just reread the Pitfalls and Best Practices again to remind me why I should use certain attrbutes, so looking for the magic sequence so I can add, find, and delete identities.

Is the SecAttrApplicationTag supported by CopyMatching, and if so will it find the identity that contains the key I added separately with a SecAttrApplicationTag? if so, I guess this will get my find and delete for me. Will give that a try.

If I add the key and the cert separately, will I be able to find them as an identity without adding the identity explicitly with SecItemAdd or do I still need to do that?

Sorry for all the questions, but some of these details are not obvious.

alf

latest code segment:

func addIdentityToKeychain(certificatePEM: String, privateKeyPEM: String, tag: String) -> OSStatus {
	guard let certificateData = certificatePEM.data(using: .utf8) else {
		print("Error converting certificate PEM to data")
		return errSecParam
	}
	guard let privateKeyData = privateKeyPEM.data(using: .utf8) else {
		print("Error converting private key PEM to data")
		return errSecParam
	}
	guard let secCertificate = convertPEMToSecCertificate(certificatePEM) else {
		print("Error converting PEM TO SecCertificate")
		return errSecParam
	}
	guard let secKey = convertPEMToSecKey(privateKeyPEM) else {
		print("Error converting PEM TO SecKey")
		return errSecParam
	}

	let (pkcs12Data, err) = createPKCS12Data(certificate: secCertificate, key: secKey, p12Name: "p12Name", p12Password: "p12pass")
	
	guard let pkcs12Data = pkcs12Data else {
		return errSecParam
	}

	do {
		try writeDataToFilePath(data: pkcs12Data, filePath: "/Users/alfred_eisenberg/Downloads/pkcs12.p12")
	}
	catch {
		print("Error writing data to file: \(error)")
	}
	
	var items: CFArray?
	let status = SecPKCS12Import(pkcs12Data as NSData, [kSecImportExportPassphrase:  "p12pass"] as NSDictionary, &items)
	if status != errSecSuccess {
		print("\(#function): \(#line), SecPKCS12Import failed, status: \(status) \(SecCopyErrorMessageString(status, nil) as String? ?? "Unknown error")")
		return status
	}

	let dics = items! as! Array<Dictionary<String, Any>>
	let firstItem = dics[0]
	
	let identity = firstItem[kSecImportItemIdentity as String] as! SecIdentity?

	var privateKey: SecKey?
	let statusKey = SecIdentityCopyPrivateKey(identity!, &privateKey)
	guard statusKey == errSecSuccess else {
		return statusKey
	}
	var certificate: SecCertificate?
	let statuscert = SecIdentityCopyCertificate(identity!, &certificate)
	guard statuscert == errSecSuccess else {
		return statuscert
	}

//	dumpCertFromSecCertificate(cert: certificate!)
	
	let attrs = [
		kSecClass: kSecClassIdentity,
		kSecAttrLabel: tag.data(using: .utf8)!,
//		kSecAttrApplicationTag: tag.data(using: .utf8)!,
//		kSecAttrApplicationLabel: tag.data(using: .utf8)!,
//		kSecImportExportPassphrase: kCFNull!, // Optional passphrase (or your CFString passphrase)
		kSecValueRef: identity!,
		kSecUseDataProtectionKeychain: true
	] as NSDictionary
	
	let stat = SecItemAdd(attrs, nil)
	
	if stat != errSecSuccess {
		print("\(#function): \(#line), SecItemAdd of identity failed, status: \(stat) \(SecCopyErrorMessageString(stat, nil) as String? ?? "Unknown error")")
		return stat
	}

	print("\(#function): \(#line), SecItemAdd of identity Succeeded")
	dumpCertFromSecCertificate(cert: certificate!)
	return errSecSuccess
}

func findIdentity(forKeyTag tag: String) -> SecIdentity? {
	var query = [
		kSecClass: kSecClassIdentity,
		kSecAttrLabel: tag.data(using: .utf8)!,
//	kSecAttrApplicationLabel: tag.data(using: .utf8),
//	kSecAttrApplicationTag: = tag.data(using: .utf8),
		kSecReturnRef : kCFBooleanTrue!,
		kSecUseDataProtectionKeychain: true
	] as NSDictionary
	
	var result: AnyObject?
	let status = SecItemCopyMatching(query as CFDictionary, &result)
	
	if status == errSecSuccess, let item = result {
		print("\(#function): \(#line),  Find identity Succeeded")
		return item as! SecIdentity // Forced cast (use with caution)
	} else {
		// Handle error (e.g., no item found)
		print("\(#function): \(#line), Can't find identity with tag: \(tag), status: \(status) \(SecCopyErrorMessageString(status, nil) as String? ?? "Unknown error")")
	}
	return nil
}

func deleteIdentity(forKeyTag tag: String) -> OSStatus {
	var query = [
		kSecClass: kSecClassIdentity,
		kSecAttrLabel: tag.data(using: .utf8)!,
//		kSecAttrApplicationLabel: tag.data(using: .utf8),
//		kSecAttrApplicationTag: tag.data(using: .utf8),
		kSecUseDataProtectionKeychain: true
	] as NSDictionary
	
	let status = SecItemDelete(query as CFDictionary)
	if status != errSecSuccess {
		// Handle error (e.g., no item found)
		print("\(#function): \(#line), Can't delete identity with tag: \(tag), status: \(status) \(SecCopyErrorMessageString(status, nil) as String? ?? "Unknown error")")
	}
	
	print("\(#function): \(#line),  Delete identity Succeeded")
	return status
}

but it is not a component of key uniqueness. Is this true for identites?

I try to avoid thinking about uniqueness for digital identities. It hurts my brain!

Seriously though, the concept of item uniqueness is easy to grok if you use SQL as your guide. However, that breaks down when talking about identities because each component goes into its own SQL table, and SQL has nothing to say about how uniqueness constraints apply across tables.

Moreover, even if you ignore that and focus on theoretical solutions, the fact that multiple certificates can be issued for the same private key means that there’s no well-defined solution to the uniqueness problem. Something has to give.

Now, clearly SecItemAdd has to do something, and you could work out what it does with some tests. However, I’m disinclined to do so. One reason for that is that I’d have to do it twice, once for the data protection keychain and once for the macOS file-based keychain shim; I have zero confidence that the latter faithfully emulates the former. But the main reason is that I don’t have to. In all but the simplest cases, it’s just easier to reason about each of the identity’s component separately.

Is that a "feature" of the shim on mac?

I think that goes without saying. The purpose of the shim is to implement the SecItem API on top of the file-based keychain. Any time it deviates from the behaviour you get with the data protection keychain, that’s a bug IMO.

I usually file such bugs when I bump into them. However, fixing them can be a challenge due to compatibility concerns.

Coming back to the big picture, the tricky part here is that you plan to identify keys by a unique kSecAttrApplicationTag value. However, kSecAttrApplicationTag forms part of the uniqueness constraint for key items. So, by applying a unique application tag, you guarantee that adding the key can never fail with errSecDuplicateItem.

So, you have a choice to make: Are you happy to assume that you won’t be asked to import the same key twice? If so, you can ignore this and move on. If not, you have to factor that into your design.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"