Unable to retrieve keychain data intermittently.

At present, we have been receiving numerous reports from customers who integrate our SDK who have been encountering the failures (errSecItemNotFound) while trying to retrieve a key using SecItemCopyMatching. We are raising this query we are still in the midst of properly reproducing this issue though it has been reported to occur in several devices during the OS upgrade to the iOS 17 betas. This issue is still occurring in the latest iOS 17 beta 7. This issue was not present in previous iOS version. At present, we are of the conclusion that this issue is occurring randomly amongst devices that upgraded to the iOS 17 betas and it is not limited to older devices.

What we believe is occurring is that:

  1. A key is created and stored into Keychain using SecItemAdd.
  2. The same key is queried at a later timepoint but encounters the error errSecItemNotFound.
  3. Our SDK then attempts to regenerate a new key for the same label and attribute to store it using SecItemAdd, but the system then reports errSecDuplicateItem at the key already exists.
  4. The workaround here includes a manual deletion of the said key.

This issue seems to occur only during an OS upgrade to the iOS17 betas with the likelihood that the key was already present in Keychain prior to the upgrade. I share below the snippet relating to how this said key is generated, stored and retrieved. // Initial key is added

// A random data of 32 bytes length is generated 
CFDataRef dataRef = <32bytes of data>; 
*attr = CFDictionaryCreateMutable(kCFAllocatorDefault, 0, &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks); 
BREAK_IF_NULL(*attr); 
CFDictionarySetValue(*attr, kSecClass, kSecClassGenericPassword); 
CFStringRef aKey = CFStringCreateWithCString(kCFAllocatorDefault, "KEY_ACCOUNT", kCFStringEncodingUTF8); 
CFDictionaryAddValue(*attr, kSecAttrAccount, aKey); 
CFDictionarySetValue(*attr, kSecReturnData, kCFBooleanFalse); CFDictionarySetValue(*attr, kSecAttrAccessible, kSecAttrAccessibleAfterFirstUnlock); 
label = CFStringCreateWithFormat(NULL, NULL, CFSTR("A_LABEL")); 
CFDictionarySetValue(attr, kSecAttrService, label); 
CFDictionarySetValue(attr, kSecValueData, dataRef); SecItemAdd(attr); 

// Query for retrieval of key

label = CFStringCreateWithFormat(NULL, NULL, CFSTR("A_LABEL")); CFDictionarySetValue(attributes, kSecAttrService, label); 
CFDictionarySetValue(attributes, kSecReturnData, kCFBooleanTrue); 
CFDictionarySetValue(attributes, kSecMatchLimit, limit); 
CFDictionarySetValue(attributes, kSecReturnAttributes, returnAttributes);
osStatus = SecItemCopyMatching(attributes, result); 
if (errSecItemNotFound == osStatus) { } 

Please do let me know if more information could be useful. At present, we have ensured that the key generated are well-within the size limits and is stored simply as a kSecClassGenericPassword with limited access control to the key. Additionally, the query used was intended to be generalised to avoid encountering such occurrences.

Answered by DTS Engineer in 803068022

The fix for the specific issue discussed upthread (FB13066335) shipped in iOS 17.1.

However, that’s one specific fix. It’s perfectly feasible that there are other issues with similar symptoms. In my experience very few people are committed enough to investigate issues like this with the determination shown by the OP (thanks mervyn.ong!).

If you’re seeing a mysterious keychain problems, I recommend that you read through:

A significant majority of such problems are caused by folks using the SecItem API incorrectly [1].

If that doesn’t help, you’ll need to kick of your own investigation. I have a post, Investigating hard-to-reproduce keychain problems, that outlines my recommended approach for this.

If you manage to isolate this per the advice in that post, feel free to start a new thread with the details.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Well, at a meta level you could argue that they’re caused by the SecItem API being so hard to use correctly, and I’d probably agree with you |-:

To add additionally that the key generated is also protected using the following access control attributes:

context = [[LAContext alloc] init];
secAccessControlRef = SecAccessControlCreateWithFlags(kCFAllocatorDefault,
                                                              kSecAttrAccessibleAfterFirstUnlock,
                                                              kSecAccessControlApplicationPassword,
                                                              &error);
NSData *data = [@"B_password" dataUsingEncoding:NSUTF8StringEncoding];
[context setCredential:data type:LACredentialTypeApplicationPassword];

CFDictionarySetValue(*attr, kSecUseAuthenticationContext, (__bridge const void *)context);
CFDictionarySetValue(*attr, kSecAttrAccessControl, secAccessControlRef);

It’s hard to say what’s going on here without seeing more of your code, but my experience is that problems like this:

  1. Our SDK then attempts to regenerate a new key for the same label and attribute to store it using SecItemAdd, but the system then reports errSecDuplicateItem at the key already exists.

are usually caused by folks misunderstanding how the keychain enforces item uniqueness. See the Uniqueness section of SecItem: Fundamentals, making sure to follow the link to Queries and the Uniqueness Constraints.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thank you for sharing. Unfortunately, what I foresee here is that the SecItemAdd does correctly report the errSecDuplicateItem as the key does indeed exist (it was created beforehand). Hence, I doubt the issue lies with the SecItemAdd. Our SDK has logic to create the key if the query fails.

However, it is my opinion that the issue as hand is the SecItemCopyMatching is incorrectly reporting that the key does not exist when it should.

I further amend the above code snippet that during the query, we do also provide the kSecUseAuthenticationContext to ensure that the key can be fetch. In the expected behaviour (which consistently occurs to the majority of devices, the key is correctly retrieved. It is unfortunately this subset where the key is not.

However, it is my opinion that the issue as hand is the SecItemCopyMatching is incorrectly reporting that the key does not exist when it should.

That’s certainly possible.

Did you have any luck reproducing this?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hi, back with an update.

Our customers have been able to reproduce the issue and we have been able to retrieve some logs. The error from SecItemCopyMatching is errSecAuthFailed. The error code usually points to and invalid password or invalidate context but we can confirm those are not the case in this scenario.

This issue is occurring during an OS upgrade to iOS 17 and the key becomes unrecoverable. We have received info as well that the issue is appears to happen only on older devices which upgrade to iOS 17 (iPhone 11 family, iPhone 12 family, iPhone XS, iPhone XR). Newer devices 13/14 do not report this issue at all.

Our customers have been able to reproduce the issue and we have been able to retrieve some logs

Please please please tell me that included a sysdiagnose log (-:

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

yes! I have sent it over :)

yes!

Cool.

I have sent it over

Sent it over where?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

via Feedback Assistant

Got it. Thanks.

For those reading along at home, I’ll be helping mervyn.ong in a different context.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hello, is it possible to keep us informed on that issue ? @mervyn.ong you said that the issue occur "during" upgrade : do you mean that you are starting your app during the installation phase of iOS 17 ? Or is it after the installation, reboot, ... that you observe the issue ?

Thanks in advance.

hello, did you manage to get a resolution for this issue? seeing similar issues.

@eskimo Where you able to identify anything with sysdiagnose log, with iOS 17 being rolled out, lots of customers are facing this issue. Especially with users iPhone 12 and less, when they migrates from 16 to 17 they are having this issue.

Can you please let us know is errSecDuplicateItem error is been thrown for any specific condition in iOS 17 which was not there in iOS 16 ?? This question is because we did not have this issue when user was using in IOS 16.

Waiting for your reply ??

As I mentioned above, I’ll be helping mervyn.ong in a different context. I expect I’ll post a summary of what we uncover when we’re done, but we’re far from done right now.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Accepted Answer

I wanted to pass along the resolution here.

This problem is caused by a bug in iOS 17 (FB13066335, r. 114525162). We believe that the bug is fixed in the currently seeding iOS 17.1b3.

The bug caused keychain items protected by .applicationPassword to become unreadable. However, it only affects items created by iOS 14 and earlier. That explains why only a small fraction of users were hitting it.

A wide variety of apps hit this problem because this technique was used by a popular third-party library.

With the fix in place, the keychain items will become readable again (assuming the app hasn’t done anything to ‘break’ them otherwise).

And, no, I can’t say when iOS 17.1 will be released (-: See tip 3 in Quinn’s Top Ten DevForums Tips.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Is there any updates on this issue? It seems like this issue is still occurring in iOS 17.1

Unable to retrieve keychain data intermittently.
 
 
Q