MAJOR Core Data Issues with iOS 18 and sdk - Data Missing for many users?!

I just released an App update the didn't touch ANYTHING to do with Core Data (nothing changed in our Coredata code for at least 8 months). The update uses SDK for iOS 18 and Xcode 16.2 and the app now requires iOS 18 and was a minor bug patch and UI improvements for recent iOS changes.

Since the update we are getting a steady trickle of users on iOS 18, some who allow the App to store data in iCloud (Cloudkit) and others who do not, all reporting that after the update to our recent release ALL their data is gone?!

I had not seen this on ANY device until today when I asked a friend who uses the App if they had the issue and it turned out they did, so I hooked their device up to Xcode and ALL the data in the CoreData database was gone?! They are NOT using iCloud. There were no errors or exceptions on Xcode console but a below code returned NO records at all?!

Chart is custom entity and is defined as:

@interface Chart : NSManagedObject {}
let moc = pc.viewContext
let chartsFetch = NSFetchRequest<NSFetchRequestResult>(entityName:"Charts")     // Fetch all Charts
 do {
     let fetchedCharts = try moc.fetch(chartsFetch) as! [Chart]
     for chart in fetchedCharts {
         ....
      }
}

A break point inside the do on fetchedCharts show there are NO objects returned.

This is a serious issue and seems like an iOS 18 thing. I saw some people talking in here about NSFetchRequest issues with iOS 18. I need some guidance here from someone Apple engineer here who knows what the status of these NSFetchrequest bugs are and what possible workarounds are. Becasue this problem will grow for me as more users update to iOS 18.

Answered by DTS Engineer in 820458022

we DID open the CoreData store and found 0 (ZERO) Chart. entities where before the update there had been many?! Was that not clear from the explanation associated with the code snippet?

Yeah, I saw the fetch in your code snippet. What I'd like to confirm though is that the Core Data store you open has the right path. When your app is updated, the absolute path to the app's container ("Application_Home") may change. (See the “Managing Data Across Updates” section in Testing iOS App Updates.) If you rely on the absolute path in some way to manage your Core Data store URL, it may turn out that you create and open a new empty store in somewhere else, while the original store is still there in your app's container.

The other point I believe is worth mentioning: If your app implements the logic to create a new (empty) store when hitting a Core Data failure, you might run into the issue as well. A Core Data failure can happen when the store is data-protected or corrupted.

If your app tries to access the Core Data store when running in the background and the store is data-protected, you will likely hit a SQLITE_AUTH error. See Encrypting Your App’s Files for more information on data protection.

Regarding corrupted stores, here are several cases I can think of:

a. You missed the -wal file when copying the database using file system APIs.

b. Your app is suspended or killed when Core Data is accessing the store. This can happen when your app is suspended (because of running out of the background execution time) while accessing a Core Data store. In this case, you will likely get a crash with the exception code 0xdead10cc before hitting a corrupted store, indicating that your app “has been terminated by the OS because it held on to a file lock or sqlite database lock during suspension.” See TN2151 for more details.

c. Lightweight migration process is re-entered. Depending on your data set, a Core Data lightweight migration can take long time. If you access your Core Data stack from a different thread while a migration process is ongoing, you can trigger another migration on the same store, which may lead to a corrupted store.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.

What you described is indeed a disaster, but it is hard to provide meaningful comments simply based on the symptom that the data was lost.

The first thing I'd check is if the Core Data store was indeed gone. You can do so by retrieving a Core Data store in your app’s container from an iOS device and checking if there is any data there. See here for more details.

It may also help if you can share the exact steps you used to trigger the issue, and why you infer that the issue is related to iOS 18 updating.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.

Hi, but in the original question, I mentioned that on the 1 device we have seen with the issue, we DID open the CoreData store and found 0 (ZERO) Chart. entities where before the update there had been many?! Was that not clear from the explanation associated with the code snippet?

Anyway, we did this from within a Utility class we wrote several years ago in Swift to manage initialization of CoreData, de-duping and other housekeeping functions.

On the device exhibiting the missing data, we put a break point in a utility function that fetches ALL the Chart entity records and lists a few fields from each. However on this device, the fetch returned nothing and ALL the data appears to be gone?!

We saw this using the code snippet example in the original question, where a break point after the fetch showed fetchedCharts to have zero returned objects.

This seems to be randomly happening to a growing group of people. But we are seeing NO exceptions of any kind from CoreData.

I saw an earlier report in the forum of developers back in Nov. 2024 running into some kind of bug where NSFetchRequest in Swift has a bug and returns no data. Was this issue fixed yet?

we DID open the CoreData store and found 0 (ZERO) Chart. entities where before the update there had been many?! Was that not clear from the explanation associated with the code snippet?

Yeah, I saw the fetch in your code snippet. What I'd like to confirm though is that the Core Data store you open has the right path. When your app is updated, the absolute path to the app's container ("Application_Home") may change. (See the “Managing Data Across Updates” section in Testing iOS App Updates.) If you rely on the absolute path in some way to manage your Core Data store URL, it may turn out that you create and open a new empty store in somewhere else, while the original store is still there in your app's container.

The other point I believe is worth mentioning: If your app implements the logic to create a new (empty) store when hitting a Core Data failure, you might run into the issue as well. A Core Data failure can happen when the store is data-protected or corrupted.

If your app tries to access the Core Data store when running in the background and the store is data-protected, you will likely hit a SQLITE_AUTH error. See Encrypting Your App’s Files for more information on data protection.

Regarding corrupted stores, here are several cases I can think of:

a. You missed the -wal file when copying the database using file system APIs.

b. Your app is suspended or killed when Core Data is accessing the store. This can happen when your app is suspended (because of running out of the background execution time) while accessing a Core Data store. In this case, you will likely get a crash with the exception code 0xdead10cc before hitting a corrupted store, indicating that your app “has been terminated by the OS because it held on to a file lock or sqlite database lock during suspension.” See TN2151 for more details.

c. Lightweight migration process is re-entered. Depending on your data set, a Core Data lightweight migration can take long time. If you access your Core Data stack from a different thread while a migration process is ongoing, you can trigger another migration on the same store, which may lead to a corrupted store.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.

Hi,

Thanks for your response. I don't think we do any of the things you are concerned about.

Here is some of the code I use to set up the CoreData/Cloudkit stack and have been using this for a few years now. Nothing changed here in at least 12 months (I am not exactly a piker in this realm :) :

class CoreDataUtil: NSObject
{
    @objc var pc = NSPersistentCloudKitContainer(name:"iphemeris")
    private lazy var historyRequestQueue = DispatchQueue(label:"history")
    
    let coreDataLogSubsystem = "com.iPhemeris.coreData"
    let coreDataLogger = Logger(subsystem:"com.iPhemeris.coreData", category:"Cloudkit")
    
    @objc var useiCloud:Bool = userSettings.bool(forKey:UseIcloudKey) {
        didSet {
            userSettings.set(useiCloud, forKey:UseIcloudKey)
            if(pc.persistentStoreCoordinator.persistentStores.count > 0)
            {
                let store = pc.persistentStoreCoordinator.persistentStores[0]
                do {
                    try pc.persistentStoreCoordinator.remove(store)
                }
                catch {
                    coreDataLogger.log("*** CoreDataUtil toggle use iCLoud ON/OFF FAILED: \(error, privacy:.public)")
                }
            }
            self.initCoreDataStack()
        }
    }
    
    @objc func initCoreDataStack()
    {
        coreDataLogger.log("*** Initialize CoreData Stack - Start")
        
        guard let description = pc.persistentStoreDescriptions.first else {
            fatalError("*** CoreDataUtil - could not find persistent store description.")
        }
        description.setOption(true as NSNumber, forKey:NSPersistentHistoryTrackingKey)
        description.setOption(true as NSNumber, forKey:NSPersistentStoreRemoteChangeNotificationPostOptionKey)
        description.setOption(true as NSNumber, forKey:NSMigratePersistentStoresAutomaticallyOption)
        description.setOption(true as NSNumber, forKey:NSInferMappingModelAutomaticallyOption)

        if(useiCloud == true) {
            coreDataLogger.log("*** User is using iCloud")
            let cloudKitContainerIdentifier = "iCloud.cribaudo.iphemeris"
            let options = NSPersistentCloudKitContainerOptions(containerIdentifier:cloudKitContainerIdentifier)
            description.cloudKitContainerOptions = options
        }
        else {
            coreDataLogger.log("*** User is NOT using iCloud")
            description.cloudKitContainerOptions = nil
        }
        
        pc.persistentStoreDescriptions.first?.url = storeURL()
        pc.viewContext.automaticallyMergesChangesFromParent = true
        pc.loadPersistentStores { _, error in
            if let error = error as NSError? {
                self.coreDataLogger.error("*** loadPersistentStore ERROR: \(error, privacy:.public), \(error.userInfo, privacy:.public)")
            }
#if DEBUG
            // Can be called every time but is needed after model changes
            //self.initializeCloudKitSchemaForNewModel()
#endif
            self.pc.viewContext.automaticallyMergesChangesFromParent = true
            
            // De-dupe old records out of Database.
            let uuidFixed = NSUbiquitousKeyValueStore.default.object(forKey:CoreDataUUIDNullAndDuplicatesFixed)
            if(uuidFixed == nil) {
                self.coreDataLogger.log("*** User does not have uuidFixed Set.")
                
                // OLD DeDupe Code - probably can delete soon.
                // Run UUID De-dupe if it was not yet run for this users account
                self.findAndfixDuplicateUUID()
            }
            else {
                self.coreDataLogger.log("*** User has uuidFixed Set.")
                
                self.deDupeMigratedData()
                self.findAndfixMissingFirstChar()
            }
            // Notify everyone the data is ready
            self.sendMOCReadyNotificationForPlatform()
        
            self.coreDataLogger.log("*** Current CoreData Model Version -> \(self.pc.managedObjectModel.versionIdentifiers.description, privacy:.public)")
            //self.listAllChartUUID()
        }
        // Try to get some error notification of iCloud Storage quota exceeded?
        NotificationCenter.default.addObserver(self, selector:#selector(processCloudKitErrors), name:NSPersistentCloudKitContainer.eventChangedNotification, object:nil)
        
        coreDataLogger.log("*** Initialize CoreData Stack - End")
    }
    
    func initializeCloudKitSchemaForNewModel()
    {
        coreDataLogger.log("*** Initialize CloudKit Scheme for New Model - Start")
        if(!useiCloud) { return }
        do {
            // Use the container to initialize the development schema.
            let options = NSPersistentCloudKitContainerSchemaInitializationOptions()
            try pc.initializeCloudKitSchema(options:options)
        }
        catch {
            coreDataLogger.error("*** CoreDataUtil Initializing new schema: \(error, privacy:.public)")
        }
        coreDataLogger.log("*** Initialize CloudKit Scheme for New Model - End")
    }
    
    func storeOptions() -> [AnyHashable:Any]
    {
        return [
            NSMigratePersistentStoresAutomaticallyOption:true,
            NSInferMappingModelAutomaticallyOption:true
        ]
    }
    
    func storeURL() -> URL
    {
        let urls = FileManager.default.urls(for:.documentDirectory, in:.userDomainMask)
        guard let docURL = urls.last else {
            fatalError("*** CoreDataUtil RW: StoreURL Error fetching document directory")
        }
        let storeURL = docURL.appendingPathComponent("iPhemeris.sqlite")
        return storeURL
    }
    

sdds

    @objc var useiCloud:Bool = userSettings.bool(forKey:UseIcloudKey) {
        didSet {
            userSettings.set(useiCloud, forKey:UseIcloudKey)
            if(pc.persistentStoreCoordinator.persistentStores.count > 0)
            {
                let store = pc.persistentStoreCoordinator.persistentStores[0]
                do {
                    try pc.persistentStoreCoordinator.remove(store)
                }
                catch {
                    coreDataLogger.log("*** CoreDataUtil toggle use iCLoud ON/OFF FAILED: \(error, privacy:.public)")
                }
            }
            self.initCoreDataStack()
        }
    }

I have no evidence to say anything for sure, but I am a bit worrying about what would happen if a Core Data save happens while the store is being removed. To turn on and off iCloud, folks typically release everything, including the persistent container and all Core Data objects the app holds, and then create a new Core Data stack.

And just FYI, regarding the in-app toggle to enable / disable CloudKit synchronization, you might want to take a look this discussion, if not yet.

I don't think we do any of the things you are concerned about.

In that case, I think I’ve reached the limit of what I can do here :-(.

To move forward, I guess you will probably need to find a reproducible case, which I know is not easy, or look into a sysdiagnose captured from a device that the issue just happened to hopefully find a hint.

I saw an earlier report in the forum of developers back in Nov. 2024 running into some kind of bug where NSFetchRequest in Swift has a bug and returns no data. Was this issue fixed yet?

If you can provide a feedback report ID, or a link that contains the details of the issue, I would take a closer look.

As I mentioned in my first post, you can try to download the app's container, find the data store, and check if the data is there – If the store has the data and NSFetchRequest returns nothing, you confirm that the issue is related to NSFetchRequest, and might be able to create a reproducible case from there.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.

Thank you for your response.

  1. The thread I saw about NSFetch being broken in iOS 18 was this one wherein you are the engineer that states that NSFetch is broken back in Sep/Nov 2024 time frame. No Bug Report # was referenced in that thread but perhaps this link will help you find it. This was the first thing I saw when I started researching what might be going on with my App:

NSFetch Broken Discussion

  1. I don't think our issue is users disabling iCloud right before App launch, unless iOS 18 is somehow doing that during update to OS or App? I didn't think the container moved when you set cloudKitContainerOptions=nil. I thought the store stayed pretty much where it was on the device? But I would like to try your suggestion relative to downloading the store and seeing if data is in it but NSFetch Returns nothing. I am not clear on how to do that. How do we get the store and check if the "store has data" without doing an NSFetch? I was not able to find a way to do this and if the problem is with NSFetch then we are sort of in a Catch 22. Any thoughts or suggestions?

  2. I saw your other discussion a few days ago relative to giving users an In-App way to turn on/off Cloudkit. However, another Apple Engineer a while back (see link to forum thread below) stated it was ok to do this by setting cloudKitContainerOptions=nil and that it was relatively harmless?! Has something changed? Based on that forum discussion we implemented what you see in our code. So far it has proved harmless as far as we know and never have we seen any harmful results from it. Here is that thread: OK to diasble/enable iCloud Forum Discussion

  3. In past Apple recommended we ask permission to use iCloud. This was quite a few years back. Has this recommendation changed? Given all the users that lose their data because they do not turn it on and then delete the app without a backup and lose the data container, we are thinking to NOT ask and just always enable it. But there are a lot of issues with that. What do we do when the user disables it from iCloud Settings? Of for users that have a full iCloud, typically choked with pictures? Or for users that have a real issue with iCloud (a lot of those in Europe for some reason). But even if I disregarded all that I am not really sure how to transition to this, I guess we could just hide the switch and enable it for all but what happens when their iCloud is full.

The thread I saw about NSFetch being broken in iOS 18 was this one wherein you are the engineer that states that NSFetch is broken back in Sep/Nov 2024 time frame. ... NSFetch Broken Discussion

Oh, that was related to fetched property, which is totally different from NSFetchRequest.

But I would like to try your suggestion relative to downloading the store and seeing if data is in it but NSFetch Returns nothing. I am not clear on how to do that. How do we get the store and check if the "store has data" without doing an NSFetch? I was not able to find a way to do this and if the problem is with NSFetch then we are sort of in a Catch 22. Any thoughts or suggestions?

You can try the following steps:

  1. Grab the app container from the device that demonstrates the issue. As mentioned, this is detailed here.
  2. Find your Core Data store from the app container.
  3. Use Terminal.app + sqlite3 on your Mac to examine the data. You can search Internet for how to use the command line tool, if needed.

another Apple Engineer a while back (see link to forum thread below) stated it was ok to do this by setting cloudKitContainerOptions=nil and that it was relatively harmless?

Technically, it has always been the case, and nothing has changed since then. However, the concern is about data privacy and security, as I laid out in my post, and not about technical difficulty.

To be clear, my post also mentioned that the topic is controversial and that some folks have built their business model based on the technique. After all, it's up to you to decide if the mentioned data privacy and security concern matters to you.

Given all the users that lose their data because they do not turn it on and then delete the app without a backup and lose the data container, we are thinking to NOT ask and just always enable it.

The iCloud setting is a user-level setting. There is no API for apps to change the setting programmatically.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.

Thanks for your response. I will have to see if I can get my hands on a container from a device having the issue and before they start messing with it by saving new data.

Is there a way to get the container from a device NOT having Xcode or Development tools if they can connect to a Mac?

The issue is definitely real for a small percentage of devices that are mostly overseas in Spain or Italy. Not sure why that is a factor but most are not in USA and not english speaking?

I see from Firebase analytics that there are already over 10,000 user running the update on iOS 18 and having the latest build. We are getting a steady stream of 2 or 3 complaints a day. Many of whom DO NOT use iCloud which is also weird. A few using iCloud but most are NOT. Any thoughts on why that might be?

Is there a way to get the container from a device NOT having Xcode or Development tools if they can connect to a Mac?

Not that I am aware of.

Many of whom DO NOT use iCloud which is also weird. A few using iCloud but most are NOT. Any thoughts on why that might be?

If the impacted users didn't use iCloud, you can rule out CloudKit, which will simplify the issue a bit. However, I still can't say anything for sure – Debugging a random data loss can be hard and my sitting here and guessing won't be helpful. To eventually figure it out, you might consider reaching out the impacted users to hopefully find more hints, or ideally, create a reproducible case.

now it is almost as if a user has NSPersistentCloudKitContainer options set to nil and the data is deleted. Can you confirm or provide a link to documentation confirming?

Setting the CloudKit option to nil does not delete the data in the existing Core Data store.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.

Ziqiao are you aware of any behavioral changes in iOS 18 that causes an NSPersistentCloudKitContainer to delete data? In past we were told it could be used in all cases with history on, even if user was not allowing cloudkit. This was true until iOS 18 it seems now if that initializes and user is not using iCloud would it delete data. I thought I was seeing people talking about that and that it does that for security. if That is true that is a huge damn change and which Apple should tell people about?!

are you aware of any behavioral changes in iOS 18 that causes an NSPersistentCloudKitContainer to delete data?

Not that I am aware of.

To be clear, NSPersistentCloudKitContainer does clean the data on the device if it gets notified of a change on the current logged-in Apple ID, but that has been the as-designed behavior since the very beginning. Note that it only cleans the local data; the data on CloudKit is still there, and will be synchronized back when the user logs back in with the same account, and so no data loss will happen.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.

Well something changed since we switched to iOS 18 SDK and it definitely did not work this way in earlier versions of iOS. At some point after iOS 15 SDK something changed and the "cleaning" process you mentioned is literally wiping users data and if it was never in iCloud in the first place it is totally gone now?!

We have always used an NSPersistentCloudKitContainer since we first implemented it years back regardless of if the person enabled iCloud (in our App or not). This was based on statements Apple Engineers made in the past that you could always use NSPersistentCloudKitContainer since it was subclass of nspersistentcontainer and with History on but options set to nil for NOT ALLOWED.

But now it seems that users running iOS 18 and who did not allow us to use iCloud (are getting data wiped). It is not clear if they are entirely logged out of iCloud or just have iCloud Drive off or what, but something is different in iOS 18. I really need a way to zero in on what is going on.

This is one of those catch 22 situations where by the time we see the issue the data is gone and we do not know what transpired to delete it or if anything actually changed and instead it was some kid of change of behvaior change in iCloud.

MAJOR Core Data Issues with iOS 18 and sdk - Data Missing for many users?!
 
 
Q