`NSPersistentStore` metadata update synchronisation

Question

Created Nov ’20

Replies 4

Boosts 1

Views 1.1k

Participants 2

I'm attempting to synchronise an offline data store in core data with an online store.

To maintain the last synchronisation point for both data coming in to the local cache, and data going out, I'm using change tokens. An NSPersistentHistoryToken to track the last data that went out, and a server provided token for the data coming in.

In terms of the best place to store the tokens, it seems to me it would be in the local store (NSPersistentStore) itself – saving the incoming data and the token can then be tied into a neat save transaction; if there's problem saving the incoming data, the token won't be updated.

One way of achieving this would be to create a new Entity in my model to track the tokens, but it appears NSPersistentStoreCoordinator already has a tidy pair of meta data functions that might be perfect for this: metadata(for:) and setMetadata(_:for:).

However, as I am considering updating the metadata after operations on separate managed object contexts (one for incoming remote data, one for outgoing remote data) I'm concerned about synchronisation. That's because the metadata functions operate at the persistent store coordinator level rather than the context level.

Potentially, if two perform(_:) blocks are running concurrently the metadata could become stale between the calls to metadata(for:) and setMetadata(_:for:), what's more, as store meta data is apparently only persisted upon a context save we'd want to restrict another context getting access to this new metadata until it's been permanently persisted as otherwise it may write changes for another context that were ultimately rolled-back.

What's the reccommended approach to handle this? Is it as simple as wrapping the meta data fetch, update and context save calls in to an operation that gets called on a serialised queue? Something like:

Code Block let metadataSerialQueue = DispatchQueue("metadata-serial-queue")

Code Block context1.perform {
    // context update operations
    ...
    metadataSerialQueue.sync {
        var metadata = psc.metadata(for: store)
        // update change token
        metadata["change-token"] = changeToken
        psc.setMetadata(metadata, for: store)
        do {
            try context1.save()
        }
        catch let e as NSError {
            print("Failed to save context in store change. Rolling back. \(e)")
            context1.rollback()
        }
    }
}

Or is that pushing store metadata outside of its use case?

Boost

Answer 1

mzarra OP

Dec ’20

Short answer: Yes.

As part of your workflow when you are requesting an update from the server; you should update your change tokens. Normally, I would see a data request flow be something like:

Main: Send request to a background thread
BG: Request from Server
BG: Receive Payload
BG: Consume Payload
BG: Write to Disk
BG: Notify main thread
Main: Update UI

With your change tokens in the store, same thing:

Main: Send request to a background thread
BG: Fetch change tokens to send to server
BG: Request from Server with change tokens
BG: Receive Payload
BG: Consume Payload
BG: Update change tokens in metadata
BG: Write to Disk
BG: Notify main thread
Main: Update UI

Since the entire action of updating data from the server is a single flow, there is no risk of the token being out of sync of the request.

This is assuming you have one synchronous server request queue. If you are doing server requests in parallel then that will cause issues.

To have parallel update requests you would need to spin them as sub operations of a master operation. When all subs are complete then the master updates the change token with a final save of the context.

1

Answer 2

tcldr OP

Dec ’20

This is great – thank you.

One thing I'm still confused about though is the timing of saving store metadata. Specifically, NSPersistentStore's metadata(for:)/setMetadata(_:for:) method pair. In the docs for the setMetadata(_:for:) method, it states:

Important: Setting the metadata for a store does not change the information on disk until the store is actually saved.

This is the point that really isn't clear to me. As far as I'm aware, saving a context, is the only way to save a store – and therefore store metadata. But metadata(for:)/setMetadata(_:for:) don't indicate they are context aware – they operate at the PSC/Store level after all. So doesn't that mean that when any context is saved, it will persist changes to the store metadata that could have been made anywhere else?

If so, does this not create the possibility that if you have multiple contexts (even if just a background context and a view context), a save on one context may persist changes to your store metadata that were made during a performChanges block of another context?

For example, say I have your sequence processing on my background context:

Fetch change tokens to send to server
Request from Server with change tokens
Receive Payload
Consume Payload
Update change tokens in metadata
Write to Disk
Notify main thread

But, for some reason, at the 'write to disk' stage (no. 6) – we fail. So we rollback the changes on our BG context – and I assume we must now update the store metadata again with our previous change token.

Meanwhile however, our user has made an update on the view context and saves – right between points 5-6. If I understand how setMetadata(_:for:) works according to the quoted docs above, that would mean the save on the view context now persists the changes made at no. 5 even though it was made in a performChanges block on our BG context.

So, that means that if we happen to crash during the process of the rollback on our failed background update, the store will now contain incorrect metadata.

If so, I guess the solution here would to simply create an additional model entity for your metadata and use that instead of NSPersistentStore's metadata(for:)/setMetadata(_:for:) pair – but wanted to make sure I'm not re-inventing the wheel.

0

Answer 3

mzarra OP

Dec ’20

I suspect the confusion stems from what a NSManagedObjectContext actually is. A context is a scratchpad that allows you to store edits to objects that you may want to save in the future. If you were to dealloc a NSManagedObjectContext before saving it, all changes go away like they never happened, including any updates to the metadata for the store.

By seeing the context as a scratchpad, the metadata API documentation starts to make sense. When you update the metadata at the context level, that is a temporary change. When you save that context, you are telling the store to update its state based on what is in the context.

Committing the scratchpad to permanence.

Now, conflicts. If you were to change the same metadata key in two different contexts; you produced a race condition.

First save wins.

Second save fails with, I suspect, an interesting error.

0

Answer 4

tcldr OP

Dec ’20

If you were to dealloc a NSManagedObjectContext before saving it, all changes go away like they never happened, including any updates to the metadata for the store.

I'm not sure that's the case. I understand the role of a managed object context as a scratchpad for the store, and that with any NSManagedObjects owned by the context, this is certainly the case. But in the case of store metadata specifically i.e the dictionary that is retrieved and set by NSPersistentStore's metadata(for:)/setMetadata(_:for:) methods – this isn't reflected in the tests I've done.

I've experimented with adding a store metadata entry in one context, then calling rollback. If the store metadata was context sensitive you would expect the metadata entry to be now be gone. However, you can then go on to read the store metadata in another context and the pre-rollback changes are still there!

The strange part to me is that you have to call save on any context for the store metadata to persist, which I think creates the confusion.

0