CKSyncEngine API design problems and maintenance status inside Apple?

This is a real issue, if we only know what record IDs to send but don't know which batches have previously finished uploading (so we don't know which fields were changed during previous upload process), we have to re-upload all fields in nextRecordZoneChangeBatch again and again.

Say we have a ImagePost, and user only edited the title a few times, and we have to re-upload every CKAsset associated with it. That could be hundreds of MBs for every record.

Take CoreData as a concrete example. It is not that realistic and easy to track changed fields in CoreData item itself. A CoreData save may fail, or conflict with savings from other threads, the conflict solving logic is hard to write and error prone if we are to record changes in database item itself. In order to solve this, we may:

• We only mark it as having changes before a save, and only do full re-upload of an item if the having changes mark still exist after app launches.
• Or we can use history transactions records to figure out what fields changed, and if we use it, we have to keep all history transactions and never delete any.

We need to know which batch finished, or we cannot use those methods.

The CKRecord we get from CKRecord(coder: unarchiver) is not a fully materialized one, it only contains necessary systemFields, so it cannot figure out and upload only changedKeys. I wonder want can I do to workaround this.

Follow up

If the engine works serially and do not ask for next batch before user finished handleEvent, the problem is solve and I had a misunderstanding.

CloudKit doesn't provide anything for you to tell which batch a change is tied to. That is an internal implementation detail that doesn't matter to developers.

Regarding your following concern:

How do I decide if i have to mark "***" as finished uploading or not? If I mark *** as finished that's wrong, the new name "zzz" is not uploaded. I have to compare every field of *** with the savedRecord to decide if I finished uploading or not? That is not acceptable as the performance and memory will be bad.

Is there any strong reason that you need to mark a record as "finished uploading"?

Considering the following synchronization process when using CKSynEngine, it doesn't seem like you need to do that:

1. You make a change A.
2. You tell CKSynEngine by adding a pending record zone change for A.
3. You make another change B.
4. You tell CKSynEngine by adding another pending record zone change for B.
5. At some point, nextRecordZoneChangeBatch(_:syncEngine:) is triggered, and you return a batch of CloudKit records.
6. At some point, SentRecordZoneChanges is triggered. You grab the server record from the event, and merge it with the current local version.
7. If the local version is the winner in the conflict resolution process, add a new pending change.

In this process:

• At step 5, you can analyze and consolidate the pending changes. In this case, the change created at step 2 may be redundant, if it is still in the queue.
• After merging the record at step 6, you can analyze and consolidate the pending changes as well.

At step 6, if the server and local records have conflicts, you might need extra information to resolve them. In that case, you can add the information to the CloudKit record type. The info will then be available in the server record for you to examine. The sample-cloudkit-sync-engine sample demonstrates how to use userModificationDate to help resolve conflicts.

If the engine works serially and do not ask for next batch before user finished handleEvent, the problem is solve and I had a misunderstanding.

The pending changes are processed serially in the order they are added.

However, I won't assume that one nextRecordZoneChangeBatch(_:syncEngine:) is immediately followed by one SentRecordZoneChanges. Instead, I believe that nextRecordZoneChangeBatch(_:syncEngine:) can be triggered multiple times before SentRecordZoneChanges is triggered.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.

Thank you for your response. @DTS Engineer

In the sample code, it works fine because populateRecord(_:) always uploads all keys of a record. However, in a production application, we often only want to upload changed keys to save network overhead and memory usage.

To do that efficiently, we need to know which version of a record was last successfully uploaded, so we can figure out which keys have changed since then.

For example, suppose we have a Core Data entity “Item” with the following property changes:

let instance = Item("Item-1")

• Change A: instance.x is updated.
• Change B: instance.y is updated.
• Change C: instance.x and instance.z are updated.

If Change C happens after I tell the sync engine about A and B (between 5 and 6, before those changes finish uploading), the engine will eventually call nextRecordZoneChangeBatch(_:) again for “Item-1.” At that point, I need to determine which fields are newly changed and only populate those.

Without a batch identifier, we don’t know exactly which subset of keys got uploaded in the previous batch and which ones still need updating. This forces us to re-upload everything every time.

Suppose:

• Change A: instance.x is updated. (batch 1)
• Change B: instance.y is updated. (batch 2)
• handleEvent — but is it for batch 1, batch 2, or both?
• Change C: instance.x and instance.z are updated. (batch 3)

If handleEvent is coalesced so that I only receive it once batch 1 and 2 are definitively finished, I know exactly which keys made it up to the server and can upload only [x, z], not [x, y, z] for batch 3.

If handleEvent is NOT coalesced and if batch 2 eventually fails later, and I never find out which batch failed, then I wouldn’t know that key [y] actually needs re-uploading. This is why I need to identify which batch finished or failed.

@DTS Engineer Hi, do we have any updates?

@DTS Engineer

I believe the state property should not be used, and should be removed. Right after calling add(pendingChanges:), but before that new state is actually persisted, the app could crash, run out of memory, be force-quit, encounter a kernel panic, or lose power. The pending changes are lost, making the engine unable to synchronize properly on the next launch.

Database changes is like a stream of events, and I would like to track where we were, so we can pick up from where we left. The current design, however, makes us unable to do that.

Because the engine may or may not combine changes of the same CKRecord, and in the handle event callback, we don't have any information about this. The record may be in pending state, and if I add more pending changes to the same record, it may be combined and send in one network request, so I only receive ONE callback for it. But the record may be already in sending state, and if I add more pending changes to the same record, it will make another network request, and I will receive TWO callbacks for it. So I cannot decide the current progress in the callback. When the app launches again, I cannot pick up from where I left off.

I talked to the CKSyncEngine folks and confirmed that, assuming I have the following batches:

• Change A in batch 1
• Change B in batch 2

In today's implementation, the sequence will be the following:

1. nextRecordZoneChangeBatch for batch 1
2. SentRecordZoneChanges with change A
3. nextRecordZoneChangeBatch for batch 2
4. SentRecordZoneChanges with change B

However, the behavior is not formally documented, and may be subject to change in the future. I think it is probably safer to not rely on the assumption, and just go ahead to figure out the difference between the server record and the local one when getting SentRecordZoneChanges, and add it as a pending change to sync with the server.

Best,
——
Ziqiao Chen
 Worldwide Developer Relations.