Is Core Data the right tool?

Question

Created May ’22

Replies 5

Boosts 0

Views 1.2k

Participants 5

My app generates piecemeal near terabyte data distributed over one hundred files. Due to RAM and internal HD limitations I must store to an external HD in piecemeal fashion. That is to say, append fresh records directly to an existing file without reading-in the whole file, appending, and writing it out again. Ultimately (when data generation ceases) I will check each file to ensure uniqueness of member data structures.

Is Core Data the tool I should use?

If not, how do I append directly to disk-files without first reading it in?

Boost

Answer 1

Claude31 OP

May ’22

If I understand correctly your question, you can append data to a file directly:

https://stackoverflow.com/questions/27327067/append-text-or-data-to-text-file-in-swift

0

Answer 2

G.U.T.explorer OP

May ’22

Thanks Claude31. Using the last entry in that link, I could make an array of FileHandleBuffer to handle the one hundred active files. However, it appears I must know the Data:Capacity:size ? Which I don't know.

0

Answer 3

ABeeThatsBusy OP

May ’22

Just answering your original question - Core data is not the right choice for this at all.

1

Answer 4

nk_kennedy OP

May ’22

I agree with ABeeThatsBusy. You definitely don't want to attempt to do anything like this with Core Data.

I don't know what a FileHandleBuffer is. Are you talking about a FileHandle class? Don't ever use that class. It can throw Objective-C exceptions. If you are managing terabytes of data, those would be a regular occurrence.

The lowest level C file descriptors can append data to an existing file. Many of Apple's higher-level convenience methods and classes assume that you will have all data in memory and every write will completely replace the existing file. Don't do that.

0

Answer 5

DTS Engineer OP

Apple

May ’22

Are you talking about a FileHandle class? Don't ever use that class. It can throw Objective-C exceptions.

The modern API for FileHandle no longer throws Objective-C exceptions. Contrast:

The read(upToCount:) method method with

The readData(ofLength:) method

Having said that, I generally agree with your sentiment that a low-level API is likely to be the right choice for this task. When working in Swift I recommend Swift System. You can then build your own abstraction layered on top of that.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0