Is Core Data the right tool?

My app generates piecemeal near terabyte data distributed over one hundred files. Due to RAM and internal HD limitations I must store to an external HD in piecemeal fashion. That is to say, append fresh records directly to an existing file without reading-in the whole file, appending, and writing it out again. Ultimately (when data generation ceases) I will check each file to ensure uniqueness of member data structures.

Is Core Data the tool I should use?

If not, how do I append directly to disk-files without first reading it in?

Up vote post of G.U.T.explorer Down vote post of G.U.T.explorer
962 views
Posted by
G.U.T.explorer
Reply
Add a Comment

Replies

If I understand correctly your question, you can append data to a file directly:

https://stackoverflow.com/questions/27327067/append-text-or-data-to-text-file-in-swift

Posted by
Claude31
Up vote reply of Claude31 Down vote reply of Claude31
Add a Comment

Thanks Claude31. Using the last entry in that link, I could make an array of FileHandleBuffer to handle the one hundred active files. However, it appears I must know the Data:Capacity:size ? Which I don't know.

Posted by
G.U.T.explorer
Up vote reply of G.U.T.explorer Down vote reply of G.U.T.explorer

  • Actually, since the size of the structures ARE known if they are populated, i.e, they are a class instance with optional variables, I could use that full size. It may be inefficient, disk wise, but is the best option so far.

    —
    G.U.T.explorer

  • Nope, my class has optional arrays of unknown count.

    —
    G.U.T.explorer
Add a Comment

Just answering your original question - Core data is not the right choice for this at all.

Posted by
ABeeThatsBusy
Post not yet marked as solved Up vote reply of ABeeThatsBusy Down vote reply of ABeeThatsBusy
Add a Comment

I agree with ABeeThatsBusy. You definitely don't want to attempt to do anything like this with Core Data.

I don't know what a FileHandleBuffer is. Are you talking about a FileHandle class? Don't ever use that class. It can throw Objective-C exceptions. If you are managing terabytes of data, those would be a regular occurrence.

The lowest level C file descriptors can append data to an existing file. Many of Apple's higher-level convenience methods and classes assume that you will have all data in memory and every write will completely replace the existing file. Don't do that.

Posted by
nk_kennedy
Up vote reply of nk_kennedy Down vote reply of nk_kennedy
Add a Comment

Are you talking about a FileHandle class? Don't ever use that class. It can throw Objective-C exceptions.

The modern API for FileHandle no longer throws Objective-C exceptions. Contrast:

Having said that, I generally agree with your sentiment that a low-level API is likely to be the right choice for this task. When working in Swift I recommend Swift System. You can then build your own abstraction layered on top of that.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Posted by
eskimo
Up vote reply of eskimo Down vote reply of eskimo
Add a Comment