Hello everyone
II'm making iOS app with lot of huge tables, which is stored in plists. One XML file is about half of megabyte
Ill need about 6-10 such tables
I just wander, is it right approach to use plists for storing big tables, of I should use another way?
What issues I can get with such huge plists?
Thanks for advice!
john daniel wrote:
I think most people take up more space than that for icons on buttons.
Right. But this will be considerably bigger if you put it in an XML plist.
@DenisO, For a rectangular array like this I’d probably put the data into a file in binary form and then memory map that. You can then access the values at random and the VM subsystem will take care of the rest.
Furthermore, I’d wrap this in a nice, Swift-friendly data structure. Something like this:
class DataMap {
let rowCount = 16
let columnCount = 16
init?(file: URL) {
guard let d = try? NSData(contentsOf: file, options: [.alwaysMapped]) else {
return nil
}
guard d.length == (self.rowCount * self.columnCount * MemoryLayout<Double>.size) else {
return nil
}
self.data = d
let base = d.bytes.assumingMemoryBound(to: Double.self)
self.buffer = UnsafeBufferPointer(start: base, count: d.length)
}
private let data: NSData
private let buffer: UnsafeBufferPointer<Double>
subscript (row: Int, column: Int) -> Double {
precondition((0..<self.rowCount).contains(row))
precondition((0..<self.columnCount).contains(column))
return self.buffer[row * self.rowCount + column]
}
}Note that I’ve hard-coded the row and column counts, which makes sense if all of these objects have the same dimensions. If not, those values would need to be parameters to the initialiser.
Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware
let myEmail = "eskimo" + "1" + "@apple.com"I just wander, is it right approach to use plists for storing big tables, of I should use another way?
No, definitely not. Property lists are designed as a convenience mechanism for dealing with (relatively) small data sets. While things will probably work with a megabyte property list, there are definitely better options.
Is this a read-only data set bundled in your app? Or something you need to read and write?
Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware
let myEmail = "eskimo" + "1" + "@apple.com"I've gotten away with some crazy large read-only plist data sources, but usually they represented one table each. After a while, my issue became more of maintenance than deployment or performance, and moving away from a simple plist would have righted that ship, I'm sure.
If you have legitimately large data sets, and the skills to apply more comprehensive methods, I'd go that route.
Yes, there is a read-only data sets.
So, what you suggest to replace plists?
Btw, I have another one app, for iPad, with plist. There only one array plist used, with about 100 dictionaries, every dictionary contains about 20-30 records, some of them pretty big (1000+ symbols string). Is it too big as well?
Yes, there is a read-only data sets.
OK. That’s important because dealing with read/write data is a whole different kettle of fish.
So, what you suggest to replace plists?
That depends on a bunch of other factors. What does this data look like? How are you accessing it? And do you need simultaneous access to all of your tables? Or just one table at a time?
When dealing with large data sets the goạl is to avoid bringing all of the data in memory. There’s three basic strategies for this:
Streaming
Exploit the VM system
Implement an object load > use > unload lifecycle
If your access pattern is linear, it makes sense to stream through the data with file system APIs. The canonical example of this is movie playback.
If the data will fit in virtual memory but is too big for physical memory, you might compile the data to a format that’s can be used directly and then memory map that file. The VM system will then take care of loading data as you access it, and unloading data when the system needs to recover physical memory.
In more complex cases it might make sense to use a database-style design. Both SQLite and Core Data work well for this. These support random access and complex queries.
Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware
let myEmail = "eskimo" + "1" + "@apple.com"Its tables with numbers in format 00.00 or 0.00000
200-300 rows
about 100 columns
It’s a reference table, to get correct value by column and row name
As an example, here is multiply table
\ 1 2 3
1 1 2 3
2 2 4 3
3 3 6 9
So, basically, to calculate some values, I need to get reference value from one or two tables.
For example
x = (a+b+c)/(get value from table A for 2*3)
Look into Apple's Core Data - which is basically a (non-trivial) data model abstraction layer over your data storage - allows you to manage relationships and fetching rules between different data entities, over large and complex data sets.
https://developer.apple.com/reference/coredata
Behind the Core Data layer, your data storage could be sqlite db, or binary object.
Is the table rectangular? That is, does each row have the same number of columns? And vice versa?
Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware
let myEmail = "eskimo" + "1" + "@apple.com"Yes, it is
Each row have the same number of columns, and vice versa, yes.
real one will be similar with this - https://sanctusaura.files.wordpress.com/2010/06/relative-humidity2.jpg
300 (rows) * 100 (columns) * 8 (size of double) = 240000 bytes = 234K.
I think most people take up more space than that for icons on buttons.
Good point 🙂
Replied yesterday, but it still on moderation because of link to picture with table example.
So yes, all tables is rectangular, and each row have the same number of columns, and vice versa, yes.
john daniel wrote:
I think most people take up more space than that for icons on buttons.
Right. But this will be considerably bigger if you put it in an XML plist.
@DenisO, For a rectangular array like this I’d probably put the data into a file in binary form and then memory map that. You can then access the values at random and the VM subsystem will take care of the rest.
Furthermore, I’d wrap this in a nice, Swift-friendly data structure. Something like this:
class DataMap {
let rowCount = 16
let columnCount = 16
init?(file: URL) {
guard let d = try? NSData(contentsOf: file, options: [.alwaysMapped]) else {
return nil
}
guard d.length == (self.rowCount * self.columnCount * MemoryLayout<Double>.size) else {
return nil
}
self.data = d
let base = d.bytes.assumingMemoryBound(to: Double.self)
self.buffer = UnsafeBufferPointer(start: base, count: d.length)
}
private let data: NSData
private let buffer: UnsafeBufferPointer<Double>
subscript (row: Int, column: Int) -> Double {
precondition((0..<self.rowCount).contains(row))
precondition((0..<self.columnCount).contains(column))
return self.buffer[row * self.rowCount + column]
}
}Note that I’ve hard-coded the row and column counts, which makes sense if all of these objects have the same dimensions. If not, those values would need to be parameters to the initialiser.
Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware
let myEmail = "eskimo" + "1" + "@apple.com"Thanks!
Will try this out
But, will try Core Data first 😉
I was thinking of the space required in memory rather than on disk.
Other options, like a binary file, SQLite database, or Core Data will certainly work, but each carries its own set of trade offs. picking the best format depends on how the data will eventually be used and/or how often the original data needs to be updated.
Done.
It works now.
Some main problems had to solve with CoreData:
1) Use pre-populated "DB" files in app
2) Make it work with iOS 8
3) Tranfser data from Excel to CSV, with table structure change (got 5000+ rows), and then, with separate app, to SQLite files
Now, need to polish and simplify fetch code
Thanks everyone!
You helped me a lot