18 Replies
      Latest reply: Feb 15, 2017 12:02 AM by DenisO RSS
      DenisO Level 1 Level 1 (0 points)

        Hello everyone

         

        II'm making iOS app with lot of huge tables, which is stored in plists. One XML file is about half of megabyte

        Ill need about 6-10 such tables

         

        I just wander, is it right approach to use plists for storing big tables, of I should use another way?

        What issues I can get with such huge plists?

         

         

        Thanks for advice!

        • Re: Plist max size
          eskimo Apple Staff Apple Staff (6,660 points)

          I just wander, is it right approach to use plists for storing big tables, of I should use another way?

          No, definitely not.  Property lists are designed as a convenience mechanism for dealing with (relatively) small data sets.  While things will probably work with a megabyte property list, there are definitely better options.

          Is this a read-only data set bundled in your app?  Or something you need to read and write?

          Share and Enjoy

          Quinn “The Eskimo!”
          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
          let myEmail = "eskimo" + "1" + "@apple.com"

            • Re: Plist max size
              DenisO Level 1 Level 1 (0 points)

              Yes, there is a read-only data sets.

              So, what you suggest to replace plists?

              Btw, I have another one app, for iPad, with plist. There only one array plist used, with about 100 dictionaries, every dictionary contains about 20-30 records, some of them pretty big (1000+ symbols string). Is it too big as well?

              • Re: Plist max size
                eskimo Apple Staff Apple Staff (6,660 points)

                Yes, there is a read-only data sets.

                OK.  That’s important because dealing with read/write data is a whole different kettle of fish.

                So, what you suggest to replace plists?

                That depends on a bunch of other factors.  What does this data look like?  How are you accessing it?  And do you need simultaneous access to all of your tables?  Or just one table at a time?

                When dealing with large data sets the goạl is to avoid bringing all of the data in memory.  There’s three basic strategies for this:

                • Streaming

                • Exploit the VM system

                • Implement an object load > use > unload lifecycle

                If your access pattern is linear, it makes sense to stream through the data with file system APIs.  The canonical example of this is movie playback.

                If the data will fit in virtual memory but is too big for physical memory, you might compile the data to a format that’s can be used directly and then memory map that file.  The VM system will then take care of loading data as you access it, and unloading data when the system needs to recover physical memory.

                In more complex cases it might make sense to use a database-style design.  Both SQLite and Core Data work well for this.  These support random access and complex queries.

                Share and Enjoy

                Quinn “The Eskimo!”
                Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                let myEmail = "eskimo" + "1" + "@apple.com"

                  • Re: Plist max size
                    DenisO Level 1 Level 1 (0 points)

                    Its tables with numbers in format 00.00 or 0.00000
                    200-300 rows
                    about 100 columns 
                    It’s a reference table, to get correct value by column and row name
                    As an example, here is multiply table
                    \  1 2 3
                    1 1 2 3
                    2 2 4 3
                    3 3 6 9

                    So, basically, to calculate some values, I need to get reference value from one or two tables.
                    For example
                    x = (a+b+c)/(get value from table A for 2*3)

                      • Re: Plist max size
                        eskimo Apple Staff Apple Staff (6,660 points)

                        Is the table rectangular?  That is, does each row have the same number of columns?  And vice versa?

                        Share and Enjoy

                        Quinn “The Eskimo!”
                        Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                        let myEmail = "eskimo" + "1" + "@apple.com"

                        • Re: Plist max size
                          john daniel Level 2 Level 2 (35 points)

                          300 (rows) * 100 (columns) * 8 (size of double) = 240000 bytes = 234K.

                           

                          I think most people take up more space than that for icons on buttons.

                            • Re: Plist max size
                              DenisO Level 1 Level 1 (0 points)

                              Good point

                                • Re: Plist max size
                                  eskimo Apple Staff Apple Staff (6,660 points)

                                  john daniel wrote:

                                  I think most people take up more space than that for icons on buttons.

                                  Right.  But this will be considerably bigger if you put it in an XML plist.

                                  @DenisO, For a rectangular array like this I’d probably put the data into a file in binary form and then memory map that.  You can then access the values at random and the VM subsystem will take care of the rest.

                                  Furthermore, I’d wrap this in a nice, Swift-friendly data structure.  Something like this:

                                  class DataMap {
                                      let rowCount = 16
                                      let columnCount = 16
                                      init?(file: URL) {
                                          guard let d = try? NSData(contentsOf: file, options: [.alwaysMapped]) else {
                                              return nil
                                          }
                                          guard d.length == (self.rowCount * self.columnCount * MemoryLayout<Double>.size) else {
                                              return nil
                                          }
                                          self.data = d
                                          let base = d.bytes.assumingMemoryBound(to: Double.self)
                                          self.buffer = UnsafeBufferPointer(start: base, count: d.length)
                                      }
                                      private let data: NSData
                                      private let buffer: UnsafeBufferPointer<Double>
                                      subscript (row: Int, column: Int) -> Double {
                                          precondition((0..<self.rowCount).contains(row))
                                          precondition((0..<self.columnCount).contains(column))
                                          return self.buffer[row * self.rowCount + column]
                                      }
                                  }
                                  

                                  Note that I’ve hard-coded the row and column counts, which makes sense if all of these objects have the same dimensions.  If not, those values would need to be parameters to the initialiser.

                                  Share and Enjoy

                                  Quinn “The Eskimo!”
                                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                  let myEmail = "eskimo" + "1" + "@apple.com"

                                    • Re: Plist max size
                                      DenisO Level 1 Level 1 (0 points)

                                      Thanks!
                                      Will try this out
                                      But, will try Core Data first

                                      • Re: Plist max size
                                        john daniel Level 2 Level 2 (35 points)

                                        I was thinking of the space required in memory rather than on disk.

                                         

                                        Other options, like a binary file, SQLite database, or Core Data will certainly work, but each carries its own set of trade offs. picking the best format depends on how the data will eventually be used and/or how often the original data needs to be updated.

                            • Re: Plist max size
                              KMT Level 8 Level 8 (8,010 points)

                              I've gotten away with some crazy large read-only plist data sources, but usually they represented one table each. After a while, my issue became more of maintenance than deployment or performance, and moving away from a simple plist would have righted that ship, I'm sure.

                               

                              If you have legitimately large data sets, and the skills to apply more comprehensive methods, I'd go that route.

                                • Re: Plist max size
                                  DenisO Level 1 Level 1 (0 points)

                                  What "data storage" you using now, instead plists?

                                    • Re: Plist max size
                                      KMT Level 8 Level 8 (8,010 points)

                                      Look into Apple's Core Data - which is basically a (non-trivial) data model abstraction layer over your data storage - allows you to manage relationships and fetching rules between different data entities, over large and complex data sets.

                                       

                                      https://developer.apple.com/reference/coredata

                                       

                                      Behind the Core Data layer, your data storage could be sqlite db, or binary object.

                                        • Re: Plist max size
                                          DenisO Level 1 Level 1 (0 points)

                                          Will check it closely, thanks
                                          So far, I found it not really convenient to populate data to DB (need to write some code) and no useful features to put pre-populated DBs in app (need to write a code to replace empty files with full one from binary at first app start)
                                          Work with plist file much easier for me so far

                                            • Re: Plist max size
                                              KMT Level 8 Level 8 (8,010 points)

                                                   >Work with plist file much easier for me so far

                                               

                                              I think it is common to start with a simple plist, then after you get the app in the store, and your skills and requirements grow, to migrate to CD, etc, later.

                                               

                                              I think for now you'll be ok using a plist data source.

                                               

                                              Good luck.

                                               

                                              Ken

                                      • Re: Plist max size
                                        DenisO Level 1 Level 1 (0 points)

                                        Done.

                                        It works now.

                                        Some main problems had to solve with CoreData:

                                        1) Use pre-populated "DB" files in app

                                        2) Make it work with iOS 8

                                        3) Tranfser data from Excel to CSV, with table structure change (got 5000+ rows), and then, with separate app, to SQLite files

                                         

                                        Now, need to polish and simplify fetch code

                                         

                                        Thanks everyone!
                                        You helped me a lot