8 Replies
      Latest reply on Jan 14, 2019 8:35 AM by Crudebyte
      Crudebyte Level 1 Level 1 (15 points)

        Since all Apple platforms are now running APFS, is there any API that allows to resize a file in non-sparse way, such that APFS would immediately allocate the total amount of blocks for the entire requested file size immediately?

         

        Obviously one could resize a file and then seek & write random data to the file to force this behaviour, but a dedicated API for that purpose would make much more sense due to efficiency reasons.

         

        ---

         

        Use Case: there are scenarios where the copy on write / on demand allocation behaviour of APFS is not the desired one. For instance assume a software which allows to purchase some very large content which is downloaded from a server after payment. In this case it makes sense if the software immediately allocates the total amount of FS space for the content to avoid that the download aborts with a large delay somewhere in between the download process e.g. after hours due to insufficient disk space, which would be very frustrating for users.

        • Re: APFS: Resize file (non-sparse)
          eskimo Apple Staff Apple Staff (11,355 points)

          Have you looked at F_PREALLOCATE?  I haven’t been tracking this issue in recent years, so I’m not sure how well it’s supported on APFS, but that’s the standard way of dealing with this issue.

          See the fcntl man page for details.

          Share and Enjoy

          Quinn “The Eskimo!”
          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
          let myEmail = "eskimo" + "1" + "@apple.com"

            • Re: APFS: Resize file (non-sparse)
              Crudebyte Level 1 Level 1 (15 points)

              I played around with F_PREALLOCATE and it does the job, thanks!

               

              In case anybody else is looking for a working code, here is my little test app:

               

              #import <Foundation/Foundation.h>
              #include <fcntl.h>
              #include <sys/stat.h>
              #include <sys/types.h>
              
              // some hard coded test file to be created ...
              #define FILE_PATH_C "/tmp/test.data"
              #define FILE_PATH @FILE_PATH_C
              
              // ... with this hard coded file size
              #define FILE_SIZE (400*1024*1024) // 400 MiB
              
              int main() {
                  // remove file of previous run (if any)
                  [[NSFileManager defaultManager] removeItemAtPath:FILE_PATH error:nil];
                  // (re)create file
                  [[NSFileManager defaultManager] createFileAtPath:FILE_PATH contents:nil attributes:nil];
              
                  NSFileHandle* hFile = [NSFileHandle fileHandleForWritingAtPath:FILE_PATH];
              
                  // resize file logically (won't allocate any blocks with APFS)
                  [hFile truncateFileAtOffset:FILE_SIZE];
              
                  // force allocation of all physical blocks with APFS
                  fstore_t store = {
                      .fst_posmode = F_PEOFPOSMODE,
                      .fst_flags = F_ALLOCATECONTIG | F_ALLOCATEALL,
                      .fst_length = FILE_SIZE
                  };
                  int res = fcntl(hFile.fileDescriptor, F_PREALLOCATE, &store);
                  if (res < 0) {
                      printf("fcntl error: %d -> %s\n", res, strerror(errno));
                      return -1;
                  }
              
                  [hFile synchronizeFile];
                  [hFile closeFile];
              
                  // print logical file size and actual physical file size on disk
                  struct stat s;
                  ::stat(FILE_PATH_C, &s);
                  printf("File Size=%lld, Allocated=%lld\n", s.st_size, store.fst_bytesalloc);
              
                  return 0;
              }
              

               

              Usage of flag F_ALLOCATEALL is mandatory here. I also added the flag F_ALLOCATECONTIG to ensure it allocates all blocks as contiguous space, however it also worked without the latter, but I probably leave it there just to be sure.

               

              What I found interesting, if you leave out the truncateFileAtOffset: call, then you end up with a physical file size of 400 MiB on disk, however the logical file size would be zero. I didn't even consider this to be possible before.

               

              What I don't know yet is the behaviour of that fcntl() call on a HFS+ file system, whether it succeeds as with APFS, or whether it throws an error. Would be interesting to know, because obviously such a code should be written to be backward compatible with older systems which are not running APFS yet.

               

              If somebody can try out this code on a HFS+ disk would be very much appreciated!

               

              clang -framework Foundation foo.mm
              ./a.out
                • Re: APFS: Resize file (non-sparse)
                  eskimo Apple Staff Apple Staff (11,355 points)

                  What I don't know yet is the behaviour of that fcntl call on a [HFS Plus] file system

                  F_PREALLOCATE originated well before APFS, so I’d expect it to behave sensibly on older systems.  There’s a direct line leading from F_PREALLOCATE (Mac OS X 10.0) to FSAllocateFork (Mac OS 9) to AllocContig (Inside Mac: IV, that is, the Mac Plus) to Allocate (Inside Mac: II, that is, the Mac 128).

                  If somebody can try out this code on a [HFS Plus] disk would be very much appreciated!

                  You can test that yourself by creating an HFS Plus disk image.

                  Share and Enjoy

                  Quinn “The Eskimo!”
                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                  let myEmail = "eskimo" + "1" + "@apple.com"

                    • Re: APFS: Resize file (non-sparse)
                      Crudebyte Level 1 Level 1 (15 points)

                      The pure existence of a flag in previous Mac OS versions does not give any indication about its actual resulting behaviour. That's why I asked. The standard POSIX function mlockall() comes to my mind for instance which was a pure stub on Mac for many years, so it compiled but it did not do anything, and most probably still doesn't today.

                       

                      And yes, I am aware that I can simply create a HFS+ volume, but as you can imagine the precise behaviour might also depend on the OS version. ;-)

                        • Re: APFS: Resize file (non-sparse)
                          eskimo Apple Staff Apple Staff (11,355 points)

                          but as you can imagine the precise behaviour might also depend on the OS version

                          Quite.  If you need that level of precision my recommendation is that you set up VMs for all the macOS versions you support and test it on each.

                          Share and Enjoy

                          Quinn “The Eskimo!”
                          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                          let myEmail = "eskimo" + "1" + "@apple.com"

                            • Re: APFS: Resize file (non-sparse)
                              Crudebyte Level 1 Level 1 (15 points)

                              I tried this code on iOS and got a different behaviour there. First of all on iOS the fcntl() call only succeeds when passing exactly .fst_flags = F_ALLOCATEALL, so my previous combinaion of F_ALLOCATECONTIG | F_ALLOCATEALL would cause a "not enough disk space" error code on iOS.

                               

                              The latter is rather trivial, but what matters more; the resulting physical size is not the requested size on iOS. For instance if I request a physical file size of 893901652 then I actually get 893900456 instead on iOS. Is that the expected behaviour?

                                • Re: APFS: Resize file (non-sparse)
                                  eskimo Apple Staff Apple Staff (11,355 points)

                                  I’m sorry but you’ve run off the end of my expertise in this matter.  I recommend that you open a DTS tech support incident and talk to our APFS specialist.  Make sure to include a reference to this thread for context.

                                  Share and Enjoy

                                  Quinn “The Eskimo!”
                                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                  let myEmail = "eskimo" + "1" + "@apple.com"

                                    • Re: APFS: Resize file (non-sparse)
                                      Crudebyte Level 1 Level 1 (15 points)

                                      I figured it out by myself in the meantime: store.fst_bytesalloc returned by fcntl() is the amount of newly allocated physical space. Since my code above first resized the file logically before requesting a resize physically, the file already had a bunch of bytes physically pre-allocated at the point where fcntl() was called, so the value of store.fst_bytesalloc was due to this a bit smaller than the requested amount (store.fst_bytesalloc := finalPhysicalSizeAfterfcntlCall - physicalSizeBeforefcntlCall).

                                       

                                      After interchanging those two requests, that is first allocating physically with fcntl() and then resizing the file logically, store.fst_bytesalloc returns always exactly the amount of bytes requested.

                                       

                                      However I will probably still file a TSI regarding this issue, since there are still some open questions. For instance my code works fine for new files (i.e. file size == 0), but does not work as expected for existing files with file size > 0. In the latter case the fcntl() call would sometimes reduce(!) the physical size and sometimes abort the fcntl() call with an "invalid argument" error.