How can I copy files inside SMB volumes as fast as the Finder?

The copyfile() method of the macOS API does not seem to use SMB server-side copying, while copying in Finder does appear to use this.

First off, please file a bug on this and then post the bug number back here.

The only API option I've found that will do this are the long deprecated Carbon File Manager copy APIs. The Carbon File Manager was originally implemented using the same support library the Finder uses and, while the Finder's overall implementation has changed considerably, this particular case is still going through exactly the same code*.

*For the curious, "server side copying" was a LONG standing feature (pre-Mac OS X) of AFP (Apple Filing Protocol). VFS Support for server side copying was originally implemented for AFP and was otherwise unused. MANY years later, SMB then implemented server side copying using the same VFS hook AFP implemented... at which point the Finder (and Carbon File Manager) suddenly support server side copying on SMB.

The Carbon File Manager API can look pretty strange to modern eyes, so here is what that copy looks like in code:

#import <CoreServices/CoreServices.h>
#import <Foundation/Foundation.h>

bool CopyFileCarbon(NSURL* source, NSURL* destFolder, NSString* newfileName)
{
    bool retVal = false;
    OSStatus err = noErr;
    FSRef file1Ref;
    BOOL isDir;
    err = FSPathMakeRef(source.path.fileSystemRepresentation, &file1Ref, &isDir);
    if(err == noErr)
    {
        FSRef file2Ref;
        err = FSPathMakeRef(destFolder.path.fileSystemRepresentation, &file2Ref, &isDir);
        if(err == noErr)
        {
            FSRef newFileRefl;
            err = FSCopyObjectSync(&file1Ref, &file2Ref,(__bridge CFStringRef) newfileName, &newFileRefl, 0);
            if(err == noErr)
            {
                retVal = true;
            }
        }
    }

    return retVal;
}

A few notes on this:

• Overall, the "FSCopy*" APIs are NOT a good alternative to copyfile (or NSFileManager). Those other APIs are significantly faster for more "general" copying and there may be other edge cases where they don't do the "right" thing. For example, I don't know how well they handle APFS->APFS copies that involve complex cloned file collections.

• Under normal circumstances I would never recommend using a long deprecated API like this, however, I understand that the performance benefits here can be large enough to justify unusual API usage. However, if you're going to use this API, you need to ensure you're specifically detecting when server side copying would apply and only using it there.

• Server side copying is not universally supported. You can check if a particularl file system supports it by looking for "VOL_CAP_INT_COPYFILE" from "getattrlist".

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thanks for the write up! The bug report for this issue is FB13783472 .

You can check if a particularl file system supports it by looking for "VOL_CAP_INT_COPYFILE" from "getattrlist".

The linked documentation for getattrlist reads

VOL_CAP_INT_COPYFILE If this bit is set the volume format implementation supports the (private and undocumented) copyfile() function.

Is this the same copyfile function we can use today, or something else? Or was it “private and undocumented“ some long time ago?

So when VOL_CAP_INT_COPYFILE is set the volume supports server-side copying, but we still have to figure out if it‘s faster than copyfile, right? How would we check that?

The linked documentation for getattrlist reads VOL_CAP_INT_COPYFILE If this bit is set the volume format implementation supports the (private and undocumented) copyfile() function.

Is this the same copyfile function we can use today, or something else?

It marks a completely unrelated VNOP ("vnode operation"-> a specific function a VFS driver may implement), the name overlap being basically coincidental. Keep in mind that VNOP_COPYFILE (that's the underlying VFS function) was specifically a "special purpose" operation that specifically support copying "within a device". You can actually see the smb code for this here and you'll see that there are lot of cases where it immediately errors out because it doesn't handle that particular case.

Or was it “private and undocumented“ some long time ago?

There isn't any public API that directly maps to VNOP_COPYFILE and the only public API I've found that does use it are the Carbon File Manager APIs I referenced above.

So when VOL_CAP_INT_COPYFILE is set the volume supports server-side copying, but we still have to figure out if it‘s faster than copyfile, right? How would we check that?

That's tricky one to answer. I lay out one basic "filter" below, but I could some up with lots of different approaches. Similarly, I've listed the attributes returned by gettattrlist, but most of this information could be retrieved from multiple sources.

1. Do the source and destination both support VOL_CAP_INT_COPYFILE?

2. Are the source and destination both the same volume type? Using gettattrlist, this would be ATTR_VOL_FSTYPENAME.

Note on #2: Currently I believe the only formats that support it are SMB and AFP, but there maybe 3rd party file systems that support it as well.

That narrows you down to an "eligible" volume, however, that doesn't actually tell you whether the source and destination are on the same server. There are two ways you can approach that:

1. Check if the source and destination are both from the same mount point. Using gettattrlist, this would be ATTR_VOL_MOUNTPOINT.

2. Try and determine if both volumes are from the same server. This is a much trickier one, unless the app has some "external" knowledge about the volumes (for example, it mounted both of them). It might be inferable from ATTR_VOL_MOUNTEDDEVICE, but I don't think that will actually work in all cases.

A few notes about these issues:

• If you haven't already, please file a bug report on this and post the bug number back here. Bug have been filed on this already, but new bugs document developer interest and help drive prioritization choices.

• The actual performance benefit form server side copy can vary widely. The most extreme cases is file system copies within a remote APFS volume*, where file cloning means you're comparing a constant time operation against a linear (based on file size) operation. The other extreme is cross volume copies, where you end up comparing two linear operations. In practice, server side copies should always be faster, but the difference can be quite variable.

*Note that this case is pretty common, particularly in "consumer" environments, which is why it's something users notice.

• The downside to VNOP_COPYFILE is that it's implemented through a single blocking syscall. That makes it impossible to provide any kind of progress (this the main reason copyfile doesn't use it) and that the function will block in your app until each file copy complete. In the worst case (for example, copying a multi-gigabyte file between two very slow devices), this could cause the function to block for seconds/minutes. This is a case you specifically need to test "FSCopyObject..." with and make sure your app handles "reasonably".

• Related to testing, I would recommend setting up a specific test suite for whatever "FSCopyObject..." function you decide to use and regularly verify that it continues to work the way you expect. The Carbon File Manager is long deprecated and the ONLY reason I suggest it here is that there isn't any alternative and the common performance gap is SO large that it simply can't be ignored (seconds vs hours). However, the risk here is that the API will break at some point and, because it's been deprecated so long, we won't catch it in our own testing. Don't let yourself get "caught" by that reality.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware