I was able to confirm with a customer of mine that calling copyfile with a source file that is a symbolic link on a NTFS partition always causes the error
NSPOSIXErrorDomain 12 Cannot allocate memory
They use NTFS drivers from Paragon.
They tried copying a symbolic link from NTFS to both APFS and NTFS with the same result. Is this an issue with macOS, or with the NTFS driver?
Copying regular files on the other hand always works. Copying manually from the Finder also seems to always work, both with regular files and symbolic links, so I'm wondering how the Finder does it.
Here is the sample app that they used to reproduce the issue. The first open panel allows to select the source directory and the second one the destination directory. The variable filename holds the name of the symbolic link to be copied from the source to the destination. Apparently it's not possible to select a symbolic link directly in NSOpenPanel, as it always resolves to the linked file.
@main
class AppDelegate: NSObject, NSApplicationDelegate {
func applicationDidFinishLaunching(_ notification: Notification) {
let openPanel = NSOpenPanel()
openPanel.canChooseDirectories = true
openPanel.canChooseFiles = false
openPanel.runModal()
let filename = "Modules"
let source = openPanel.urls[0].appendingPathComponent(filename)
openPanel.runModal()
let destination = openPanel.urls[0].appendingPathComponent(filename)
do {
let state = copyfile_state_alloc()
defer {
copyfile_state_free(state)
}
var bsize = UInt32(16_777_216)
if copyfile_state_set(state, UInt32(COPYFILE_STATE_BSIZE), &bsize) != 0 {
throw NSError(domain: NSPOSIXErrorDomain, code: Int(errno))
}
if copyfile_state_set(state, UInt32(COPYFILE_STATE_STATUS_CB), unsafeBitCast(copyfileCallback, to: UnsafeRawPointer.self)) != 0 || copyfile_state_set(state, UInt32(COPYFILE_STATE_STATUS_CTX), unsafeBitCast(self, to: UnsafeRawPointer.self)) != 0 || copyfile(source.path, destination.path, state, copyfile_flags_t(COPYFILE_NOFOLLOW)) != 0 {
throw NSError(domain: NSPOSIXErrorDomain, code: Int(errno))
}
} catch {
let error = error as NSError
let alert = NSAlert()
alert.messageText = "\(error.localizedDescription)\n\(error.domain) \(error.code)"
alert.runModal()
}
}
private let copyfileCallback: copyfile_callback_t = { what, stage, state, src, dst, ctx in
if what == COPYFILE_COPY_DATA {
if stage == COPYFILE_ERR {
return COPYFILE_QUIT
}
var size: off_t = 0
copyfile_state_get(state, UInt32(COPYFILE_STATE_COPIED), &size)
}
return COPYFILE_CONTINUE
}
}
I was able to confirm with a customer of mine that calling copyfile with a source file that is a symbolic link on a NTFS partition always causes the error
So, as it happens, the code to copyfile is opensource so, if you assume the issue is NOT a shortage of memory (which it isn't), then I suspect this is where the error came from:
static int copyfile_open(copyfile_state_t s)
...
switch (s->sb.st_mode & S_IFMT)
{
case S_IFLNK:
islnk = 1;
if ((size_t)s->sb.st_size > SIZE_T_MAX) {
s->err = ENOMEM; /* too big for us to copy */
return -1;
}
...
I don't known enough about NTFS semantics to know what that value "should" be, but SIZE_T_MAX is defined as ULONG_MAX (that's defined as 0xffffffffffffffffUL on 64bit), so failing that check is definitely "odd". Similarly, I'm not sure why that check was added in copyfile, but it doesn't seem inherently "unreasonable".
They use NTFS drivers from Paragon.
They tried copying a symbolic link from NTFS to both APFS and NTFS with the same result. Is this an issue with macOS, or with the NTFS driver?
Arguably a bit of both (since the Finder does work), but I would argue that it's mostly the VFS driver. At a conceptual level, the role of the VFS layer is to act as the bridge between the on disk format and the file I/O and management semantics of the operating system. I don't know where the VFS driver got that st_size from, but returning a size >~18 Exabytes seems like an unusual choice.
My recommendation would be that you both file a bug with Apple AND contact Paragon about this. There VFS driver should work properly with copyfile, but it's possible we should revisit our check as well.
Copying regular files on the other hand always works. Copying manually from the Finder also seems to always work, both with regular files and symbolic links, so I'm wondering how the Finder does it.
The answer there's pretty simple. The Finder didn't use copyfile and it didn't look at st_size (I suspect it relied st_blocks/st_blksize). More specifically, the Finder's copy implementation is built around NSURL which primarily relies on getattrlist/getattrlistbulk, not stat. getattrlist doesn't have a direct equivalent to stat.st_size, but if stat is returning dramatically different values than getattrlist's size values, then I would absolutely consider that a bug in the VFS driver.
As some background context, the Finder has it's own engine because, historically, the Finder couldn't have used copyfile (because it didn't exist) and we've never really tried move it to copyfile. In theory it could have adopted copyfile after in was introduced, however, in practice that wouldn't really have worked. In particular:
-
The Finder's copy process has VERY specific semantics that copyfile doesn't make any attempt to match. For example, the grey "in progress" icon is tied to a special date value that originally came from Classic MacOS. Changing those details risk creating unnecessary problems and mimic'ing them in copyfile would have created unnecessary complexity (not that it's alway been "understod" that app wouldn't mimic the Finder).
-
There are behavioral differences (for example, server side copying) between the Finder and copyfile, so moving to copyfile would create regressions.
-
The Finder copy is significantly faster than copyfile. Notably, copyfile is single threaded (to minimize it's impact "inside" the calling process), but multithreading can significantly speed up total copy time.
In terms of workarounds, you can try NSFileManager but I think you'll end up getting a similar failure. NSFileManager has been transitioning toward Swift and, as part of that process, I think it'll end up calling copyfile as well. I think your best option here is going to be to catch the error yourself and then either create the link yourself or copy it with a different function. I'm hesitant to recommend it, but I think you best copy option her would be the Carbon file manager, as that's the only copy API I'm confident will completely bypass copyfile (and stat). There's a code snippet for that in this forum post.
__
Kevin Elliott
DTS Engineer, CoreOS/Hardware
