After creating an hardlink sandboxed app cannot write to it anymore

I'm developing an application that uses hardlinks to track certain files created by the app. Initially, before the hardlink is created, the files behave as expected. For example, if the app generates a .number file, I can open and edit it with Numbers without any issues. However, once the hardlink is created, the file appears locked, and Numbers can no longer write to it.

What are you actually trying to do here and why are hardlinks important here?
The problem here is, setting the immediate issue aside, I'm not sure this will actually work very well in practice. Two problems I see off the top of my head:

1. The standard "safe save" architecture works by having apps modify a copy of the file, the (ideally, atomically) exchange the new file with the old file. On file system that specifically this atomic operation (APFS and HFS+), I think that your hard link will still work. However, on all other file systems your hardlink will fail (as the new file is in fact "a new file"). In addition, I don't know what will happen if/when your hard link is open at the time of modification, but every scenario I can think of it "weird".

2. Cloud storage makes hard links very, very messy. Your hard link creates a reference to the file at a particular point in time, but the system then deletes that file (because it doesn't need it) and then later downloads it again. Your app ends up with a "copy" of that earlier state which is completely disconnected from the files current state.

For example, the comparison here:

Checking the logs in the Console app, I see that Numbers throws an NSCocoaErrorDomain error with code 513. This problem only occurs with sandboxed apps—non-sandboxed apps like Visual Studio Code work fine.

It's possible this is specifically tied to "the sandbox", but I'm not confident of that. I'd expect Numbers and Visual Studio (or Xcode) to have VERY different relationships with the files they manipulate. Numbers things of itself as "owning" the file you're editing and doesn't expect anything to be manipulating or modifying it. It's also designed to work with files that are FAR larger and more complex than anything Visual Studio manipulates. A 100mb spreadsheet is "large", while a 100mb source file is "insane".

Coming from the other direction, Visual Studio (again, like basically any IDE) expects to have a relatively "weak" relationship with it's files. It's not unusual for source files to be manipulated by other tools or directly replaced from "underneath" the editor. More generally, it also expects that it will be asked to track/manage files that it can't actually open/manipulate. All of those things make it far more "tolerant" of change than Numbers would/should be.

That last point is critical (and probably what's causing the failure in numbers). Numbers is design to operate over iCloud using file coordination to integrate changes, etc. Most IDE don't really bother with that- the read the contents at specific moments, notify (ideally) when a collision occurs, and make the user responsible for resolving any issue.

This issue doesn’t occur with symlinks, but my application specifically requires hardlinks, and I haven't been able to find a workaround.

One thing to understand here is that macOS has 3 mechanism for tracking files, not 2, and that third mechanism exists for a reason. Each mechanism represents an answer to the philosophical question "What is a file?":

1. Symbolic link

-> "A file is a named object at a specific location in the logical hierarchy of the system".

What matters here is the specific path to a given location, so ANY change to that path means you're referencing a different object.

2. Hard link

-> "A file is the data storage construct that the file system is responsible for tracking and managing".

Renaming a file means nothing (the file name is metadata about the object, not the object itself). Hard links can't cross file systems because the tracking construct is ONLY coherent within the specific file system. If you replace the original file with a new file, then the hard link should break. That new file isn't the same object as the old file, even if it has the same name.

The thing to understand here is that NEITHER of these ideas actually correspond to the users idea of what a file is. For example:

• It's perfectly normal to rename a file, it's still the same file.

• If I copy a file, modify it, then replace the original, that new file is still the same "file" that I started with.

Users do have a reasonably coherent definition of what "a file" is, but that definition can't be neatly summarized and doesn't really match with an easily described file system construct. That leads to:

3. Bookmarks (or, the construct previously known as an "Alias")

-> "A file is the thing the user thinks it is and I'll do my best to keep track of it".

Conceptually, they behave somewhere between symbolic links and hard links. They'll happily track files across renames (like a hardlink) but they can also resolve when one file has been replaced with another file (like a symbolic link). They can also cross file system (like a symbolic link) but they're FAR more robust against path changes than a symbolic link would be. Architecturally, they basically work by collecting a set of low level file system data about the target, then using a heuristic to try and "find" their target if/when all of that metadata doesn't match*

*This explains what "bookmarkDataIsStale" at resolution actually means. The heuristic was able to find the file using the information it had, but some of the information in the bookmark data didn't actually match the file it "found". That means you should generate the bookmark again so that your bookmark is accurate again.

None of these mechanism is necessarily "better" than the others. Our documentation tends to focus on bookmarks because they're the more complicated than the other two and, in most case, what an app actually wants to do is match the users expectation of what a "file" is. Similarly, the sandbox system strongly focuses on bookmarks because it's the only construct that's really "integrated" with the sandbox (hard links basically "just work", symbolic links can't really work at all). What's important is to use the right tool for the right task.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Here’s a refined version of your forum response:

First of all, thank you for the prompt response. I understand the potential issues with hardlinks, and I'm now questioning whether they are the best solution for my needs. Let me explain my requirements more clearly, and perhaps you could suggest a better approach.

I'm developing an app that functions similarly to Google Drive, with the following basic user flow:

1. The user selects a location on their filesystem where all files from the web platform will be stored.
2. Once the setup is complete, the app continuously monitors all file and folder changes (such as additions, deletions, renaming, moving, and editing).
3. Every minute, the app syncs these changes with the remote platform.

The app is built using Electron, with file tracking managed by a Node.js library.

The main reason I was considering hardlinks is to preserve file changes when the app is offline, particularly for tracking file moves and renames. However, as you pointed out, hardlinks may not be reliable for handling file updates, as different apps manage updates in various ways, which could break the hardlinks.

A key aspect of my application is that files are versioned on the remote platform, and I need to maintain this versioning even when the app is offline.

I think bookmarks might fit my use case, but I'd prefer a cross-platform solution, and I'm concerned that bookmarks may not be supported across different operating systems.

Do you have any other suggestions on how I could address this issue?

First of all, thank you for the prompt response. I understand the potential issues with hardlinks, and I'm now questioning whether they are the best solution for my needs. Let me explain my requirements more clearly, and perhaps you could suggest a better approach.

I'm developing an app that functions similarly to Google Drive, with the following basic user flow:

How similar? In particular, do you also want to have the concept of "cloud" and "local" files? Or do you intend to simply have "all" files exist locally? How do you intend to handle collisions (for example, where the server and user have moved different files to the same location)?

My immediate thought would be to use the "Replicated File Provider extension" API, as that's the API we specifically designed for "Cloud Storage Apps", but it's possible your requirements are so much simpler that this might be overkill.

Looking at your flow here:

1. The user selects a location on their filesystem where all files from the web platform will be stored.

2. Once the setup is complete, the app continuously monitors all file and folder changes (such as additions, deletions, renaming, moving, and editing).

3. Every minute, the app syncs these changes with the remote platform.

....

The main reason I was considering hardlinks is to preserve file changes when the app is offline, particularly for tracking file moves and renames.

The two big questions are are:

1. What are you users actually going to do with this?

2. How well do you want this to "work"?

For example, if you were replicating the contents of a static web server which the user could then modify and edit, a very simple "read the file system, push what changed" implementation might work ok. The layout is staying stable enough that it's not hard to track and this kind of content doesn't really care about "tracking files" the way broader types of content.

At the other end of things, this approach works pretty poorly for "general storage" (like Google Drive). You can't integrate closely enough with the system APIs to actual track what's going on all that well. Lots of things are happening simultaneously, which can make it very hard to know what "actually" happened when you're simply watching the file system hierarchy over time. Last, and perhaps most importantly, unless you've spent a lot of time thinking about how "all" of this works, it's VERY easy to build something that "seems to work" but then ends up failing badly because of details and edge cases you hadn't considered.

Finally, on this point:

I think bookmarks might fit my use case, but I'd prefer a cross-platform solution, and I'm concerned that bookmarks may not be supported across different operating systems.

I don't know what platforms you're targeting, but I don't think this is the kind of problem that can be solved all that well without system level support. There is a reason macOS and Windows both have specific APIs for "Cloud storage systems". It's really hard to make them work without very low level system support and (at least in the case of macOS) it was easier to create a dedicated API that met their needs instead of trying keep things working using the APIs the were previously using (like kauth).

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thank you for your response! For now, I’ve decided not to implement offline tracking. Instead, I'll focus on monitoring file changes while my app is active, which seems to be working well by simply watching for file system events. If I eventually need a more integrated solution, I’ll definitely consider the Replicated File Provider extension API, as you suggested. It makes sense that direct integration with the file system would offer the most reliable tracking. I really appreciate your help, thanks again!