Endpoint Security System Extenisons and detecting access to sandboxed file

The short answer here is “No.” However, I’m not sure I fully understand what you’re asking for and I’d like to clarify that before I send you off to file an ER.

It sounds like you’re trying to mute file system events for a process where those events relate to a file that’s inside the process’s sandbox. Is that right?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

That's right. Ideally we would mute the event. But simply checking a flag to determine that the file is sandboxed and handling it by es_respond_auth_result(client, message, ES_AUTH_RESULT_ALLOW, cache) would be sufficient. If it is a file our process running under root cannot access for scanning, then there is no reason to try opening it. When Installd unpacks app bundle into its sandbox, eventually it will move it into /Applications and it can be scanned there.

The reason why we try to avoid opening files we cannot open is the fact that even the failure takes time. Even decision whther to use open() or access() has impact on how long the failure lasts. If the failure to open file takes 20ms, then for example an app with 50000 files could take 1000s or 16.6 minutes.

Currently I'm strugling with determining whether to use access() or open() to check if the file can be accessed. There are cases where access() check is faster and cases where open() is faster (even if the result is failure) - it depends on various factors, like process has access to file, or doesn't have basic unix permissions or file is sandboxed. That's why I'm looking for ways to avoid unnecessary file system calls.

But simply checking a flag to determine that the file is sandboxed

Hmmm, you’re mixing up terms here. Files are not sandboxed, but rather processes.

If it is a file our process running under root cannot access for scanning, then there is no reason to try opening it.

And that’s a very different task. You’re not trying to determine whether the file is sandboxed but rather whether you can open the file. There are a variety of reasons for that to fail. I got into more detail about this in On File System Permissions.

Currently I'm strugling with determining whether to use access or open to check if the file can be accessed.

That depends on what you plan to do with the file. If your eventual goal is to read (or write) the file, open is the best choice.

It sounds like what you care about is the performance of the failure case. To my mind these should be the same. That is, open should not be slower to fail than access [1]. If it is, that’s definitely worth filing a bug about.

Still, there’s a bigger picture issue here. It sounds like your ultimate goal is to have your ES client mute events for files that it can’t open. I’m not sure if it’s feasible for macOS to support that, but it doesn’t hurt to file an enhancement request with your rationale.

If you do file any bugs, please post your bug numbers, just for the record.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] On local file systems. Network file systems are a different story here.