Issue in Sequoia OS(15.2) with USB FAT32 remounting, when monitored with ES_EVENT_TYPE_AUTH_MOUNT event

Description: The issue with USB FAT32 is seen in Sequoia OS. Most of the times issue is seen when FAT32 USB is mounted along with other USBs like XFAT. The scenario is where USB mounting is monitored using Endpoint Security framework event ES_EVENT_TYPE_AUTH_MOUNT and when event is received, it will be denied for mounting is it is in read-write mode. And, program tries to mount the USB in read-only mode.

Steps to Reproduce:

  1. Use the xcode program (which will be sent) for testing. Run the executable on macos having Sequoia OS.
  2. start executing the binary after successful compilation. Make sure it's running.
  3. Take 2 USB drives one with FAT32 and another one with XFAT.
  4. Try to mount the USBs and watch the logs on the terminal where the binary is running.
  5. We can see, the USB mounting in read-only mode fails for FAT32 where as it passes for other USB. The issue with mounting is not seen always, but, seen when more than 1 USB mounted and FAT32 we see most of the times.
  6. Once the mounting fails for the USB, we keep seeing this issue if we try to mount the USB using command line or any other way, until we remove the device and reconnect it.

#include <EndpointSecurity/EndpointSecurity.h>
#include <bsm/libbsm.h>
#include <iostream>
#include <os/log.h>

#define MAX_THREADS_LIMIT 64

es_client_t *g_client = nullptr;
dispatch_queue_t dispatchQueue;
static std::atomic<int> m_numThreads;

bool mountVolumeCommandLine(const std::string diskPath, const bool &isReadOnly) {
    std::string command("");
    const std::string quote = "\"";
    if(isReadOnly) {
        command = "diskutil mount readOnly "+ quote + diskPath + quote;
    } else {
        command = "diskutil mount "+ quote + diskPath + quote;
    }
    FILE *mount =
    popen(command.c_str(), "r");
    if (mount == NULL) {
        os_log_error(OS_LOG_DEFAULT, "Failure!! mounting of %{public}s failed using command = %{public}s", diskPath.c_str(),command.c_str());
        return false;
    } else {
        std::string result = "";
        os_log(OS_LOG_DEFAULT, "successful!! executed mount for  %{public}s using command =  %{public}s ",diskPath.c_str(), command.c_str());
    }
    pclose(mount);
    return true;
}

void handleEvents(const es_message_t *msg) {
    m_numThreads++;
    switch(msg->event_type) {
        case ES_EVENT_TYPE_AUTH_MOUNT: {
            std::string diskPath = msg->event.mount.statfs->f_mntfromname;
            std::string volumePath = msg->event.mount.statfs->f_mntonname;
            mountVolumeCommandLine(diskPath, true);
            break;
        }
        default: break;
    }
    m_numThreads--;
}


bool sendAuthResponse(const es_message_t *msg, const es_auth_result_t &result) {
    es_respond_result_t res = es_respond_auth_result(g_client, msg, result, false);
    if (res != ES_RESPOND_RESULT_SUCCESS) {
        os_log_error(OS_LOG_DEFAULT, "SampleEndpointSecurity Failed to respond to auth event error");
        return false;
    }
    return true;
}

int createESClient(const es_handler_block_t &handler)
{
    dispatchQueue =
    dispatch_queue_create("com.test.es_notify", DISPATCH_QUEUE_SERIAL);
    
    dispatch_set_target_queue(dispatchQueue,
                              dispatch_get_global_queue(QOS_CLASS_BACKGROUND, 0));
    while(1) {
        es_new_client_result_t res = es_new_client(&g_client, handler);
        if(ES_NEW_CLIENT_RESULT_SUCCESS != res) {
            g_client = nullptr;
            std::cout<<"client creation failed"<<std::endl;
            if(ES_NEW_CLIENT_RESULT_ERR_NOT_ENTITLED == res) {
                os_log_error(OS_LOG_DEFAULT, "SampleEndpointSecurity ESClient creation Error: Program requires proper entitlement");
                sleep(300);
            } else if(ES_NEW_CLIENT_RESULT_ERR_NOT_PERMITTED == res) {
                os_log_error(OS_LOG_DEFAULT,"SampleEndpointSecurity ESClient creation Error: Program needs proper permission for using ESClient");
            } else {
                os_log_error(OS_LOG_DEFAULT,"SampleEndpointSecurity ESClient creation Error: %d", res);
            }
            return 1;
        }
        else {
            break;
        }
    }

    es_clear_cache_result_t resCache = es_clear_cache(g_client);
    if(ES_CLEAR_CACHE_RESULT_SUCCESS != resCache) {
        os_log_error(OS_LOG_DEFAULT, "\n SampleEndpointSecurity es_clear_cache: %d\n", resCache);
        return 1;
    }
    return 0;
}

int main()
{
    es_handler_block_t handler = ^void(es_client_t * _Nonnull, const es_message_t * _Nonnull msg){
        bool processEvent = false;
        if(!msg->process->is_es_client) {
            switch(msg->event_type) {
                case ES_EVENT_TYPE_AUTH_MOUNT: {
                    std::string diskPath = msg->event.mount.statfs->f_mntfromname;
                    std::string volumePath = msg->event.mount.statfs->f_mntonname;
                    long flags = msg->event.mount.statfs->f_flags;
                    if(flags & MNT_RDONLY) {
                        os_log(OS_LOG_DEFAULT, "ALLOW readOnly mount event for volumePath= %{public}s and diskPath=%{public}s", volumePath.c_str(), diskPath.c_str());
                        sendAuthResponse(msg, ES_AUTH_RESULT_ALLOW);
                    } else {
                        os_log(OS_LOG_DEFAULT, "DENY the mount event for volumePath=%{public}s and diskPath=%{public}s", volumePath.c_str(), diskPath.c_str());
                        sendAuthResponse(msg, ES_AUTH_RESULT_DENY);
                        processEvent = true;
                    }
                    break;
                }
                default: {
                    os_log(OS_LOG_DEFAULT,"SampleEndpointSecurity default case event_type:  (%d)", msg->event_type);
                    break;
                    // Not interested
                }
            }
            if(processEvent && m_numThreads.load() < MAX_THREADS_LIMIT) {
                es_retain_message(msg);
                dispatch_async(dispatchQueue, ^{
                    handleEvents(msg);
                    es_release_message(msg);
                });
            }
        }
    };
    if(createESClient(handler) == 1) {
        return 1;
    }
    es_event_type_t events[] = {ES_EVENT_TYPE_AUTH_MOUNT
    };
    es_return_t subscribed = es_subscribe(g_client,
                                          events,
                                          // Count of es_event_type_t entries stored in events[]
                                          sizeof(events) / sizeof(es_event_type_t)
                                          );
    
    if(ES_RETURN_ERROR == subscribed) {
        os_log_error(OS_LOG_DEFAULT, "SampleEndpointSecurity es_subscribe: ES_RETURN_ERROR\n");
        return 1;
    }
    dispatch_main();
    return 0;
}

Description: The issue with USB FAT32 is seen in Sequoia OS. Most of the times issue is seen when FAT32 USB is mounted along with other USBs like XFAT. The scenario is where USB mounting is monitored using Endpoint Security framework event ES_EVENT_TYPE_AUTH_MOUNT and when event is received, it will be denied for mounting is it is in read-write mode. And, program tries to mount the USB in read-only mode.

I'm not sure of exactly what's going on in your code, but I have a number of issues and concerns:

  1. If you're going to be blocking/altering mounts, then I strongly recommend that you use DiskArbitration as your "primary" blocking system, with ES acting as your security "backstop". The issue here is that DiskArbitration is a higher level API that's better integrated into the system, which should make the entire user experience smoother.

  2. Related to that point, I haven't tested this but I believe you could initiate your read-only mount in (or shortly after) the disk appearance. For the standard automount path (basically, "when the user plugs in a drive"), I think the timing would work out such that the read-only mount would occur "before" the rw mount, removing the need ot reject anything.

  3. Note that it IS possible for a mount to bypass DiskArb by calling mount directly, which is why your ES client (which cannot be bypassed) would still act as the final security "backstop". However, the systems "normal" activity goes through DiskArb, so your ES implementation can be much simpler, only need to block unusual edge cases or direct "attacks".

  4. As a general comment, initiating system level changes from "inside" an ES callback is potentially dangerous, particularly with a highly complex operation like "mount", as you're basically generating new activity at exactly the same time your own actions are being "reacted" to. It's very likely that this kind of issue is what actually causing this

  1. Once the mounting fails for the USB, we keep seeing this issue if we try to mount the USB using command line or any other way, until we remove the device and reconnect it.

diskutil actual mount through DiskArb so, assuming that the original read/write mount as an automount, then the same daemon (diskabitrationd) is both trying to deal with the mount syscall you just failed and the new mount command you've initiated. It's not a solution I'd recommend relying on, but I suspect that delaying your mount request by a few seconds would avoid the issue by allowing DiskArb to fully process your failure before you initiate new work.

Note that DiskArb's big advantage for #4 is that it's approval system all happens before any syscall or vfs involvement. In more concrete terms, when what rejecting a DiskArb mount actually means is that diskabitrationd simply never calls mount. The VFS (or ES) system is never aware that anything "happened" because the mount attempt was cancelled before the lower level system was ever involved.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Issue in Sequoia OS(15.2) with USB FAT32 remounting, when monitored with ES_EVENT_TYPE_AUTH_MOUNT event
 
 
Q