Spotlight / Finder Search / Finder Tags not working on virtual file system Monterey/Ventura

Vnop_ioctl returns KERN_SUCCESS (but no implementation).

What do you mean by this? Do you mean that you've implemented that entry point but hard coded it to always succeed? Blindly returning success for ioctl calls you don't recognize or implement is not a great idea.

An important moment: Previously, the search didn't work at all. Researching the web has shown me, that Spotlight indexation and Finder search are tightly glued.

Yes. The Spotlight index is the primary mechanism used by Finder search. I believe there is a fallback implementation* for more limited searches when indexing is unavailable/impossible, but I can see there being problems if the implementation doesn't "fit" within the expected boundaries, leaving spotlight thinking the drive should be indexed but unable to do so.

*To simplify the conversation, I've used the same division between "Finder" and "Spotlight" that you used, however, I think Spotlight actually implements both search "types".

Moving to here:

So basically I was trying to enable support for spotlight, thinking that would be the source of the problem. I was receiving "Unknown indexing state".

I think this is actually where your problem starts. Are you a network FS or a local one? The system is designed to avoid indexing remote volumes in favor of server side search, support of which has (historically) been somewhat limited. Basically, AFP, our own SMB implementation, and a few servers when specifically configured to support it. I think server side serach was actually added in a more recent SMB revision, but it's not something the system just "expects" to work. So, the question here is why the system things your network file system "should" support it?

It terms of specifics, does your volume set "VOL_CAP_INT_SEARCHFS"? And, if so, does it fully implement the behavior described in the "searchfs" man page?

The problem was resolved FOR SONOMA by making by VFS appear as local one (adding flags for MNT_LOLCAL and MNT_DOVOLFS).

This is not a strategy I'd recommend. The system makes assumptions based on what you tell it, so trying to "trick" it like this tends to create more problems than it solves. Case in point, this is not success:

Maybe there some abstract rules for what is required on VFS side to be recognised okay'ish by Spotlight ?

Spotlight looks a lot of different things when evaluating a volumes for indexing, many of which are outside of the VFS scope. In the case here, I suspect the older systems are trying to find your local file systems device node and then it's IOKit path so that they can examine your devices physical characteristics (bus type, location, etc). Those are assumed to exist, given that this is what "MNT_LOCAL" means. None of those details are something I can easily describe or summarize.

Kevin Elliott
DTS Engineer, CoreOS/Hardware

Sorry, I don't know how to quote parts of the answer, so I'll go the "old style" way.

1. Yes. For IOCTL, I've made a hook for function which simply returns success for any time. It was previously made just to test whether my filesystem requires ioctl at all. In fact, it does. I will add implementation sometime later.
2. "I believe there is a fallback implementation* for more limited searches when indexing is unavailable/impossible" - this is indeed truth. Whenever the indexing returns error, it seems that no search is possible. However, when indexing can be disabled (and all of the spotlight indexes are cleared manually), the search works indeed. However I haven't found any documented information on what exactly works here.
3. Yes, my filesystem is indeed a network one. I have read that indexation for network filesystems is disabled. But again - actually, I do not need the spotlight indexation. I need only the finder search. The spotlight mentioned over and over - is simply the most obvious thing, that seems to be glued to it from some point.
4. I have checked VOL_CAP_INT_SEARCHFS, however I thought it was a wrong track - none of the filesystems I've checked (fuse, openzfs, ntfs-3g, etc), actually support the search vnop. No, it is not supported, nor it is even mentioned. Thanks for the hint.
5. The MNT_LOCAL flag (-o local mount option) is what most fuse filesystems put inside of them. I had downloaded several, and tried to mimic the behaviour. Because whether it works for them, shouldn't it work for me ? That was the case, which enabled all of the required logic for me. Actually, it has broken some of file editing operations, but those also were fixed by adding MNT_DOVOLFS flag. I am not pretending this is perfect solution. It just works, for reasons unknown for me. I would happily prefer a proper solution, If I succeed to find one.
6. For older systems - neither can I agree nor disagree. The system logs do not show any errors at all sadly. Whether I change the mount options or my kext itself - then I get a ton of them. But whether I use the kext stable for Sonoma, with those mimics, then there are no problems according to logs. So no idea what to do there really.

Thank you for your advice. I will look more closely into vnop_searchfs and ioctls calls. I think I've already became bald from macOS kernel mode :D I'll try those and reply back, whether it helped or not.

Yes. For IOCTL, I've made a hook for function which simply returns success for any time. It was previously made just to test whether my filesystem requires ioctl at all. In fact, it does. I will add implementation sometime later.

Have you looked at what ioctl calls you're getting (if any)? Of everything you've said, this is the one that I was immediately most concerned about. I'm pretty sure it's also NOT something FUSE does.

I have checked VOL_CAP_INT_SEARCHFS, however I thought it was a wrong track - none of the filesystems I've checked (fuse, openzfs, ntfs-3g, etc), actually support the search vnop. No, it is not supported, nor it is even mentioned. Thanks for the hint.

What did your file system say it supported? All the ones you listed were local and most local file system don't support searchfs because it's easier just to let spotlight do it's thing. It was created for network file systems, where the spotlight approach just isn't a good idea.

The MNT_LOCAL flag (-o local mount option) is what most fuse filesystems put inside of them. I had downloaded several, and tried to mimic the behaviour. Because whether it works for them, shouldn't it work for me ?

No, not necessarily. The problem with looking at something like fuse is that most people only look at the user space component, not what's in the kernel which is where all of the "work" actually happens. Similarly, if you do look at the kernel driver it's harder to follow (because the implementation is "split") and VFS driver are pretty hard to follow even when you have a good understanding of how they're "supposed" to work.

Lastly, and this is happens a lot with VFS drivers, the answer to "why doesn't it work for me?" is almost always "because you are NOT in fact doing the same thing". The VFS system is riddled with details and edge cases where small differences end up breaking things. That's also why I can't tell you "here's what's wrong". I'd have a hard time even coming up with a list of possible failures, much less guessing at which one went "wrong".

Kevin Elliott
DTS Engineer, CoreOS/Hardware

Lastly, and this is happens a lot with VFS drivers, the answer to "why doesn't it work for me?" is almost always "because you are NOT in fact doing the same thing". The VFS system is riddled with details and edge cases where small differences end up breaking things. That's also why I can't tell you "here's what's wrong". I'd have a hard time even coming up with a list of possible failures, much less guessing at which one went "wrong".

I think I've finally understood what you are trying to say. Each case is unique, and trying to do what other filesystems do may or may not help us, so it's not the perfect solution to simply mimic everything.

Have you looked at what ioctl calls you're getting (if any)? Of everything you've said, this is the one that I was immediately most concerned about. I'm pretty sure it's also NOT something FUSE does.

Yes, I did check the ioctl commands. For Sonoma, there are only two vnop_ioctl calls. The debug message by me:

1. VNOP_IOCTL ENTRANCE, a_command HEX [0x40046812] DECIMAL [1074030610]

2. VNOP_IOCTL ENTRANCE, a_command HEX [0x40046813] DECIMAL [1074030611]

From what I've found, those commands are mentioned is OpenZFS as SPOTLIGHT_IOC_GET_MOUNT_TIME

https://github.com/openzfsonosx/openzfs/blob/development/include/os/macos/zfs/sys/zfs_vnops_os.h#L48

and SPOTLIGHT_IOC_GET_LAST_MTIME

https://github.com/openzfsonosx/openzfs/blob/development/include/os/macos/zfs/sys/zfs_vnops_os.h#L51

However, if we talk about Monterey, there are hundreds of calls to vnop_ioctl. However, grouping them up has shown, that there are only four unique calls :

1. Once, previously discovered

VNOP_IOCTL ENTRANCE, a_command HEX [0x40046812] DECIMAL [1074030610] (SPOTLIGHT_IOC_GET_MOUNT_TIME)

1. Once, previously discovered

VNOP_IOCTL ENTRANCE, a_command HEX [0x40046813] DECIMAL [1074030611] (SPOTLIGHT_IOC_GET_LAST_MTIME)

1. Unidentified command, called hundreds of times

VNOP_IOCTL ENTRANCE, a_command HEX [0x40084A4A] DECIMAL [1074285130]

1. Another unidentified command, also called hundreds of times

VNOP_IOCTL ENTRANCE, a_command HEX [0x40044A48] DECIMAL [1074022984]

What did your file system say it supported? All the ones you listed were local and most local file system don't support searchfs because it's easier just to let spotlight do it's thing. It was created for network file systems, where the spotlight approach just isn't a good idea.

The thing is, the vnop_searchfs is never called. And I couldn't find the reason for it.

To stop blindly shooting, I will give some useful information about my vfs.

1. mount flags (always the same, set in code):

#define MNT_FLAGS (MNT_NOEXEC|MNT_NOSUID|MNT_NODEV|MNT_IGNORE_OWNERSHIP|MNT_NOATIME|MNT_NOBLOCK /|MNT_LOCAL|MNT_DOVOLFS/); vfs_setflags(fsmount, MNT_FLAGS);

MNT_LOCAL and MNT_DOVOLFS were turned off by your note and for testing purposes.

1. VFS operations supported

   .vfs_start .vfs_mount .vfs_unmount .vfs_root .vfs_getattr .vfs_setattr .vfs_sync .vfs_vget

2. Supported attributes

3.1 Capabilities format

        | VOL_CAP_FMT_SYMBOLICLINKS
        | VOL_CAP_FMT_HARDLINKS
        | VOL_CAP_FMT_CASE_SENSITIVE
        | VOL_CAP_FMT_CASE_PRESERVING
        | VOL_CAP_FMT_2TB_FILESIZE
        | VOL_CAP_FMT_PATH_FROM_ID
        | VOL_CAP_INT_SEARCHFS

3.2 Capabilities interfaces

        | VOL_CAP_INT_ATTRLIST
        | VOL_CAP_INT_VOL_RENAME
        | VOL_CAP_INT_FLOCK
        | VOL_CAP_INT_EXTENDED_ATTR

3.3. Common attr

        | ATTR_CMN_FSID
        | ATTR_CMN_FILEID
        | ATTR_CMN_OBJTYPE
        | ATTR_CMN_FNDRINFO

3.4. Volattr

        | ATTR_VOL_NAME
        | ATTR_VOL_MOUNTFLAGS
        | ATTR_VOL_ATTRIBUTES
        | ATTR_VOL_IOBLOCKSIZE
        | ATTR_VOL_CAPABILITIES

3.5. DirAttr, FileAttr, ForkAttr, are 0.

1. Supported vnops

 vnop_getattr_desc
 vnop_setattr_desc
 vnop_lookup_desc
 vnop_open_desc
 vnop_close_desc
 vnop_read_desc
 vnop_write_desc
 vnop_readdir_desc
 vnop_mkdir_desc
 vnop_rmdir_desc
 vnop_create_desc
 vnop_remove_desc
 vnop_rename_desc
 vnop_mmap_desc
 vnop_mnomap_desc
 vnop_reclaim_desc
 vnop_fsync_desc

#if 0 // TODO: vnop_link incomplete. To be revisited at a later time vnop_link_desc #endif

 vnop_symlink_desc
 vnop_readlink_desc
 vnop_getxattr_desc
 vnop_setxattr_desc
 vnop_listxattr_desc
 vnop_removexattr_desc
 vnop_ioctl_desc // debug only
 vnop_searchfs_desc // debug only

*vnop_ioctl and vnop_searchfs are realised like

{

LOG(VNOP_[IOCTL/SEARCHFS] Entrance...);
return KERN_SUCCESS;

}

**As mentioned, vnop_searchfs isn't called at any time on any OS.

I'm sorry for large context, but I think that will be more useful . Thank you for your patience.

I'm sorry, it was a typo.

VOL_CAP_INT_SEARCHFS is indeed turned for Capabilities Interfaces, not for Capabilities Format.

In previous messages, I mentioned that succeeding all ioctls seemed like a poor idea and I think it might be worth expanding on why that is. By design, ioctl are intended to provide a "side channel" from user space to your vfs driver. By definition, NONE of them are required for a VFS driver to properly function... because anything that was required would have just been defined as as standard VNOP. Some of our file systems to implement many of them but that's either because the file system has evolved enormously over time (HFS+) or because the file system is VERY different than other file system (apfs). Many of the other file systems the system supports don't implement any and that's what the "default" implementation should be for any new file system.

Looking at your specific case:

VNOP_IOCTL ENTRANCE, a_command HEX [0x40084A4A] DECIMAL [1074285130]

I cannot tell you what that ioctl does (it's private and not included in our opensource release), but I can describe the format of ioctl which should make it clear why implementing this could be a problem.

Using "SPOTLIGHT_IOC_GET_MOUNT_TIME" as an example, here is how it's defined in xnu:

#define SPOTLIGHT_IOC_GET_MOUNT_TIME              _IOR('h', 18, u_int32_t)

The "_IOR" gives the value some amount of structure, creating a kind of "namespace" system for ioctl values. The system isn't perfect (as you can see in the file above, "h" is used for ioctl's other than spotlight) but, in general, the ioctl's for a particular "letter" all come from a specific components/subsystem/etc.

In any case, once you understand the format you can figure out the original macro value with a bit of experimentation. I actually have a simple test program wich I used for figuring this stuff out:

#import <Foundation/Foundation.h>
#import <sys/ioctl.h>

int main(int argc, const char * argv[]) {
    @autoreleasepool {
        uint32_t res1 = 0x40084A4A;
        NSLog(@"Val: %d", res1);
        NSLog(@"Val: 0x%X\n", res1);

//Fiddle with this value to find original macro        
        uint32_t res = _IOR('J', 74, uint64_t);

        NSLog(@"Val: 0x%X", res);
        NSLog(@"Val: %d", res);
        NSLog(@"Equal: %d", res == res1);
        return 0;
    }
}

SO, 0x40084A4A was defined as:

IOR('J', 74, uint64_t);

Searching for 'J' in XNU then gets you this in "fsctl.h":

/* This used to be defined in APFS, we keep the group as 'J' for backward compatability */
#define FSIOC_GET_GRAFT_INFO _IOWR('J', 102, fsioc_get_graft_info_t)

In other words, 'J' is the namespace APFS is using for private ioctl calls. I hope this should already be clear, but implementing the unknown, private ioctl calls of other file systems is a very bad idea. I can't tell you what either of those calls actually do (they're private), but I will say that I think implementing them has left the system VERY confused about how it's supposed to interact with your file system.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

I've done some more research. I think I've found a little clue. Previously, I've tried to stick to spotlight. However, it was a wrong track indeed.

I've dtraced Samba (smbfs). They are a heavy user of mds, which seems to appear in 3/4 trace calls, when finder search is initiated.

Mine kext, actually, returns error on:

1. mdls call:

user@users-MacBook-Air myfs % pwd

/Volumes/myfs

user@users-MacBook-Air myfs % mdls /Volumes/myfs/abcdfolder

/Volumes/myfs/abcdfolder: could not find /Volumes/myfs/abcdfolder.

However, if we use non-mds commands, we actually do find the required results.

user@users-MacBook-Air myfs % ls -lah

total 6144

drwxrwxrwx 6 user staff 5B 19 Jun 15:38 .

drwxr-xr-x 7 root wheel 224B 25 Jun 18:00 ..

-rw-r--r--@ 1 user staff 6.0K 25 Jun 17:04 .DS_Store

d-wx--x--t 3 user staff 1B 17 Jun 16:35 .TemporaryItems

drwx------ 2 user staff 3B 25 Jun 16:19 .fseventsd

drwxr-xr-x 2 user staff 5B 20 Jun 14:41 abcdfolder

user@users-MacBook-Air myfs % find abcdfolder

abcdfolder

abcdfolder/myfilepanic.txt

abcdfolder/testfile

abcdfolder/newfile123.txt

abcdfolder/helpme.txt

abcdfolder/newfile.txt

1. mdfind call:

user@users-MacBook-Air myfs % mdfind abcdfolder

[no output]

For some reason, on Sonoma macOS, the MNT_LOCAL and MNT_DOVOLFS do the trick and allow the mdls, mdfind and even mdutil (spotlight) to work. As was already discovered, MNT_LOCAL for network shares is too bad and simply unacceptable.

What exactly does mds need from me, what is given by MNT_LOCAL on Sonoma exquisitely ? Some attributes, flags, vnops ?

P.S. Samba filesystem does not have any ioctl calls, neither has searchfs implemented.

P.S.S. There are no system / my driver's internal / dtruss / dtrace errors upon any of md'... family calls failing. Simply the error message printed.

I've done some more research. I think I've found a little clue. Previously, I've tried to stick to spotlight. However, it was a wrong track indeed.

Not really. "mds" is the daemon that implements "spotlight". Note that "MD" mean "Meta Data", so "mds" is "Meta Data Server". "Spotlight" is the marketing name, "MD" is the API prefix for the Spotlight API. All of the "md*" commands actually work through the spotlight data. SO, what mdls actually does is create an MDItem for the file path, then pull out all of the attributes for that item in the database.

What's why this happens:

P.S.S. There are no system / my driver's internal / dtruss / dtrace errors upon any of md'... family calls failing. Simply the error message printed.

None of the md tools actually interact with disk that haven't been adopted/indexed by spotlight.

I've dtraced Samba (smbfs). They are a heavy user of mds, which seems to appear in 3/4 trace calls, when finder search is initiated.

Yes. mds is what implements "Spotlight", which is how the Finder searches.

For some reason, on Sonoma macOS, the MNT_LOCAL and MNT_DOVOLFS do the trick and allow the mdls, mdfind and even mdutil (spotlight) to work.

What exactly does mds need from me, what is given by MNT_LOCAL on Sonoma exquisitely ? Some attributes, flags, vnops ?

That's not a question I can easily answer. The problem here is that "MNT_LOCAL" doesn't actually "do" anything. Mount flags are passed into the VFS driver so it can for basically "informational" purposes and they're published "out" of the VFS driver for the same reason. In other words, setting "MNT_LOCAL" can change how the system behaves because you've now told the system that you are in fact a local volume.

Quick side question here- where are you mounting the volume? Have you tried mounting the volume in "/Volumes/"?

Finally, there's another complication around smb:

P.S. Samba filesystem does not have any ioctl calls, neither has searchfs implemented.

smb support is fairly tightly integrated into the system, which means the system isn't ONLY interacting with the vfs driver. It can also talk directly to the smb daemon, "bypassing" the kernel. Having looked into this more closely, there are at least three different search mechanism the system might use, depending on the the server it's talking to. There was a presentation at SDC 2022 by Brad Suinn called "What's new in the macOS SMB Client" that outlined them. You can find it on the SNIA website or YouTube, but here is the brief summary:

1. There is an older proprietary protocol used for server side searches on macOS servers.

2. We added support for WSP in MacOS 13 (Ventura).

3. When neither is an option, spotlight falls back to crawling the drive and building it's own index (which does not include file content data).

Note that all of this (with the possible exception of #3) is implemented outside the VFS layer and not usable by your file system.

Also, on this point:

neither has searchfs implemented

Note that samba is the exception here, not the rule. As far as I can tell, the majority of our file system DO implement VNOP_SEARCHFS, including both HFS and APFS. Have you tried implementing it simply to see "what happens"? Keep in mind that for testing purposes, you "search" doesn't have to actually "work". You can just hard code the expected result just to see what happens.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

At first, I must thank you for your attempts to help me.

So the md was another wrong trace. I'm completely lost than :( Again...

Quick side question here- where are you mounting the volume? Have you tried mounting the volume in "/Volumes/"?

Previously, I was mounting my Volume into /tmp/myfs . During the work with this problem I've tried changing the mount point to ~/Documents/myfs , ~/Desktop/myfs, and lastly /Volumes/myfs .

The /Volumes/myfs was the last thing needed to make things work, paired with MNT_LOCAL (mounting it into other places seem to break everything).

Note that samba is the exception here, not the rule. As far as I can tell, the majority of our file system DO implement VNOP_SEARCHFS, including both HFS and APFS. Have you tried implementing it simply to see "what happens"? Keep in mind that for testing purposes, you "search" doesn't have to actually "work". You can just hard code the expected result just to see what happens.

Yes, for several times already. The vnop of searchfs simply never gets called. A debug line right in the beginning of the function is never printed in any logs. And yes, VOL_CAP_INT_SEARCHFS is turned on. No sense of full implementation before a single call occurs.

By the way, I have encountered some information related NetFS plugins.

Article:

https://techiesfromthecradle.blogspot.com/2012/10/macosx-fuse-ssh-finder-and-other-goodies.html

Example implementation:

https://github.com/benavento/mac9p/blob/fca70ccee5c8840d31fcfeedd4c89d09e336c449/plugin/plugin.c#L525C8-L525C30

Does my filesystem need such plugin ? Or this is just a userspace layer, which simply communicates with kernel and gives data to some vnop/vfsop calls, hence they might be manually coded, "pushed" as arguments on specific cases ?

Sorry for potentially duplicating questions. It has been over 2 month since I've started on working with this problem, and whatever found since 2004 forums was already tested. Thank you for your patience.