At first, I must thank you for your attempts to help me.
So the md was another wrong trace. I'm completely lost than :( Again...
Quick side question here- where are you mounting the volume? Have you tried mounting the volume in "/Volumes/"?
Previously, I was mounting my Volume into /tmp/myfs .
During the work with this problem I've tried changing the mount point to ~/Documents/myfs , ~/Desktop/myfs, and lastly /Volumes/myfs .
The /Volumes/myfs was the last thing needed to make things work, paired with MNT_LOCAL (mounting it into other places seem to break everything).
Note that samba is the exception here, not the rule. As far as I can tell, the majority of our file system DO implement VNOP_SEARCHFS, including both HFS and APFS. Have you tried implementing it simply to see "what happens"? Keep in mind that for testing purposes, you "search" doesn't have to actually "work". You can just hard code the expected result just to see what happens.
Yes, for several times already. The vnop of searchfs simply never gets called. A debug line right in the beginning of the function is never printed in any logs. And yes, VOL_CAP_INT_SEARCHFS is turned on. No sense of full implementation before a single call occurs.
By the way, I have encountered some information related NetFS plugins.
Article:
https://techiesfromthecradle.blogspot.com/2012/10/macosx-fuse-ssh-finder-and-other-goodies.html
Example implementation:
https://github.com/benavento/mac9p/blob/fca70ccee5c8840d31fcfeedd4c89d09e336c449/plugin/plugin.c#L525C8-L525C30
Does my filesystem need such plugin ? Or this is just a userspace layer, which simply communicates with kernel and gives data to some vnop/vfsop calls, hence they might be manually coded, "pushed" as arguments on specific cases ?
Sorry for potentially duplicating questions. It has been over 2 month since I've started on working with this problem, and whatever found since 2004 forums was already tested. Thank you for your patience.
Post
Replies
Boosts
Views
Activity
I've done some more research. I think I've found a little clue.
Previously, I've tried to stick to spotlight. However, it was a wrong track indeed.
I've dtraced Samba (smbfs).
They are a heavy user of mds, which seems to appear in 3/4 trace calls, when finder search is initiated.
Mine kext, actually, returns error on:
mdls call:
user@users-MacBook-Air myfs % pwd
/Volumes/myfs
user@users-MacBook-Air myfs % mdls /Volumes/myfs/abcdfolder
/Volumes/myfs/abcdfolder: could not find /Volumes/myfs/abcdfolder.
However, if we use non-mds commands, we actually do find the required results.
user@users-MacBook-Air myfs % ls -lah
total 6144
drwxrwxrwx 6 user staff 5B 19 Jun 15:38 .
drwxr-xr-x 7 root wheel 224B 25 Jun 18:00 ..
-rw-r--r--@ 1 user staff 6.0K 25 Jun 17:04 .DS_Store
d-wx--x--t 3 user staff 1B 17 Jun 16:35 .TemporaryItems
drwx------ 2 user staff 3B 25 Jun 16:19 .fseventsd
drwxr-xr-x 2 user staff 5B 20 Jun 14:41 abcdfolder
user@users-MacBook-Air myfs % find abcdfolder
abcdfolder
abcdfolder/myfilepanic.txt
abcdfolder/testfile
abcdfolder/newfile123.txt
abcdfolder/helpme.txt
abcdfolder/newfile.txt
mdfind call:
user@users-MacBook-Air myfs % mdfind abcdfolder
[no output]
For some reason, on Sonoma macOS, the MNT_LOCAL and MNT_DOVOLFS do the trick and allow the mdls, mdfind and even mdutil (spotlight) to work. As was already discovered, MNT_LOCAL for network shares is too bad and simply unacceptable.
What exactly does mds need from me, what is given by MNT_LOCAL on Sonoma exquisitely ? Some attributes, flags, vnops ?
P.S. Samba filesystem does not have any ioctl calls, neither has searchfs implemented.
P.S.S. There are no system / my driver's internal / dtruss / dtrace errors upon any of md'... family calls failing. Simply the error message printed.
I'm sorry, it was a typo.
VOL_CAP_INT_SEARCHFS is indeed turned for Capabilities Interfaces, not for Capabilities Format.
Lastly, and this is happens a lot with VFS drivers, the answer to "why doesn't it work for me?" is almost always "because you are NOT in fact doing the same thing". The VFS system is riddled with details and edge cases where small differences end up breaking things. That's also why I can't tell you "here's what's wrong". I'd have a hard time even coming up with a list of possible failures, much less guessing at which one went "wrong".
I think I've finally understood what you are trying to say. Each case is unique, and trying to do what other filesystems do may or may not help us, so it's not the perfect solution to simply mimic everything.
Have you looked at what ioctl calls you're getting (if any)? Of everything you've said, this is the one that I was immediately most concerned about. I'm pretty sure it's also NOT something FUSE does.
Yes, I did check the ioctl commands. For Sonoma, there are only two vnop_ioctl calls. The debug message by me:
VNOP_IOCTL ENTRANCE, a_command HEX [0x40046812] DECIMAL [1074030610]
VNOP_IOCTL ENTRANCE, a_command HEX [0x40046813] DECIMAL [1074030611]
From what I've found, those commands are mentioned is OpenZFS as
SPOTLIGHT_IOC_GET_MOUNT_TIME
https://github.com/openzfsonosx/openzfs/blob/development/include/os/macos/zfs/sys/zfs_vnops_os.h#L48
and SPOTLIGHT_IOC_GET_LAST_MTIME
https://github.com/openzfsonosx/openzfs/blob/development/include/os/macos/zfs/sys/zfs_vnops_os.h#L51
However, if we talk about Monterey, there are hundreds of calls to vnop_ioctl. However, grouping them up has shown, that there are only four unique calls :
Once, previously discovered
VNOP_IOCTL ENTRANCE, a_command HEX [0x40046812] DECIMAL [1074030610] (SPOTLIGHT_IOC_GET_MOUNT_TIME)
Once, previously discovered
VNOP_IOCTL ENTRANCE, a_command HEX [0x40046813] DECIMAL [1074030611] (SPOTLIGHT_IOC_GET_LAST_MTIME)
Unidentified command, called hundreds of times
VNOP_IOCTL ENTRANCE, a_command HEX [0x40084A4A] DECIMAL [1074285130]
Another unidentified command, also called hundreds of times
VNOP_IOCTL ENTRANCE, a_command HEX [0x40044A48] DECIMAL [1074022984]
What did your file system say it supported? All the ones you listed were local and most local file system don't support searchfs because it's easier just to let spotlight do it's thing. It was created for network file systems, where the spotlight approach just isn't a good idea.
The thing is, the vnop_searchfs is never called. And I couldn't find the reason for it.
To stop blindly shooting, I will give some useful information about my vfs.
mount flags (always the same, set in code):
#define MNT_FLAGS (MNT_NOEXEC|MNT_NOSUID|MNT_NODEV|MNT_IGNORE_OWNERSHIP|MNT_NOATIME|MNT_NOBLOCK /|MNT_LOCAL|MNT_DOVOLFS/);
vfs_setflags(fsmount, MNT_FLAGS);
MNT_LOCAL and MNT_DOVOLFS were turned off by your note and for testing purposes.
VFS operations supported
.vfs_start
.vfs_mount
.vfs_unmount
.vfs_root
.vfs_getattr
.vfs_setattr
.vfs_sync
.vfs_vget
Supported attributes
3.1 Capabilities format
| VOL_CAP_FMT_SYMBOLICLINKS
| VOL_CAP_FMT_HARDLINKS
| VOL_CAP_FMT_CASE_SENSITIVE
| VOL_CAP_FMT_CASE_PRESERVING
| VOL_CAP_FMT_2TB_FILESIZE
| VOL_CAP_FMT_PATH_FROM_ID
| VOL_CAP_INT_SEARCHFS
3.2 Capabilities interfaces
| VOL_CAP_INT_ATTRLIST
| VOL_CAP_INT_VOL_RENAME
| VOL_CAP_INT_FLOCK
| VOL_CAP_INT_EXTENDED_ATTR
3.3. Common attr
| ATTR_CMN_FSID
| ATTR_CMN_FILEID
| ATTR_CMN_OBJTYPE
| ATTR_CMN_FNDRINFO
3.4. Volattr
| ATTR_VOL_NAME
| ATTR_VOL_MOUNTFLAGS
| ATTR_VOL_ATTRIBUTES
| ATTR_VOL_IOBLOCKSIZE
| ATTR_VOL_CAPABILITIES
3.5. DirAttr, FileAttr, ForkAttr, are 0.
Supported vnops
vnop_getattr_desc
vnop_setattr_desc
vnop_lookup_desc
vnop_open_desc
vnop_close_desc
vnop_read_desc
vnop_write_desc
vnop_readdir_desc
vnop_mkdir_desc
vnop_rmdir_desc
vnop_create_desc
vnop_remove_desc
vnop_rename_desc
vnop_mmap_desc
vnop_mnomap_desc
vnop_reclaim_desc
vnop_fsync_desc
#if 0 // TODO: vnop_link incomplete. To be revisited at a later time
vnop_link_desc
#endif
vnop_symlink_desc
vnop_readlink_desc
vnop_getxattr_desc
vnop_setxattr_desc
vnop_listxattr_desc
vnop_removexattr_desc
vnop_ioctl_desc // debug only
vnop_searchfs_desc // debug only
*vnop_ioctl and vnop_searchfs are realised like
{
LOG(VNOP_[IOCTL/SEARCHFS] Entrance...);
return KERN_SUCCESS;
}
**As mentioned, vnop_searchfs isn't called at any time on any OS.
I'm sorry for large context, but I think that will be more useful . Thank you for your patience.
Sorry, I don't know how to quote parts of the answer, so I'll go the "old style" way.
Yes. For IOCTL, I've made a hook for function which simply returns success for any time. It was previously made just to test whether my filesystem requires ioctl at all. In fact, it does. I will add implementation sometime later.
"I believe there is a fallback implementation* for more limited searches when indexing is unavailable/impossible" - this is indeed truth. Whenever the indexing returns error, it seems that no search is possible. However, when indexing can be disabled (and all of the spotlight indexes are cleared manually), the search works indeed. However I haven't found any documented information on what exactly works here.
Yes, my filesystem is indeed a network one. I have read that indexation for network filesystems is disabled. But again - actually, I do not need the spotlight indexation. I need only the finder search. The spotlight mentioned over and over - is simply the most obvious thing, that seems to be glued to it from some point.
I have checked VOL_CAP_INT_SEARCHFS, however I thought it was a wrong track - none of the filesystems I've checked (fuse, openzfs, ntfs-3g, etc), actually support the search vnop. No, it is not supported, nor it is even mentioned. Thanks for the hint.
The MNT_LOCAL flag (-o local mount option) is what most fuse filesystems put inside of them. I had downloaded several, and tried to mimic the behaviour. Because whether it works for them, shouldn't it work for me ? That was the case, which enabled all of the required logic for me. Actually, it has broken some of file editing operations, but those also were fixed by adding MNT_DOVOLFS flag. I am not pretending this is perfect solution. It just works, for reasons unknown for me. I would happily prefer a proper solution, If I succeed to find one.
For older systems - neither can I agree nor disagree. The system logs do not show any errors at all sadly. Whether I change the mount options or my kext itself - then I get a ton of them. But whether I use the kext stable for Sonoma, with those mimics, then there are no problems according to logs. So no idea what to do there really.
Thank you for your advice. I will look more closely into vnop_searchfs and ioctls calls. I think I've already became bald from macOS kernel mode :D I'll try those and reply back, whether it helped or not.