Post

Replies

Boosts

Views

Activity

Reply to Spotlight / Finder Search / Finder Tags not working on virtual file system Monterey/Ventura
At first, I must thank you for your attempts to help me. So the md was another wrong trace. I'm completely lost than :( Again... Quick side question here- where are you mounting the volume? Have you tried mounting the volume in "/Volumes/"? Previously, I was mounting my Volume into /tmp/myfs . During the work with this problem I've tried changing the mount point to ~/Documents/myfs , ~/Desktop/myfs, and lastly /Volumes/myfs . The /Volumes/myfs was the last thing needed to make things work, paired with MNT_LOCAL (mounting it into other places seem to break everything). Note that samba is the exception here, not the rule. As far as I can tell, the majority of our file system DO implement VNOP_SEARCHFS, including both HFS and APFS. Have you tried implementing it simply to see "what happens"? Keep in mind that for testing purposes, you "search" doesn't have to actually "work". You can just hard code the expected result just to see what happens. Yes, for several times already. The vnop of searchfs simply never gets called. A debug line right in the beginning of the function is never printed in any logs. And yes, VOL_CAP_INT_SEARCHFS is turned on. No sense of full implementation before a single call occurs. By the way, I have encountered some information related NetFS plugins. Article: https://techiesfromthecradle.blogspot.com/2012/10/macosx-fuse-ssh-finder-and-other-goodies.html Example implementation: https://github.com/benavento/mac9p/blob/fca70ccee5c8840d31fcfeedd4c89d09e336c449/plugin/plugin.c#L525C8-L525C30 Does my filesystem need such plugin ? Or this is just a userspace layer, which simply communicates with kernel and gives data to some vnop/vfsop calls, hence they might be manually coded, "pushed" as arguments on specific cases ? Sorry for potentially duplicating questions. It has been over 2 month since I've started on working with this problem, and whatever found since 2004 forums was already tested. Thank you for your patience.
Jun ’24
Reply to Spotlight / Finder Search / Finder Tags not working on virtual file system Monterey/Ventura
I've done some more research. I think I've found a little clue. Previously, I've tried to stick to spotlight. However, it was a wrong track indeed. I've dtraced Samba (smbfs). They are a heavy user of mds, which seems to appear in 3/4 trace calls, when finder search is initiated. Mine kext, actually, returns error on: mdls call: user@users-MacBook-Air myfs % pwd /Volumes/myfs user@users-MacBook-Air myfs % mdls /Volumes/myfs/abcdfolder /Volumes/myfs/abcdfolder: could not find /Volumes/myfs/abcdfolder. However, if we use non-mds commands, we actually do find the required results. user@users-MacBook-Air myfs % ls -lah total 6144 drwxrwxrwx 6 user staff 5B 19 Jun 15:38 . drwxr-xr-x 7 root wheel 224B 25 Jun 18:00 .. -rw-r--r--@ 1 user staff 6.0K 25 Jun 17:04 .DS_Store d-wx--x--t 3 user staff 1B 17 Jun 16:35 .TemporaryItems drwx------ 2 user staff 3B 25 Jun 16:19 .fseventsd drwxr-xr-x 2 user staff 5B 20 Jun 14:41 abcdfolder user@users-MacBook-Air myfs % find abcdfolder abcdfolder abcdfolder/myfilepanic.txt abcdfolder/testfile abcdfolder/newfile123.txt abcdfolder/helpme.txt abcdfolder/newfile.txt mdfind call: user@users-MacBook-Air myfs % mdfind abcdfolder [no output] For some reason, on Sonoma macOS, the MNT_LOCAL and MNT_DOVOLFS do the trick and allow the mdls, mdfind and even mdutil (spotlight) to work. As was already discovered, MNT_LOCAL for network shares is too bad and simply unacceptable. What exactly does mds need from me, what is given by MNT_LOCAL on Sonoma exquisitely ? Some attributes, flags, vnops ? P.S. Samba filesystem does not have any ioctl calls, neither has searchfs implemented. P.S.S. There are no system / my driver's internal / dtruss / dtrace errors upon any of md'... family calls failing. Simply the error message printed.
Jun ’24
Reply to Spotlight / Finder Search / Finder Tags not working on virtual file system Monterey/Ventura
Lastly, and this is happens a lot with VFS drivers, the answer to "why doesn't it work for me?" is almost always "because you are NOT in fact doing the same thing". The VFS system is riddled with details and edge cases where small differences end up breaking things. That's also why I can't tell you "here's what's wrong". I'd have a hard time even coming up with a list of possible failures, much less guessing at which one went "wrong". I think I've finally understood what you are trying to say. Each case is unique, and trying to do what other filesystems do may or may not help us, so it's not the perfect solution to simply mimic everything. Have you looked at what ioctl calls you're getting (if any)? Of everything you've said, this is the one that I was immediately most concerned about. I'm pretty sure it's also NOT something FUSE does. Yes, I did check the ioctl commands. For Sonoma, there are only two vnop_ioctl calls. The debug message by me: VNOP_IOCTL ENTRANCE, a_command HEX [0x40046812] DECIMAL [1074030610] VNOP_IOCTL ENTRANCE, a_command HEX [0x40046813] DECIMAL [1074030611] From what I've found, those commands are mentioned is OpenZFS as SPOTLIGHT_IOC_GET_MOUNT_TIME https://github.com/openzfsonosx/openzfs/blob/development/include/os/macos/zfs/sys/zfs_vnops_os.h#L48 and SPOTLIGHT_IOC_GET_LAST_MTIME https://github.com/openzfsonosx/openzfs/blob/development/include/os/macos/zfs/sys/zfs_vnops_os.h#L51 However, if we talk about Monterey, there are hundreds of calls to vnop_ioctl. However, grouping them up has shown, that there are only four unique calls : Once, previously discovered VNOP_IOCTL ENTRANCE, a_command HEX [0x40046812] DECIMAL [1074030610] (SPOTLIGHT_IOC_GET_MOUNT_TIME) Once, previously discovered VNOP_IOCTL ENTRANCE, a_command HEX [0x40046813] DECIMAL [1074030611] (SPOTLIGHT_IOC_GET_LAST_MTIME) Unidentified command, called hundreds of times VNOP_IOCTL ENTRANCE, a_command HEX [0x40084A4A] DECIMAL [1074285130] Another unidentified command, also called hundreds of times VNOP_IOCTL ENTRANCE, a_command HEX [0x40044A48] DECIMAL [1074022984] What did your file system say it supported? All the ones you listed were local and most local file system don't support searchfs because it's easier just to let spotlight do it's thing. It was created for network file systems, where the spotlight approach just isn't a good idea. The thing is, the vnop_searchfs is never called. And I couldn't find the reason for it. To stop blindly shooting, I will give some useful information about my vfs. mount flags (always the same, set in code): #define MNT_FLAGS (MNT_NOEXEC|MNT_NOSUID|MNT_NODEV|MNT_IGNORE_OWNERSHIP|MNT_NOATIME|MNT_NOBLOCK /|MNT_LOCAL|MNT_DOVOLFS/); vfs_setflags(fsmount, MNT_FLAGS); MNT_LOCAL and MNT_DOVOLFS were turned off by your note and for testing purposes. VFS operations supported .vfs_start .vfs_mount .vfs_unmount .vfs_root .vfs_getattr .vfs_setattr .vfs_sync .vfs_vget Supported attributes 3.1 Capabilities format | VOL_CAP_FMT_SYMBOLICLINKS | VOL_CAP_FMT_HARDLINKS | VOL_CAP_FMT_CASE_SENSITIVE | VOL_CAP_FMT_CASE_PRESERVING | VOL_CAP_FMT_2TB_FILESIZE | VOL_CAP_FMT_PATH_FROM_ID | VOL_CAP_INT_SEARCHFS 3.2 Capabilities interfaces | VOL_CAP_INT_ATTRLIST | VOL_CAP_INT_VOL_RENAME | VOL_CAP_INT_FLOCK | VOL_CAP_INT_EXTENDED_ATTR 3.3. Common attr | ATTR_CMN_FSID | ATTR_CMN_FILEID | ATTR_CMN_OBJTYPE | ATTR_CMN_FNDRINFO 3.4. Volattr | ATTR_VOL_NAME | ATTR_VOL_MOUNTFLAGS | ATTR_VOL_ATTRIBUTES | ATTR_VOL_IOBLOCKSIZE | ATTR_VOL_CAPABILITIES 3.5. DirAttr, FileAttr, ForkAttr, are 0. Supported vnops vnop_getattr_desc vnop_setattr_desc vnop_lookup_desc vnop_open_desc vnop_close_desc vnop_read_desc vnop_write_desc vnop_readdir_desc vnop_mkdir_desc vnop_rmdir_desc vnop_create_desc vnop_remove_desc vnop_rename_desc vnop_mmap_desc vnop_mnomap_desc vnop_reclaim_desc vnop_fsync_desc #if 0 // TODO: vnop_link incomplete. To be revisited at a later time vnop_link_desc #endif vnop_symlink_desc vnop_readlink_desc vnop_getxattr_desc vnop_setxattr_desc vnop_listxattr_desc vnop_removexattr_desc vnop_ioctl_desc // debug only vnop_searchfs_desc // debug only *vnop_ioctl and vnop_searchfs are realised like { LOG(VNOP_[IOCTL/SEARCHFS] Entrance...); return KERN_SUCCESS; } **As mentioned, vnop_searchfs isn't called at any time on any OS. I'm sorry for large context, but I think that will be more useful . Thank you for your patience.
Jun ’24
Reply to Spotlight / Finder Search / Finder Tags not working on virtual file system Monterey/Ventura
Sorry, I don't know how to quote parts of the answer, so I'll go the "old style" way. Yes. For IOCTL, I've made a hook for function which simply returns success for any time. It was previously made just to test whether my filesystem requires ioctl at all. In fact, it does. I will add implementation sometime later. "I believe there is a fallback implementation* for more limited searches when indexing is unavailable/impossible" - this is indeed truth. Whenever the indexing returns error, it seems that no search is possible. However, when indexing can be disabled (and all of the spotlight indexes are cleared manually), the search works indeed. However I haven't found any documented information on what exactly works here. Yes, my filesystem is indeed a network one. I have read that indexation for network filesystems is disabled. But again - actually, I do not need the spotlight indexation. I need only the finder search. The spotlight mentioned over and over - is simply the most obvious thing, that seems to be glued to it from some point. I have checked VOL_CAP_INT_SEARCHFS, however I thought it was a wrong track - none of the filesystems I've checked (fuse, openzfs, ntfs-3g, etc), actually support the search vnop. No, it is not supported, nor it is even mentioned. Thanks for the hint. The MNT_LOCAL flag (-o local mount option) is what most fuse filesystems put inside of them. I had downloaded several, and tried to mimic the behaviour. Because whether it works for them, shouldn't it work for me ? That was the case, which enabled all of the required logic for me. Actually, it has broken some of file editing operations, but those also were fixed by adding MNT_DOVOLFS flag. I am not pretending this is perfect solution. It just works, for reasons unknown for me. I would happily prefer a proper solution, If I succeed to find one. For older systems - neither can I agree nor disagree. The system logs do not show any errors at all sadly. Whether I change the mount options or my kext itself - then I get a ton of them. But whether I use the kext stable for Sonoma, with those mimics, then there are no problems according to logs. So no idea what to do there really. Thank you for your advice. I will look more closely into vnop_searchfs and ioctls calls. I think I've already became bald from macOS kernel mode :D I'll try those and reply back, whether it helped or not.
Jun ’24