Hey,
I have run several tests with masks in the given folder upon PhotogrammetrySession init. It seems the masks are taken into account as the results differ from when I don't provide them.
Unfortunately, the results aren't as good as we can expect when masks are provided.
Has anyone been able to make it work? How?
Example of Imagemagik conversion applied and filename:
magick mogrify -monitor -format tif -depth 8 *.png: my original masks are in png format.IMG_0001_mask.TIF
I tested the same input with the beta 3, and the results with masking enabled are quite different from the previous beta:
Auto masking works, and we can notice a major difference in the resulting USDZ.When I provide masks, the photogrammetry session now takes them into account, but the results are still not what one would expect. The object reconstruction is not working as well as when no masks are provided or when auto masking is enabled. The resulting object presents major holes.Any help would be appreciated.
I'm planning on working with object masks as well, but I'm yet to give it a try. However, instead of reading files from disk, I'm providing the session with self-constructed samples due to my specific use-case. Maybe you've already seen this, but when providing the objectMask on the PhotogrammetrySample, the documentation reads:
When a photograph of an object includes surrounding objects, such as plants, buildings, or people in an outdoor space, you can create an object mask to exclude the portions of the image that don’t contain the object. Masking extraneous image data reduces the number of landmarks RealityKit attempts to match, speeds up the object-creation process, and produces a more accurate 3D model.
Provide the object mask in kCVPixelFormatType_OneComponent8 format and with the same height and width as image. RealityKit ignores any pixel in image when the corresponding pixel in objectMask has a value of 0.0 (black) unless isObjectMaskingEnabled is set to False in the session’s configuration.
Coming from the machine learning world, I've just assumed the file type would be PNG, as that's fairly common for segmentation masks. At least it should be easy to make a PNG in the kCVPixelFormatType_OneComponent8 format, but I guess it could be a different file type if you wanted.
Are your masks using this format?
If possible, could you provide a sample of your images and corresponding masks? I'm curious as to what you're attempting to make it work with. For example, how "tight" your mask is to your object etc.
I removed some of the input pictures and the results totally vary. I need to pursue experimentation I guess.
Adding a mask to the PhotogrammetrySample crashes HelloPhotogrammetry for me as well.
Quick update: I used quite "tight" masks; I even adjusted some manually. I still wasn't happy with the results. We gave up on that front because none of the results satisfied us enough. We move our effort downstream to photogrammetry processes such as great UX for manual or automated cropping.
Still, if someone finds the right way to use it, I take the answer!