Hey,
I have run several tests with masks in the given folder upon PhotogrammetrySession init. It seems the masks are taken into account as the results differ from when I don't provide them.
Unfortunately, the results aren't as good as we can expect when masks are provided.
Has anyone been able to make it work? How?
Example of Imagemagik conversion applied and filename:
magick mogrify -monitor -format tif -depth 8 *.png: my original masks are in png format.IMG_0001_mask.TIFI'm planning on working with object masks as well, but I'm yet to give it a try. However, instead of reading files from disk, I'm providing the session with self-constructed samples due to my specific use-case. Maybe you've already seen this, but when providing the objectMask on the PhotogrammetrySample, the documentation reads:
When a photograph of an object includes surrounding objects, such as plants, buildings, or people in an outdoor space, you can create an object mask to exclude the portions of the image that don’t contain the object. Masking extraneous image data reduces the number of landmarks RealityKit attempts to match, speeds up the object-creation process, and produces a more accurate 3D model.
Provide the object mask in kCVPixelFormatType_OneComponent8 format and with the same height and width as image. RealityKit ignores any pixel in image when the corresponding pixel in objectMask has a value of 0.0 (black) unless isObjectMaskingEnabled is set to False in the session’s configuration.
Coming from the machine learning world, I've just assumed the file type would be PNG, as that's fairly common for segmentation masks. At least it should be easy to make a PNG in the kCVPixelFormatType_OneComponent8 format, but I guess it could be a different file type if you wanted.
Are your masks using this format?
If possible, could you provide a sample of your images and corresponding masks? I'm curious as to what you're attempting to make it work with. For example, how "tight" your mask is to your object etc.
Thank you for the heads up. I am trying to avoid giving the PhotogrammetrySample to avoid the management of all the particular use cases of file formats, or metadata but I certainly have missed something.
I tried with both TIF and PNG. Here are samples: https://www.icloud.com/iclouddrive/0DE-K9J1GKDrdYHzNpTo20vbw#Object_Capture_Mask_format_684718
Hi @hni, @perception. Have you guys found the best practices on how to apply masks? Thanks!
@hni, @perception. I'm also struggling with how to properly mask samples. if anyone has figured this out, I'd love to understand the correct approach...
Adding a mask to the PhotogrammetrySample crashes HelloPhotogrammetry for me as well.
-
—
hni
-
—
hni
-
—
hni
Add a CommentI tested the same input with the beta 3, and the results with masking enabled are quite different from the previous beta:
Auto masking works, and we can notice a major difference in the resulting USDZ.When I provide masks, the photogrammetry session now takes them into account, but the results are still not what one would expect. The object reconstruction is not working as well as when no masks are provided or when auto masking is enabled. The resulting object presents major holes.Any help would be appreciated.
I removed some of the input pictures and the results totally vary. I need to pursue experimentation I guess.
Quick update: I used quite "tight" masks; I even adjusted some manually. I still wasn't happy with the results. We gave up on that front because none of the results satisfied us enough. We move our effort downstream to photogrammetry processes such as great UX for manual or automated cropping.
Still, if someone finds the right way to use it, I take the answer!