I am currently developing a mobile and server-side application using the new ObjectCaptureSession on iOS and PhotogrammetrySession on MacOS.
I have two questions regarding the newly updated APIs.
From WWDC23 session: "Meet Object Capture for iOS", I know that the Object Capture API uses Point Cloud data captured from iPhone LiDAR sensor. I want to know how to use the Point Cloud data captured on iPhone ObjectCaptureSession and use it to create 3D models on PhotogrammetrySession on MacOS.
From the example code from WWDC21, I know that the PhotogrammetrySession utilizes depth map from captured photo images by embedding it into the HEIC image and use those data to create a 3D asset on PhotogrammetrySession on MacOS. I would like to know if Point Cloud data is also embedded into the image to be used during 3D reconstruction and if not, how else the Point Cloud data is inserted to be used during reconstruction.
Another question is, I know that Point Cloud data is returned as a result from request to the PhtogrammetrySession.Request. I would like to know if this PointCloud data is the same set of data captured during ObjectCaptureSession from WWDC23 that is used to create ObjectCapturePointCloudView.
Thank you to everyone for the help in advance. It's a real pleasure to be developing with all the updates to RealityKit and the Object Capture API.
My understanding from the WWDC Session Meet Object Capture for iOS, as well as talking to some of the Object Capture engineers during the WWDC 2023 Labs, is that the point cloud data captured during an ObjectCaptureSession is embedded in the HEIC file and automatically parsed during a macOS PhotogrammetrySession. So as long as you provide the input (images) directory to the PhotogrammetrySession when you instantiate it, alongside the optional checkpoint folder, I believe the macOS PhotogrammetrySession should automatically take advantage of the point cloud data for improved reconstruction times and performance.
For what it's worth, in the Taking Pictures for 3D Object Capture sample code, the depth data captured and intended to be provided to the PhotogrammetrySession isn't actually embedded depth data in the HEIC file, but a separate depth mask as a TIFF file. Granted, I know HEIC files are capable of including depth information, so it might be possible that PhotogrammetrySession can intake a HEIC file with embedded depth data (and, newly this year, embedded point cloud data), but just wanted to make that distinction. At least in the Taking Pictures for 3D Object Capture sample code, each image captured includes a HEIC file, a TIFF depth file, and a TXT gravity file - I've been passing an entire folder that contains all of those files into the PhotogrammetrySession and it seems that PhotogrammetrySession leverages all provided data accordingly.
To your latter question, while I'm not entirely sure if the PhotogrammterySession.PointCloud array that's returned from a PhotogrammetrySession.Request is actually the same thing that the ObjectCapturePointCloudView uses, my guess would be yes. It seems like ObjectCapturePointCloudView is a convenience SwiftUI view that beautifully renders the underlying point cloud data that would match the point cloud data returned in the PhotogrammetrySession.Request.
