The new object capture API is really quite remarkable, and I can already think of several possible use cases and pipelines for it.
It is great that even though it can work with just 2D images the API can use metadata like a depth maps, lens data, gravity direction, and GPS location to create a better understanding of the scene and reconstruction.
In a situation where the rough camera pose of each captured image is known (for example an ARKit tracked frame, or a static array of cameras) I would love the ability to add an initial camera transform matrix to each PhotogrammetrySample.
Without assuming too much about how the underlying system works, I assume that the camera extrinsics and intrinsics are constantly being refined as the model is being reconstructed. In this situation I would not expect the input camera poses to be taken as absolute values.
But I think being able to give those initial matrix transform values would have several benefits:
- Give an even better hint for the camera positions than just the gravity vector would alone.
- Define object scale based on camera extrinsics, even if no depth data is available.
- Predefine a coordinate space, origin and orientation for the capture.
- Have a common and consistent origin point and orientation between scans with an identical (or similar) camera setup.
Without having tried a drone capture yet perhaps there is a way to do this with GPS data. But that feels like an unnecessary hack-a-round, and likely prone to conversion errors.
