VisionOS ARKit CameraFrame Sample Parameters Extrinsics

Did I understand correctly that the device anchor is where the scene is rendered from onto the user's display?

No. Rendering happens for each eye, from the eye's position. But this is an implementation detail that hopefully should not make a difference for app developers.

What is the coordinate system in which this offset is defined, which axis is left, which one is up, which one is forward?

It is defined in the device anchor's coordinate system. IIRC for the device anchor, X axis should be to the right when viewed from the user's perspective, Y axis is up and Z axis towards the user (right handed). But you should be able to convert vectors/transforms from the camera's coordinate system to the device anchor's coordinate system via the extrinsic matrix "blindly", without needing to worry about / making assumptions about these details.

I am looking forward to learning about all the details on these extrinsics in order to make use of it.

If you have a use case which requires more detailed information about the extrinsic then it would be great if you let us know, either here in the forum or via the Feedback Assistant.

Thanks a lot for all those clarifications! I see at least two use cases in which understanding the camera extrinsics is crucial:

• An object is tracked with a non-generic algorithm which allows for a much higher tracking accuracy for the specific use case than any other out of the box tracking solution. A pose is computed relative to the camera, where is the object in world space?
• A gridded sheet is tracked using ARKit image tracking for a high-feature texture in its center. the user can color each cell of the grid with a set of distinct colors which the system should interpret. Given a 3d coordinate in world space, which pixel area is corresponding in the camera frame?

1. We are now using the WorldTrackingProvider's queryDeviceAnchor with the current timestamp CACurrentMediaTime(). Is multiplying that with the camera extrinsics the correct approach to get the extrinsics in world space?

2. For debugging purposes we are now drawing the captured frames onto a canvas which we position at one meter in front of the camera location (as described above) with pixel density 1/focal length, together with a small sphere at the center of the canvas and a tube going from the camera to the center of the canvas. It looks like the "left camera" really is the right camera (from the user's perspective), is that correct?

3. when rendered for the right eye, the tube seems to to be pointing perfectly forward, slightly from the top left of the display. Does this mean that the whole scene is rendered from the camera's position? If not, what does it mean?

4. Unfortunately we have still not been able to display the tracked object at the correct pose in world space; there is a consistent offset which is very similar to the offset between the passthrough and the rendered frame on the canvas. Thanks a lot for your assistance so far, that is very much appreciated! I will try to test all the assumptions in a minimal project which we could share if that helps, I will keep you posted here if I make any progress.

I stand corrected about the left camera being the right one from the user's perspective. That conclusion was made because you said the extrinsics are in a coordinate system in which the x axis goes towards the user's right, and the extrinsics seem to have a translation with an x component of about 2.5 cm, which would mean that the camera is to the right.

After testing putting my finger on the actual physical cameras, I saw that it is indeed the left camera. So naturally I asked myself what I am doing wrong when interpreting the extrinsics?

Well it turns out: the extrinsics do not define the transformation from the device anchor to the camera, but from the camera to the device anchor. I had to invert the matrix, everything works now.