Pass classifications from Reconstructed Scene Understanding to Metal renderer

Question

cabada OP

Created Dec ’21

Replies 2

Boosts 0

Participants 3

Hey everyone :-)

I'm trying access the information of my ARFrame so I can colorize different parts of my scene depending of their classifications.

In other words, I'm looking for a way to transform XYZ World Coordinates or XY Screen Coordinates to what is being classified by my AR Session in that position.

All the floor in the scene being colorize in red, the tables in blue, the walls in green, etc.

How could I embed in real-time these classifications from my reconstructed ARMeshAnchor into my coordinates so I can access it on my Metal Script in real-time?

As of today...

I've been able to get the XYZ World Coordinate related to a XY Screen Coordinate reading the following example: Building an Immersive Experience with RealityKit

Here is a scene painting it with different shades of purple depending on the Z world-coordinate:

P.S. I want to be able to do it within RealityKit.

Thanks in advance.

Boost

Answer 1

DTS Engineer OP

Apple

Dec ’21

Hello cabada,

I believe that this is possible, although the approach would be very round-a-bout and possibly inefficient. Effectively, it would involve grabbing the ARMeshAnchors from the ARView's underlying ARSession, rendering the geometry (with classifications) yourself using either Metal or SceneKit to an off-screen texture, and then utilizing that texture as part of a RealityKit post-processing effect.

Keep in mind that I don't necessarily recommend that you take this approach, but if you really badly want to achieve this effect, then it couldn't hurt to give this a try :)

As I say, there may be many inefficiencies in such an approach, so I also recommend that you file an enhancement request using Feedback Assistant, and request that RealityKit provide a more direct way of achieving the effect you are going for!

0

Answer 2

philipturner OP

Jan ’22

I already pulled off such an approach with scene color reconstruction and you can access it through ARHeadsetKit. It renders the true color of the objects instead of using semantic classification though (I’ve never found a use for the object categories).

0