Create Alpha Matte from SceneDepth API in SceneKit (Occlusion)

Question

Created Jul ’20

Replies 3

Boosts 0

Views 1.1k

Participants 4

Hello,

I would love to utilize the new Occlusion Feature that has been added to RealityKit.
But as my app still requires a lot of features that RealityKit doesn't yet deliver I have to keep using SceneKit for now.

I was wondering if I could convert the depth map provided by the SceneDepth API into an alpha matte that I could then feed into a SCNTechnique to achieve a 'poor mans' occlusion.

Is there some kind of CIFilter or workflow that could help me with this? Maybe some kind of edge detection?

Thankful for any hints!

Boost

Answer 1

Sunkenf250 OP

Aug ’20

Configure for arworldtracking , then use

configuration.frameSemantics = .personSegmentationWithDepth

run the configuration

0

Answer 2

aharriscrowne OP

Aug ’20

As Sunkenf250 points out, the occlusion actually comes as an ARKit feature, not specifically RealityKit. In addition, once you've added the frame semantics to your configuration, you will receive a pixel buffer in each frame's estimatedDepthData property for all recognized people, which can easily be translated to a CIImage.

0

Answer 3

DTS Engineer OP

Apple

Aug ’20

Hello,

It will be significantly easier to make use of the sceneReconstruction mesh to implement scene occlusion in SceneKit.

You would construct your SCNGeometry from the ARMeshAnchors, (see this thread: https://developer.apple.com/forums/thread/130599), and then you would apply an occlusion material to that geometry. This sample (https://developer.apple.com/documentation/arkit/tracking_and_visualizing_faces) explains how to create an occlusion material in SceneKit.

0