How can i add a UIView on the material of the ModelEntity?

Question

ChengYang2 OP

Created Nov ’21

Replies 2

Boosts 0

Views 1.5k

Participants 3

How can i show UIView on a ModelEntity?

On android, i use ViewRenderable to load a view , and attach it to a Node, to show a View in the AR scene.

But i not found any ways to show view in the AR scene of the ARView used the RealityKit.

Anyone can help me ?

Thanks very much!

Answered by arthurfromberlin in 696106022

Hi,

is your UIView static or does it change a lot? If it's just a static view you could create a snapshot, extract it's CGImage and convert that into a TextureResource (https://developer.apple.com/documentation/realitykit/textureresource/3768851-generate) that could be mapped on whatever model you want.

In case it is animated I assume you could render that view with your desired update interval into a Metal Texture and then use the DrawableQueue API to update your material with the new texture: https://developer.apple.com/documentation/realitykit/textureresource/drawablequeue

Maybe this GitHub sample I've done could help you with the latter option: https://github.com/arthurschiller/realitykit-drawable-queue

Boost

Answer 1

arthurfromberlin OP

Nov ’21

Accepted Answer

Hi,

is your UIView static or does it change a lot? If it's just a static view you could create a snapshot, extract it's CGImage and convert that into a TextureResource (https://developer.apple.com/documentation/realitykit/textureresource/3768851-generate) that could be mapped on whatever model you want.

In case it is animated I assume you could render that view with your desired update interval into a Metal Texture and then use the DrawableQueue API to update your material with the new texture: https://developer.apple.com/documentation/realitykit/textureresource/drawablequeue

Maybe this GitHub sample I've done could help you with the latter option: https://github.com/arthurschiller/realitykit-drawable-queue

0

Answer 2

Fattie OP

Nov ’22

One thing, regarding interactions, you'd have to handle that completely manually in a custom way yourself. There is no meaningful relationship between the 3D world and UX interactions. Youi'd have to program it by hand on a case-by-case basis.

If you're familiart with the typical touch handling in 3D engines where you cast through the screen to see what you're touching, that would be the way to begin the process.

0