RealityKit add gestures and animations to Entity/ModelEntity

I have a problem, I want to load a usdz file, place it and animate it in RealityKit. It should also be possible to apply standard gestures as done in

arView.installGestures(for: entity)

From what I know I have two choices to load an Entity, for example Apples "toy_biplane.usdz" (It also contains an skeletal animation).


I have

let entity = try! Entity.load(named: "toy_biplane")

or

let entity = try! Entity.loadModel(named: "toy_biplane")


when I use the first one, the usdz is loaded as an Entity, not ModelEntity. I can execute

entity.playAnimation(entity.repeat())

But I can't use

arView.installGestures(for: entity)

because it does not conform to HasCollision. I tried subclassing Entity and conforming to HasCollision and HasModel. It compiles but even if I call the generateCollisionShape(recursive: true) the gestures are not working.


So I tried the loadModel approach which returns an ModelEntity. There the arView.installGestures are working fine exactly as tried out. But when I want to play the animation, the airplane just rotates around my camer very weird.

I also tried loading them asynchronously, no success.


After many time of debugging I found out, that the Entity from the first approach contained many children from the usdz. Each of them is a part of the skeleton and has its own animation. Not so in the ModelEntity. The children property is an empty set and therefore the animation (e.g rotation of the propeller) is not applied to the skeletal element it belongs to but rather to the combined overall skeleton. Causing a rotation of the whole plane which is not what I want.


What am I doing wrong or is something of this unintended behaviour from RealityKit?


Thanks.

I've been looking at a similar problem over the last day or so. I had a look in SwiftStrikeTabletop and the in there they cast an Entity from a Reality Composer scene to Entity & HasCollision. Here's a sample I wrote :


// Setup AR Session
let arConfiguration = ARWorldTrackingConfiguration()
arConfiguration.planeDetection = .horizontal
arView.session.run(arConfiguration)
// Load the "Box" scene from the "Experience" Reality File
let boxScene = try! Experience.loadBox()
boxScene.generateCollisionShapes(recursive: true)
let box = boxScene.greybox as? Entity & HasCollision
arView.installGestures(for: box!)


This seems to work (although the Rotation gesture doesn't currently appear to respond).

+1


Hi hulbrich,


did you find a solution for this? davidjohnston´s solution (simply casting to Entity & HasCollsion) doesn´t work for me. I have the same problem, even though I am loading the models from the filesystem after downloading from a server:


• Entity.loadAsync() works for the embedded animation in a USDZ.

• ModelEntity.loadModelAsync() works for adding gestures and anything else I am doing with the objects in my scene. Animations are lost this way since apparently living under Entity not ModelEntity.
Any input is appreciated.

Thanks

Timo

Hello all,


Here is a detailed snippet that will get you the desired behavior. It will repeat all animations, while still allowing you to use the standard gestures to manipulate the model.


loadRequest = Entity.loadAsync(contentsOf: url).sink(receiveCompletion: { status in
            print(status)
        }) { entity in
           
            // Scaling entity to a reasonable size
            entity.setScale(SIMD3(repeating: 0.01), relativeTo: nil)
           
            // Creating parent ModelEntity
            let parentEntity = ModelEntity()
            parentEntity.addChild(entity)
           
            // Anchoring the entity and adding it to the scene
            let anchor = AnchorEntity(.plane(.horizontal, classification: .any, minimumBounds: .zero))
            anchor.addChild(parentEntity)
            self.arView.scene.addAnchor(anchor)
           
            // Playing availableAnimations on repeat
            entity.availableAnimations.forEach { entity.playAnimation($0.repeat()) }
           
            // Add a collision component to the parentEntity with a rough shape and appropriate offset for the model that it contains
            let entityBounds = entity.visualBounds(relativeTo: parentEntity)
            parentEntity.collision = CollisionComponent(shapes: [ShapeResource.generateBox(size: entityBounds.extents).offsetBy(translation: entityBounds.center)])
                       
            // installing gestures for the parentEntity
            self.arView.installGestures(for: parentEntity)
           
        }
       
        // Visualize the collision components
        arView.debugOptions = .showPhysics

The key here is to nest the loaded entity which has the animations inside of a ModelEntity, and to then give that ModelEntity an appropriate CollisionComponent based on the loaded model's bounds.

Unfortunately this doesn´t work for me 😟

By setting arView.debugOptions = .showPhysics, the gestures (scale, rotate, translate) work only for the collision shape itself (parentEntity) and not for 'entity'!


Thanks,

As a child of parentEntity, entity should appear to be scaled, rotated, and translated when parentEntity is scaled, rotated, and translated.

I'm getting the same results as Blueplanet. The parentEntity correctly responds to the gestures, but the child entities (in my case) just stay stuck to the same spot. Any chance this is a bug?


I was thinking maybe there is a delegate I need to watch and then individually update all the child entity positions, but seems like too much work for something simple.

Thanks! It works, even the rotation too. Though it's needed to

set Phisics for the object in Reality Composer.
No solution at all?
thank you gchiste, the code worked! I've been looking for this solutions for a few days now.

Thanks gchiste, your solution works!

Thank you gchiste. Your solution worked for me

Doesn't work for visionOS where there is no ARView :-/

RealityKit add gestures and animations to Entity/ModelEntity
 
 
Q