Children of a dragged entity get left behind when moving slowly

Hey friends, I'm using a drag gesture to rotate a parent object that contains several child colliders. When I drag slowly, sometimes the child colliders don't rotate along with the parent. Any help would be appreciated, thanks!

.gesture(
    DragGesture()
        .targetedToAnyEntity()
        .onChanged { value in
            let startLocation = value.convert(value.startLocation3D, from: .local, to: .scene)
            let currentLocation = value.convert(value.location3D, from: .local, to: .scene)
            let delta = currentLocation - startLocation
            let spinX = Double(delta.y)
            let spinY = Double(delta.x)

            let pitch = Transform(pitch: Float(spinX * -1)).matrix
            let roll = Transform(roll: Float(spinY * -1)).matrix
            value.entity.transform.matrix = roll * pitch
        })

I isolated the issue to be if a child has a some rotation already applied by default then the problem happends. I just removed the rotation to the child colliders and it worked fine but it does mean it it's going to be a hassle making compound colliders.

Children of a dragged entity get left behind when moving slowly
 
 
Q