PKCanvasView: zoom+rotate gesture=ded performance

I'm trying to add rotation functionality to the PKCanvasView, so that rotation gesture rotates the drawing, I want it to be working together with zoom gesture too, which is already implemented in the PKCanvasView. Rotation works reasonably fine when rotation and zoom can't be performed simultaneously. The moment I enable simultaneous zoom+rotation performance of rotation gets noticably bad. Visually it looks like rotation only happens on noticable angles like 10,20,30.. degrees which looks choppy. Zoom at the same time looks smooth as before. I'm trying to understand why and how to fix this.

The rotation gesture handler is here:

    @objc
    func handleRotation(_ gesture: UIRotationGestureRecognizer) {
        guard let curView = gesture.view as? PKCanvasView else {return}
        
        if gesture.state == .began || gesture.state == .changed {
            let rotation = gesture.rotation
            let gestureCenter = gesture.location(in: curView)
            let center = CGPoint(x: gestureCenter.x / curView.zoomScale, y: gestureCenter.y / curView.zoomScale)
            let finalTransform = CGAffineTransform(translationX: -center.x, y: -center.y)
                .concatenating(CGAffineTransform(rotationAngle: rotation))
                .concatenating(CGAffineTransform(translationX: center.x, y: center.y))

            curView.drawing.transform(using: finalTransform)
            
            gesture.rotation = 0
        }
    }

which just constructs rotation matrix around rotation gesture center and calls curView.drawing.transform(using: finalTransform). Rotation matrix is correct I think because without simultaneous zoom+rotation it rotates everything as intended and smoothly under any zoom. (Just for clarity: in the rotation handler the matrix does only the rotation, zoom is done in pinch gesture handler implemented by the PKCanvasView. The zoomScale is used in estimations only to figure out the content coodrinate).

The handleRotation method is set as

let rotationGestureRecognizer = UIRotationGestureRecognizer(target: self, action: #selector(handleRotation))
rotationGestureRecognizer.delegate = self
canvasView.addGestureRecognizer(rotationGestureRecognizer)

The simultaneous handling of zoom+rotation I enable with this code:

func gestureRecognizer(_ gestureRecognizer: UIGestureRecognizer, shouldRecognizeSimultaneouslyWith otherGestureRecognizer: UIGestureRecognizer) -> Bool {
    return true
}

What I've tried:

Tried to nable simultaneous recognition only for zoom+rotation (without the pan gesture and possible others), the effect is the same. For example rotation+pan work smoothly together.

I've verified that the gesture handler is called very often, every 8-16 milliseconds (rotation matrix creation and curView.drawing.transform(using: finalTransform) is included in this time period) which should be enough for visually smooth rotation.

Other observations:

When zoom and rotation both are enabled the rotation sometimes gets completely stuck, zoom and pinch is responsive but rotation is not happening visually. Only when I release fingers from the screen the rotation unstucks and content is rotated as if it didn't stuck at all. During the "stuck" period the rotation handler is called with the same frequency it's just not being visually reflected.

Other thoughts:

It kind of looks like the tranformation I set in the rotation handler gets overwritten by transformation from the zoom handler. But pan+rotation works fine together which confuses this theory.