Using SwiftUI with multiple UIGestureRecognizers?

I am currently trying to write an app that uses a navigation similar to apps like Affinity, or Procreate, where you use a single touch/drag to draw, retouch and interact, and two-finger gestures to navigate on the canvas.

I am building the app using SwiftUI as the main framework and include UIKit as necessary.

Unfortunately, SwiftUI does not yet allow for complex gestures like the ones described, but UIKit usually does. So I reverted to using UIKit as my gesture recognisers instead of relying on SwiftUI gestures. This however is causing the issue, that only the topmost gesture recogniser will be called. I was hoping for recognition of multiple simultaneous gestures like demonstrated here, but unfortunately, SwiftUI seems to cause issues with the UIViewRepresentables.

Can someone help me figure out a solution to this?

Important: I am doing this with two separate views because in the long run, they will be used on different views. In the example, however, I have them on the same view for demonstration purposes.

Gist

Usage:

ZStack {
    DragGestureView { point in
        print("One Finger")
    } dragEndedCallback: {
        print("One Finger Ended")
    }

    TwoFingerNavigationView { point in
        viewStore.send(.dragChanged(point))
        print("Two Fingers")
    } dragEndedCallback: {
        viewStore.send(.dragEnded)
        print("Two Fingers Ended")
    } pinchedCallback: { value in
        viewStore.send(.magnificationChanged(value))
    } pinchEndedCallback: {
        viewStore.send(.magnificationEnded)
    }

    content()
        .position(viewStore.location)
        .scaleEffect(viewStore.scale * viewStore.offsetScale)
}