How to use drag gestures on objects with inverted normals?

I want to build a panorama sphere around the user. The idea is that the users can interact with this panorama, i.e. pan it around and select markers placed on it, like on a map.

So I set up a sphere that works like a skybox, and inverted its normal, which makes the material is inward facing, using this code I found online:

import Combine
import Foundation
import RealityKit
import SwiftUI

extension Entity {
    func addSkybox(for skybox: Skybox) {
        let subscription = TextureResource
            .loadAsync(named: skybox.imageName)
            .sink(receiveCompletion: { completion in
                switch completion {
                case .finished: break
                case let .failure(error): assertionFailure("\(error)")
                }
            }, receiveValue: { [weak self] texture in
                guard let self = self else { return }
                var material = UnlitMaterial()
                material.color = .init(texture: .init(texture))
                
                let sphere = ModelComponent(mesh: .generateSphere(radius: 5), materials: [material])
                self.components.set(sphere)
                /// flip sphere inside out so the texture is inside
                self.scale *= .init(x: -1, y: 1, z: 1)
                self.transform.translation += SIMD3(0.0, 1.0, 0.0)
            })
        components.set(Entity.SubscriptionComponent(subscription: subscription))
    }

    struct SubscriptionComponent: Component {
        var subscription: AnyCancellable
    }
}

This works fine and is looking awesome.

However, I can't get a gesture work on this.

If the sphere is "normally" oriented, i.e. the user drags it "from the outside", I can do it like this:

import RealityKit
import SwiftUI

struct ImmersiveMap: View {
    @State private var rotationAngle: Float = 0.0

    var body: some View {
        RealityView { content in
            let rootEntity = Entity()
            rootEntity.addSkybox(for: .worldmap)
            rootEntity.components.set(CollisionComponent(shapes: [.generateSphere(radius: 5)]))
            rootEntity.generateCollisionShapes(recursive: true)
            rootEntity.components.set(InputTargetComponent())

            content.add(rootEntity)
        }
        .gesture(DragGesture().targetedToAnyEntity().onChanged({ _ in
            log("drag gesture")
        }))

But if the user drags it from the inside (i.e. the negative x scale is in place), I get no drag events.

Is there a way to achieve this?