If you want to prevent this "physical" version of the drag gesture and restrict it to only the pinching and dragging variant, you can do so by changing the entity's InputTargetComponent to only use indirect interactions like so:
entity.components.set(InputTargetComponent(allowedInputTypes: .indirect))
More info:
Direct vs Indirect Gestures
InputTargetComponent Docs