So the problem here seems to be that ScrollView will cancel the DragGesture when ScrollView takes over the movement. This is why we only see the onChanged called once. It seems to be briefly explained in the UIScrollView Overview, specifically this part: "If the user then drags their finger far enough before the timer elapses, the scroll view cancels any tracking in the subview and performs the scrolling itself."
The only solution here that worked for me is to wrap UIKit UIScrollView and then use UIScrollViewDelegate to hook into various drag events.
You can see here an example implementation which is not too hard to implement and extend.