Work around I did was
I used a ScrollView where in I had a state if it is currently scrolling
instead of DragGesture's .onChanged(), I used .updating() (this seems to be ran first before onChanged)
I checked the ScrollView's scrolling state, if it's being scrolled, don't update the View's drag offset else update it.