I ditched this approach completely and went with a solution that uses scheduled push notifications to get location updates
Wallman94
Users receive reputation points by contributing quality content to the forums.
Posts include post and replies published on the forums.
Post authors can mark replies as Accepted to indicate successful solutions.
Apple Developer Forums admins can mark replies as Apple Recommended to indicate an approved solution
Post
Replies
Boosts
Views
Activity
https://stackoverflow.com/questions/77773014/change-swiftui-map-position-without-resetting-distance
The best way I found for this was to send scheduled background push notifications from a server and update location when those are received.
I am also using a CLBackgroundActivitySession