If you're having trouble fetching a package, you may need to manually delete the SPM cache files located at:
~/Library/Caches/org.swift.swiftpm/
After that, update to the latest package versions.
Note: This will delete the clone history of all your SPM libraries, so re-cloning them takes some time. However, this should resolve my fetching issue.