I haven't found a working method of removing the package in XCode, but I have managed to circumvent this issue by removing the package through the .xcodeproj file.
What I did was as follows:
Saved & closed my project
Opened the .xcodeproj in VSCode
Opened the project.pbxproj file & deleted references to the package
Saved the file, reopened XCode, and the package was gone.
If I recall correctly, a grep command only showed occurrences of the package name in project.pbxproj, so while it's possible this method doesn't remove all occurances of the package from the app, this seemed to work well for me.
Personally, I was trying to replace a local package with a new version of the package, and removal/replacing both caused the crash. Doing this removed the "project has an existing dependency" message and allowed me to directly re-add the package.
Hope this helps! :)