I'm getting the same exact symptoms with my app that has a dependency on https://github.com/mz2/Carpaccio -- I tried your suggested solution of setting the architectures by hand, but that didn't on its own help in my case. Also, tried clearing derived data and restarted Xcode to make sure that packages were fetched and built from a clean state.
The extra curious fact is that I can literally see both the ARM64 and Intel versions of the Swift module inside the derived data, and whereas under Carpaccio.swiftmodule I can see both ARM64 and Intel versions:
./Sashimi-fpvhysoehxifhxcfgdnzcygfncpg/Build/Products/Debug/Carpaccio.o
./Sashimi-fpvhysoehxifhxcfgdnzcygfncpg/Build/Products/Debug/Carpaccio.swiftmodule
./Sashimi-fpvhysoehxifhxcfgdnzcygfncpg/Build/Products/Debug/Carpaccio.swiftmodule/arm64-apple-macos.swiftdoc
./Sashimi-fpvhysoehxifhxcfgdnzcygfncpg/Build/Products/Debug/Carpaccio.swiftmodule/arm64.swiftmodule
./Sashimi-fpvhysoehxifhxcfgdnzcygfncpg/Build/Products/Debug/Carpaccio.swiftmodule/arm64-apple-macos.swiftmodule
./Sashimi-fpvhysoehxifhxcfgdnzcygfncpg/Build/Products/Debug/Carpaccio.swiftmodule/Project
./Sashimi-fpvhysoehxifhxcfgdnzcygfncpg/Build/Products/Debug/Carpaccio.swiftmodule/Project/arm64-apple-macos.swiftsourceinfo
./Sashimi-fpvhysoehxifhxcfgdnzcygfncpg/Build/Products/Debug/Carpaccio.swiftmodule/Project/arm64.swiftsourceinfo
./Sashimi-fpvhysoehxifhxcfgdnzcygfncpg/Build/Products/Debug/Carpaccio.swiftmodule/arm64.swiftdoc
... however, the Carpaccio.o that's built only has the active architecture there:
./DerivedData/Sashimi-fpvhysoehxifhxcfgdnzcygfncpg/Build/Products/Debug/Carpaccio.o
I've systematically turned off "Build Active Architecture Only" for all my app's targets.
Something that may be explaining a difference between my and your experience is that for me this depended on library is linked into a framework target, not the "main" app target?
I also tried checking out the package in question separately, and build it with:
xcrun swift build -c release --arch arm64 --arch x86_64
... I do then indeed see both expected platforms there:
lipo -info .build/apple/Products/Release/Carpaccio.o
Architectures in the fat file: .build/apple/Products/Release/Carpaccio.o are: x86_64 arm64
Finally, got this working by setting the list of architectures by hand as you suggest, and then toggling "Any Mac (Apple Silicon, Intel)" instead of "My Mac" as the active scheme!