Guess everyone building a fat lib will have this problem. The arm64 architecture for the simulator (Apple Silicon) conflicts with the arm64 built for a device and LIPO doesn't want to merge. In my build, I just force the architecture to x86_64 and i386 when building for the sim. Now, just need to figure out cocoapods with this changes...
Edit: https://stackoverflow.com/questions/63607158/xcode-12-building-for-ios-simulator-but-linking-in-object-file-built-for-ios