I fixed it with using Swift Package Manager and unlink the cocoapods integration..
Simply delete the .xcworkspace, the Podfile (plus the Podfile.lock) and the Pods Folder and add Firebase via Swift Package Manager as described here https://github.com/firebase/firebase-ios-sdk/blob/master/SwiftPackageManager.md.