This is so strange I'm at a loss. Background: I am a DevOps engineer validating that our iOS build scripts can run via Bamboo on new Mac mini M1 machines. Out of the box, these machines are running Monterey, while our in-use machines are all still on Big Sur.
Tldr; After modifying security permissions for our custom CI keychain so it's accessible by Xcode's CLI tools (specifically, the codesign and security utilities) the "xcodebuild archive" command we run fails with NO logged errors, causing bamboo to fail the build. However, the "failed" command produces a VALID xcarchive file. If I force the archive step to exit with a zero status, the subsequent "xcodebuild exportArchive" step successfully generates a valid .ipa that can be submitted to Testflight and run as expected. I've never seen anything like this.
Details: Initially, our builds were failing with codesigning errors related to the CLI tools interacting with our custom keychain (we use a custom keychain so Bamboo only has access to the codesigning assets it needs, this is best practice, at least up til now.) We could build on the new Mac mini with Xcode IDE just fine, and we could even build from the commandline on the new Mac mini. However, running the same commands via ssh to the new Mac mini from my local machine produced the same error bamboo was getting.
We were getting errors like this:
23-Mar-2022 13:35:58 Warning: unable to build chain to self-signed root for signer "Apple Development: Elizabeth Goldstein (XXXXXXX)"
23-Mar-2022 13:35:58 /Users/bamboo/agent-home/xml-data/build-dir/path/to/Build/Products/QA-iphoneos/Shared.framework: errSecInternalComponent
23-Mar-2022 13:35:58 Command CodeSign failed with a nonzero exit code
We were already unlocking the keychain as part of the build script, and the private key was set to "Allow all access" for outside applications (two common issues that might raise this error.)
Finally, I rebuilt the system keychain by first deleting then manually re-importing Apple's root signing certificates from terminal like this:
sudo security import ./AppleWWDRCA.cer -k /Library/Keychains/System.keychain -T /usr/bin/codesign -T /usr/bin/security -T /usr/bin/productbuild
sudo security import ./AppleWWDRCAG3.cer -k /Library/Keychains/System.keychain -T /usr/bin/codesign -T /usr/bin/security -T /usr/bin/productbuild
Then I deleted and re-imported my company's signing assets into our custom keychain like this:
security import ./appstore-cert.p12 -k /Users/bamboo/Library/Keychains/our-ci.keychain-db -T /usr/bin/codesign -T /usr/bin/security -T /usr/bin/productbuild
security import ./dev-cert.p12 -k /Users/bamboo/Library/Keychains/our-ci.keychain-db -T /usr/bin/codesign -T /usr/bin/security -T /usr/bin/productbuild
So now, there are zero errors in the log that I can find, but bamboo still fails when we run these commands:
/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -workspace OUR-APP.xcworkspace -scheme iPhoneApp-QA clean build -configuration QA -derivedDataPath output -sdk iphoneos OTHER_CFLAGS="-fembed-bitcode" ENABLE_BITCODE=YES BITCODE_GENERATION_MODE=bitcode
/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -workspace OUR-APP.xcworkspace -scheme iPhoneApp-QA -sdk iphoneos -configuration QA -derivedDataPath output archive -archivePath build.xcarchive OTHER_CFLAGS="-fembed-bitcode" ENABLE_BITCODE=YES BITCODE_GENERATION_MODE=bitcode
If I change it to
/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -workspace OUR-APP.xcworkspace -scheme iPhoneApp-QA -sdk iphoneos -configuration QA -derivedDataPath output archive -archivePath build.xcarchive OTHER_CFLAGS="-fembed-bitcode" ENABLE_BITCODE=YES BITCODE_GENERATION_MODE=bitcode || exit 0
The final exportArchive command generates a valid .ipa
Any help or ideas would be greatly appreciated, since of course that's not a viable workaround, nor is pinning our OS to Big Sur forever, and we need to scale up our CI system. Thank you for reading this far!
Thanks Matt, it took us over a month to get there but your reply pointed us in the right direction. For anyone struggling with this, it turns out NONE of the work in my original post was necessary, even though it did make the errors in our Bamboo xcodebuild logs go away.
THE ANSWER
-
I had to change the bamboo launch process on our Mac mini build nodes from a daemon to an agent AND ALSO move the plist file into the Bamboo user context instead of the default System context, ex: /Users/USER_NAME/Library/LaunchAgents) instead of the system (/Library/LaunchAgents/)
-
We were trying this with both brand new M1 and Intel nodes, and for the M1 node we also need to set up Xcode to open via Rosetta. This doesn't seem necessary on the Intel node.
I cried real tears over this so I hope it saves someone else the grief!