CodeSign issue

We have a persistent CodeSign issue with the following error shown in the log when running an iOS build on a Jenkins node with macOS Catalina 10.15.7:
 
errSecInternalComponent
Command CodeSign failed with a nonzero exit code

We have a troubleshooting page and we tried all of the following, to no avail. The CodeSign issue is still there:

__________________________________________________
 
Check that all the required certificates are present in Keychain Access

Check that the access control of all certificates is correct
Check that the certificate for the corresponding territory has the right access control (either code sign or all apps)
If not, expand the certificate for the territory (arrow), select both the public and the private key underneath it and delete: you will be prompted to enter the password
Quit Keychain Access
Run the following command to import again the certificate with the right access controls:
sudo security import <certname>.p12 -P <password> -k /Library/Keychains/System.keychain -T /usr/bin/codesign

NB: It is not recommended to import a p12 with multiple certificate. Better have an approach where you import the certificates one by one, territory by territory.
NB: Sometimes, you need to repeat the process several times before you get the right access controls.

Now, check that the access control is correct in Keychain Access. Right click the private key and select Get Info. Select the Access Control tab in the info window.

Access control should be either Allow all applications to access this item or Confirm before allowing access selected with codesign listed in the applications allowed to access:


Make sure the certificates are trusted

If the certificates are NOT trusted, please install the certificate from Apple WorldWide Developer Certificate Authority:
https://developer.apple.com/certificationauthority/AppleWWDRCA.cer
If code signing still doesn't work, in Keychain Access, select File > Unlock All Keychains then File > Lock All Keychains.
Then type the following in a Terminal window to reboot the system:
sudo reboot


Check that all the required provisioning profiles are present for all territories for release-enterprise and release-appstore  and  that they match the certificates installed in Keychain Access
Cd into the ~ /Library/MobileDevice/Provisioning\ Profiles and check if the required provisioning profiles are present.
If not, either use scp to copy the missing profiles to this folder or use drag and drop from your local Finder to the remote Finder in Screen Sharing (drag and drop of files is a little fiddly, but it works).

Troubleshooting further: Check that all the required provisioning profiles are present for all territories for release-enterprise and release-appstore.
When running this command  in the  Terminal , it returns a list of valid entities:
security find-identity -vp codesigning 
For example, a list of 18 entities: 9 public key and the 9 corresponding private keys
If there are e.g 36 entities, there could be doubled. Check both the login and system keychains for duplicates. There should be 9 certificates listed in the system keychain.
Even if you have the right number of certificates in the system keychain and they access control is codesign or allow all applications, there could still be codesign issues. In that case, check the login keychain. Delete certificates that don’t need to be there. If it still doesn’t work, e.g the login keychain is empty and you have a code sign error, create a self signed certificate for Code Signing in the login keychain.
If code signing still doesn't work, in Keychain Access, select File > Unlock All Keychains then File > Lock All Keychains.
Then type the following in a Terminal window to reboot the system:
sudo reboot

__________________________________________________

What we haven’t tried yet is this:
security set-key-partition-list -S apple-tool:,apple:,codesign: -s -k "${PASSWORD}" "${KEYCHAIN}"

But we don’t know how to run it nor if this is the right thing to do.
 
Is this issue resolved for you? We are also seeing this issue that code signing fails when triggered from Jenkins but works when triggered locally.
Hi karl1971 and sethach
After upgrading to BigSur we are seeing exactly this issue as well! Has anyone of you found a resolution in the meantime?
Code Block bash
security unlock-keychain login.keychain 


This answer provided the final piece of the solution for me (which was the "set-key-partition-list" magic):

https://apple.stackexchange.com/a/285320

Specifically, my build script effectively does the following:

Set up the signing keys:

set -e
/usr/bin/security create-keychain -p $KC_PASSWORD packaging.keychain
/usr/bin/security set-keychain-settings packaging.keychain
/usr/bin/security unlock-keychain -p $KC_PASSWORD packaging.keychain
/usr/bin/security list-keychains -d user -s packaging.keychain $OTHER_KEYCHAINS_IF_ANY
/usr/bin/security import $KEY_DIR/dev-id-app.Certificates.p12 -A -k packaging.keychain -P $KEY_PASSWORD
/usr/bin/security import $KEY_DIR/dev-id-inst.Certificates.p12 -A -k packaging.keychain -P $KEY_PASSWORD
/usr/bin/security set-key-partition-list -S apple-tool:,apple: -k $KC_PASSWORD -t private packaging.keychain
/usr/bin/security list-keychains -d user
/usr/bin/security find-identity -v                        # this should list the two keys imported above

followed by the signing:

/usr/bin/codesign --force --sign "Developer ID Application: $DEV_NAME ($TEAM_ID)" --keychain packaging.keychain --deep --timestamp -o runtime -vvvv --entitlements gen.build/entitlements.plist gen.build/pkgroot/Applications/$APP_NAME.app/Contents/Resources/lib/$DYLIB_NAME.dylib
...(repeated for each *.so and *.dylib)...

/usr/bin/codesign --force --sign "Developer ID Application: $DEV_NAME ($TEAM_ID)" --keychain packaging.keychain --deep --timestamp -o runtime -vvvv --entitlements gen.build/entitlements.plist gen.build/pkgroot/Applications/$APP_NAME.app

/usr/bin/codesign --verify --deep --strict -vvvv gen.build/pkgroot/Applications/$APP_NAME.app     # this is optional

using an entitlements.plist that might look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.app-sandbox</key>
    <false/>
    <key>com.apple.security.device.bluetooth</key>
    <false/>
    <key>com.apple.security.files.user-selected.read-write</key>
    <true/>
    <key>com.apple.security.network.client</key>
    <true/>
    <key>com.apple.security.get-task-allow</key>
    <false/>
</dict>
</plist>

followed by the packaging (which also uses the keychain):

APP_BUNDLE_ID=com.example.my-app-name-lowercase   # put something suitable here, e.g. com.example.my-app-name-lowercase

pkgbuild --identifier $APP_BUNDLE_ID --version $VSN_MAJOR.$VSN_MINOR.$VSN_BUILDNUM --root gen.build/pkgroot/Applications --component-plist gen.build/components.plist --install-location /Applications gen.build/$APP_NAME.pkg

using a components.plist file that might look like this (but plug in your app name in place of $APP_NAME):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<array>
    <dict>
	    <key>BundleHasStrictIdentifier</key>
	    <true/>
	    <key>BundleIsRelocatable</key>
	    <false/>
	    <key>BundleIsVersionChecked</key>
	    <false/>
	    <key>BundleOverwriteAction</key>
	    <string>upgrade</string>
	    <key>RootRelativeBundlePath</key>
	    <string>$APP_NAME.app</string>
    </dict>
</array>
</plist>

followed by building the outer package for the installer:

productbuild --sign "Developer ID Installer: $DEV_NAME ($TEAM_ID)" --keychain packaging.keychain --distribution gen.build/distribution.plist --package-path gen.build --resources $APP_NAME/Resources gen.build/$APP_NAME-$VSN_MAJOR.$VSN_MINOR.$VSN_BUILDNUM-Mac-installer.pkg

using a distribution.plist file that might look like this (but plug in your app's identifier in place of $APP_BUNDLE_ID, app name in place of $APP_NAME, and minimum macOS version (e.g. 10.8) in place of $MACOSX_DEPLOYMENT_TARGET):

<?xml version="1.0" encoding="utf-8"?>
<installer-gui-script minSpecVersion="2">
<title>$APP_NAME</title>
<pkg-ref id="$APP_BUNDLE_ID"/>
<options customize="never" require-scripts="false"/>
<volume-check>
    <allowed-os-versions>
	<os-version min="$MACOSX_DEPLOYMENT_TARGET"/>
    </allowed-os-versions>
</volume-check>
<choices-outline>
    <line choice="default">
	<line choice="$APP_BUNDLE_ID"/>
    </line>
</choices-outline>
<choice id="default"/>
<choice id="$APP_BUNDLE_ID" visible="false">
    <pkg-ref id="$APP_BUNDLE_ID"/>
</choice>
<pkg-ref id="$APP_BUNDLE_ID">$APP_NAME.pkg</pkg-ref>
</installer-gui-script>

followed by telling the launch service that it should NOT remember that package location, to avoid weird and surprising behavior later:

/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -u `cd gen.build/pkgroot/Applications/$APP_NAME.app; /bin/pwd -P`

and then to submit for notarization:

xcrun altool --notarize-app --file gen.build/$APP_NAME-$VSN_MAJOR.$VSN_MINOR.$VSN_BUILDNUM-Mac-installer.pkg --username $NOTARIZATION_USERNAME --password $NOTARIZATION_PASSWORD --asc-provider $TEAM_ID --primary-bundle-id $APP_BUNDLE_ID

Note that the notarization password is an app-specific password specifically for notarizing (not specific to the app that you are notarizing, despite the name!), as described here:

https://stackoverflow.com/questions/56890749/macos-notarize-in-script

My script captures the RequestUUID (e.g. b8c31cd3-5f41-4adc-b062-da7eb65fc650) from the "xcrun altool" output, and then queries for the result every 10 seconds:

xcrun altool --notarization-info $REQUEST_UUID --username $NOTARIZATION_USERNAME --password $NOTARIZATION_PASSWORD --asc-provider $TEAM_ID

And then once it's done, it fetches and checks the log, then finally staples the notarization result to the package, so that it can be installed when not connected to the internet (since otherwise the installer checks with Apple's servers to see if the package has been notarized):

xcrun stapler staple gen.build/$APP_NAME-$VSN_MAJOR.$VSN_MINOR.$VSN_BUILDNUM-Mac-installer.pkg

The result is a Mac native .pkg file that can be downloaded and installed directly (by just double-clicking it in Finder) and cleanly, on everything from Mac OS X 10.7 Lion (and probably earlier) up to macOS 12.2+ Monterey, with none of that mount-the-DMG-and-drag-the-icon silliness. And the whole build process described above works while logged in via ssh - no GUI required.

CodeSign issue
 
 
Q