Distributing a command-line binary executable (2020 edition)?

Apple expects that all ARM executables are code signed. I have no problem notarizing my graphical applications as dmg files. However, the terminal applications are another story. Over the years, it seems that specifics have changed, and Apple's documentation is vague and scattered.

I really think that both Apple engineers and 3rd party developers would benefit if Apple could provide a concrete worked example of how they would like command line applications created without Xcode to be signed, notarized and distributed. While I think the intentions for code signing are noble, the experience as a 3rd party developer is attempting to comply with code signing is extremely frustrating. Having a tool that simplifies the process and guides users through the required certificates, signing, notarization, etc. would really help, in particular for people like me who are scientists not professional developers.

For my own specific example, I am trying to package a simple open source application .

When I submit this for notarization, I get a log file that says:
Code Block
The binary is not signed with a valid Developer ID certificate

However, running
Code Block
codesign --verify --verbose --strict niimath

suggests this is not the case.

Below is my script to try to submit this. Only the first four lines are modified.

Code Block
CODE_SIGN_SIGNATURE="Developer ID Application: My Name"
CODE_INSTALL_SIGNATURE="3rd Party Mac Developer Installer: My Name (12ABCDEFGH)"
APPLE_ID_USER=myname@gmail.com
APP_SPECIFIC_PASSWORD=abcd-efgh-klmn-opqr
APP_NAME=niimath
cd ~/src/niimath/src
gcc -sectcreate TEXT info_plist Info.plist -O3 -DHAVE_ZLIB -o niimathX86 niimath.c bw.c core.c tensor.c core32.c core64.c niftilib/nifti2_io.c znzlib/znzlib.c -I./niftilib -I./znzlib -lm -lz -target x86_64-apple-macos10.12 -mmacosx-version-min=10.12
strip niimathX86
gcc -sectcreate TEXT info_plist Info.plist -O3 -DHAVE_ZLIB -o niimathARM niimath.c bw.c core.c tensor.c core32.c core64.c niftilib/nifti2_io.c znzlib/znzlib.c -I./niftilib -I./znzlib -lm -lz -target arm64-apple-macos11 -mmacosx-version-min=11.0
strip niimathARM
lipo -create -output niimath niimathX86 niimathARM
rm niimathX86
rm niimathARM
BIN_DIR=../../bin
mkdir -p $BIN_DIR
cp $APP_NAME $BIN_DIR/$APP_NAME
cd $BIN_DIR
# Verify the Info.plist was embedded in the executable during linking
echo "Verifying Info.plist"
launchctl plist $APP_NAME
# Codesign the executable by enabling the hardened runtime (--options=runtime) and include a timestamp (--timestamp)
echo "Code signing..."
#codesign --timestamp --options=runtime -s "$CODE_SIGN_SIGNATURE" -v $APP_NAME
#codesign -vvv --force --deep --strict --options=runtime --timestamp -s "$CODE_SIGN_SIGNATURE" ${APP_NAME}
#codesign -vvvv --deep --strict ${APP_NAME}
#codesign -dv --verbose=4 ${APP_NAME}
#codesign -dv --verbose=4 ${APP_NAME}
# https://developer.apple.com/forums/thread/120989
codesign -vvv --force --strict --options=runtime --timestamp -s "$CODE_SIGN_SIGNATURE" ${APP_NAME}
codesign --verify --verbose --strict $APP_NAME
#productbuild --identifier "com.mricro.niimath.pkg" --sign "$CODE_SIGN_SIGNATURE" --timestamp --root /tmp/niimath / niimath.pkg
#productbuild --sign "$CODE_SIGN_SIGNATURE" --component niimath /Applications ./
#productbuild --identifier "com.mricro.pkg" --sign "$CODE_SIGN_SIGNATURE" --timestamp --root /bin / ghostscript64.pkg
#CODE_SIGN_SIGNATURE="3rd Party Mac Developer Installer: Christopher Rorden (68BQDQS28R)"
echo productbuild --identifier "com.mricro.niimath" --sign "$CODE_INSTALL_SIGNATURE" --timestamp --root ./ /usr/local/bin ${APP_NAME}_macOS.pkg
productbuild --identifier "com.mricro.niimath" --sign "$CODE_INSTALL_SIGNATURE" --timestamp --root ./ /usr/local/bin ${APP_NAME}_macOS.pkg
#productsign --sign "$CODE_INSTALL_SIGNATURE" "${APP_NAME}.pkg" "${APP_NAME}_macOS.pkg"
# Notarizing with Apple...
echo "Uploading..."
xcrun altool --notarize-app -t osx --file ${APP_NAME}_macOS.pkg --primary-bundle-id com.mricro.${APP_NAME} -u $APPLE_ID_USER -p $APP_SPECIFIC_PASSWORD --output-format xml > upload_log_file.txt
# WARNING: if there is a 'product-errors' key in upload_log_file.txt something went wrong
# we could parse it here and bail but not sure how to check for keys existing with PListBuddy
# /usr/libexec/PlistBuddy -c "Print :product-errors:0:message" upload_log_file.txt
# now we need to query apple's server to the status of notarization
# when the "xcrun altool --notarize-app" command is finished the output plist
# will contain a notarization-upload->RequestUUID key which we can use to check status
echo "Checking status..."
sleep 20
REQUEST_UUID=`/usr/libexec/PlistBuddy -c "Print :notarization-upload:RequestUUID" upload_log_file.txt`
while true; do
xcrun altool --notarization-info $REQUEST_UUID -u $APPLE_ID_USER -p $APP_SPECIFIC_PASSWORD --output-format xml > request_log_file.txt
# parse the request plist for the notarization-info->Status Code key which will
# be set to "success" if the package was notarized
STATUS=`/usr/libexec/PlistBuddy -c "Print :notarization-info:Status" request_log_file.txt`
if [ "$STATUS" != "in progress" ]; then
break
fi
# echo $STATUS
echo "$STATUS"
sleep 10
done
# download the log file to view any issues
/usr/bin/curl -o log_file.txt `/usr/libexec/PlistBuddy -c "Print :notarization-info:LogFileURL" request_log_file.txt`
# staple
echo "Stapling..."
xcrun stapler staple ${APP_NAME}_macOS.pkg
xcrun stapler validate ${APP_NAME}_macOS.pkg
open log_file.txt


Answered by Etresoft in 652135022

I made the changes you suggested and that did not fix anything.

Then maybe the problem is something else. Did you try building and notarizing it without the script first?

Can you provide a link to any script for notarizing a basic "helloworld" terminal application not using Xcode?

Here is something I posted on the developer forums a while back showing how to notarize Ghostscript.

https://developer.apple.com/forums/thread/130379

However, when I tried that with your tool, it failed. I don't build many packages anymore and my instructions for building the package in that thread are wrong. So I re-did it using your own tool.

Code Block
// I don't have the Info.plist file, so I'm omitting that.
clang -O3 -DHAVE_ZLIB -o niimathX86 niimath.c bw.c core.c tensor.c core32.c core64.c niftilib/nifti2_io.c znzlib/znzlib.c -I./niftilib -I./znzlib -lm -lz -target x86_64-apple-macos10.12 -mmacosx-version-min=10.12
clang -O3 -DHAVE_ZLIB -o niimathARM niimath.c bw.c core.c tensor.c core32.c core64.c niftilib/nifti2_io.c znzlib/znzlib.c -I./niftilib -I./znzlib -lm -lz -target arm64-apple-macos11 -mmacosx-version-min=11.0
// Create the universal binary.
lipo -create -output niimath niimathX86 niimathARM
// Create a staging area for the installer package.
mkdir -p usr/local/bin
// Copy the binary into the staging area.
cp niimath usr/local/bin
// Sign the binary.
codesign --timestamp --options=runtime -s "Developer ID Application: ***" -v usr/local/bin/niimath
// Build the package.
pkgbuild --identifier "com.example.niimath.pkg" --sign "Developer ID Installer: ***" --timestamp --root ./tmp/usr/local --install-location /usr/local/ niimath.pkg
// Submit the package to the notarization service.
xcrun altool --notarize-app --primary-bundle-id "com.example.niimath.pkg" --username "user@example.com" --password "xyz" --file niimath.pkg
// Wait 1-5 minutes.
// Staple the package.
xcrun stapler staple niimath.pkg
// Optionally make a zip for the package.
ditto -c -k --sequesterRsrc niimath.pkg niimath.zip
// Upload the download file.
scp niimath.zip user@example.com:path/to/download/link/niimath.zip
// Download the file using Safari to verify the notarization worked.


The tricky part was building the package. I had originally used "productbuild" too, but I couldn't get that to install on Big Sur. It is very picky about the package, structure, and install path. You have to use "pkgbuild" to get those setup properly. Maybe it is possible to use "productbuild" with different settings. I don't know.

Another tricky part is the verification. You will find no limit to the number of clever command-line verification tests posted here in the forums. Strangely enough, all of those clever command-line verification tests are failing because they are posting here in the forums. There is only one way to verify that notarization is working, and that is to upload the binary and then download it with Safari and install. Preferably, do this inside a virtual machine where you can test in a factory-fresh environment without developer tools, hacked-up SIP, 3rd party Qt libs, homebrew, anaconda, and god knows what else.

I omitted any kind of checking or verification. That's the thing about Notarization. Sometimes people reuse to listen and flail away at it for weeks or months. But once you figure out what you are doing wrong and fix it, it never, ever fails. So rather than trying to parse the output, just fix the problem and assume it always succeeds after 5 minutes. You might get one or two failures a year when the notarization servers are slow or down. Otherwise, it always works.

Only the first four lines are modified. 

Well that seems to be where the problem is. Change:

3rd Party Mac Developer Installer: My Name (12ABCDEFGH)

to

Developer ID Installer: My Name

I don't know if any of the rest of your script is correct or not. I don't use "--force --strict" when signing on the command line. It doesn't look like it should break anything. But those commented-out lines above have me worried. Those concrete working examples you ask for have been provided, by Apple and others, over and over again.

But otherwise, the script looks OK. 20 seconds is a bit optimistic. A minute might be better.

In general, the whole process was designed for developers building iPhone, and later, Mac apps. You could always wrap your project into a Mac app, or maybe even an iOS app. Then you can just click a button in Xcode.
Thanks for the quick response. I made the changes you suggested and that did not fix anything. Can you provide a link to any script for notarizing a basic "helloworld" terminal application not using Xcode? I really want to avoid the Xcode route, as it would be nice to port quite a few tools that are popular in my field.

Accepted Answer

I made the changes you suggested and that did not fix anything.

Then maybe the problem is something else. Did you try building and notarizing it without the script first?

Can you provide a link to any script for notarizing a basic "helloworld" terminal application not using Xcode?

Here is something I posted on the developer forums a while back showing how to notarize Ghostscript.

https://developer.apple.com/forums/thread/130379

However, when I tried that with your tool, it failed. I don't build many packages anymore and my instructions for building the package in that thread are wrong. So I re-did it using your own tool.

Code Block
// I don't have the Info.plist file, so I'm omitting that.
clang -O3 -DHAVE_ZLIB -o niimathX86 niimath.c bw.c core.c tensor.c core32.c core64.c niftilib/nifti2_io.c znzlib/znzlib.c -I./niftilib -I./znzlib -lm -lz -target x86_64-apple-macos10.12 -mmacosx-version-min=10.12
clang -O3 -DHAVE_ZLIB -o niimathARM niimath.c bw.c core.c tensor.c core32.c core64.c niftilib/nifti2_io.c znzlib/znzlib.c -I./niftilib -I./znzlib -lm -lz -target arm64-apple-macos11 -mmacosx-version-min=11.0
// Create the universal binary.
lipo -create -output niimath niimathX86 niimathARM
// Create a staging area for the installer package.
mkdir -p usr/local/bin
// Copy the binary into the staging area.
cp niimath usr/local/bin
// Sign the binary.
codesign --timestamp --options=runtime -s "Developer ID Application: ***" -v usr/local/bin/niimath
// Build the package.
pkgbuild --identifier "com.example.niimath.pkg" --sign "Developer ID Installer: ***" --timestamp --root ./tmp/usr/local --install-location /usr/local/ niimath.pkg
// Submit the package to the notarization service.
xcrun altool --notarize-app --primary-bundle-id "com.example.niimath.pkg" --username "user@example.com" --password "xyz" --file niimath.pkg
// Wait 1-5 minutes.
// Staple the package.
xcrun stapler staple niimath.pkg
// Optionally make a zip for the package.
ditto -c -k --sequesterRsrc niimath.pkg niimath.zip
// Upload the download file.
scp niimath.zip user@example.com:path/to/download/link/niimath.zip
// Download the file using Safari to verify the notarization worked.


The tricky part was building the package. I had originally used "productbuild" too, but I couldn't get that to install on Big Sur. It is very picky about the package, structure, and install path. You have to use "pkgbuild" to get those setup properly. Maybe it is possible to use "productbuild" with different settings. I don't know.

Another tricky part is the verification. You will find no limit to the number of clever command-line verification tests posted here in the forums. Strangely enough, all of those clever command-line verification tests are failing because they are posting here in the forums. There is only one way to verify that notarization is working, and that is to upload the binary and then download it with Safari and install. Preferably, do this inside a virtual machine where you can test in a factory-fresh environment without developer tools, hacked-up SIP, 3rd party Qt libs, homebrew, anaconda, and god knows what else.

I omitted any kind of checking or verification. That's the thing about Notarization. Sometimes people reuse to listen and flail away at it for weeks or months. But once you figure out what you are doing wrong and fix it, it never, ever fails. So rather than trying to parse the output, just fix the problem and assume it always succeeds after 5 minutes. You might get one or two failures a year when the notarization servers are slow or down. Otherwise, it always works.
@Etresoft thanks for your solution. This works! There is a tiny typo: "--root ./tmp/usr/local" should read "--root usr/local". I still think Apple could help out the community a lot by providing a graphical tool to help the many cross platform projects that do not use Xcode. There are a lot of moving parts, and the feedback is opaque. It seems like details change between OS releases, so search the web for answers causes issues with solutions that used to work.

There is a tiny typo: "--root ./tmp/usr/local" should read "--root usr/local".

Sorry about that. I think that is an artifact from my difficulty with productbuild vs. pkgbuild. The productbuild tool kept generating an installer that wouldn't work, so I tried various different root paths before I figured out that pkgbuild could just set the correct value.

I still think Apple could help out the community a lot by providing a graphical tool to help the many cross platform projects that do not use Xcode. There are a lot of moving parts, and the feedback is opaque. It seems like details change between OS releases, so search the web for answers causes issues with solutions that used to work. 

You can always use Feedback Assistant to submit an enhancement request.

Generally I would not recommend searching the web for answers. They are usually wrong. My example is a good case in point. It was always wrong. I just didn't notice it. This problem is worse where Macs are involved. There are far fewer people using Macs for open source projects. The few people that do often don't understand the fundamental differences between Macs, other Unixes, and Linux. The vast majority of Mac support in open source projects are little more than crude hacks done years ago and never touched since.


Distributing a command-line binary executable (2020 edition)?
 
 
Q