Jenkins cannot access keychain item for notarization

We have a mac (10.15.1) build server setup running Xcode 11 and Jenkins for CI that I want to use for notarization on release builds. However, I am unable to access the keychain item from the build server (which is started with the machine via launchd).


If I login to the mac as the jenkins user and run the same commands, everything works fine as long as I unlock the keychain. But I get a keychain error when I try to run it as part of a build job, even if I try to unlock they keychain first as part of the job:


altool[3090:62816] *** Error: The keychain returned error code: -25291. Failed to get the password for the keychain item 'blah'.

altool[3096:62956] *** Error: Use the -u option to specify the account in the keychain item 'blah'. Failed to get the password for the keychain item 'blah'.


Any tips? Jenkins seems to be able to access the syste keychain without issue, but for whatever reason I cannot move or add the password item to it; it can only seem to be in the login keychain. I really don't want to put a password in plaintext as part of the build jobl; surely there is a better way?

Accepted Reply

As far as I know,

altool
does do anything special with the keychain. I did some poking around with LLDB and, AFAICT, it’s just calling
SecItemCopyMatching
in the obvious way.

The most likely cause of this problem is the way that Jenkins is running its jobs. It’s very common for UNIX-y software to partially switch their execution context, that is, switch the traditional BSD execution context (the UIDs and GIDs, basically) but not the macOS-specific execution context (the Mach bootstrap namespace and the security context). See Technote 2083 Daemons and Agents for more background on this than is likely to be helpful.

As to what you can do about this, I think your first suggestion, using the System keychain, is likely to be the most successful. I tried this here in my office, and it seems to work.

Specifically:

  1. On 10.15 with Xcode 11.1 installed, I used System Preferences to create

    qnotarisation
    user. I made it a standard user (that is, not an admin).
  2. I logged in as that user, just to make sure the account was set up correctly.

  3. I then logged out, doing the rest of this work with the Mac’s GUI setting at the login screen.

  4. I used SSH to log in as an admin user.

  5. I created a small shell script to act as my daemon:

    $ cat /Library/PrivilegedHelperTools/altest.sh 
    #! /bin/sh
    
    
    xcrun altool --list-providers -u uuu -p @keychain:sss

    .

  6. I wrapped that in a

    launchd
    job:
    $ cat /Library/LaunchDaemons/com.example.altest.plist 
    …
    <dict>
      <key>Label</key>
      <string>com.example.altest</string>
      <key>ProgramArguments</key>
      <array>
        <string>/Library/PrivilegedHelperTools/altest.sh</string>
      </array>
      <key>UserName</key>
      <string>qnotarisation</string>
      <key>StandardOutPath</key>
      <string>/Users/qnotarisation/altest-stdout.txt</string>
      <key>StandardErrorPath</key>
      <string>/Users/qnotarisation/altest-stderr.txt</string>
    </dict>
    </plist>

    .

  7. I loaded that as a

    launchd
    daemon:
    $ sudo launchctl load /Library/LaunchDaemons/com.example.altest.plist

    .

  8. I used the

    security
    tool to created a generic password in System keychain:
    $ sudo security add-generic-password -a uuu -s sss -w ppp -A /Library/Keychains/System.keychain

    IMPORTANT This uses the

    -A
    argument to set the ACL wide open. On a real machine, you’ll probably want to use the
    -T
    argument.
  9. I started my launchd job:

    $ sudo launchctl start com.example.altest

    .

  10. After waiting a few seconds, I looked at the job’s output:

    $ cat /Users/qnotarisation/altest-stderr.txt 2019-11-27 02:58:54.910 altool[1538:23135] *** Error: Failed to retrieve providers info: (     "Error Domain=ITunesConnectionOperationErrorDomain Code=-20209 \"This Apple ID has been locked for security reasons. Visit iForgot to reset your account (https://iforgot.apple.com).\" UserInfo={NSLocalizedRecoverySuggestion=This Apple ID has been locked for security reasons. Visit iForgot to reset your account (https://iforgot.apple.com)., NSLocalizedDescription=This Apple ID has been locked for security reasons. Visit iForgot to reset your account (https://iforgot.apple.com)., NSLocalizedFailureReason=App Store operation failed.}" )

    .

As you can see, it found the password and attempting to contact the notary service (that failed, of course, because the account and password are completely bogus).

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

ps DTS is closed 25…29 Nov in observance of the US Thanksgiving holiday.

Replies

The

altool
man page lists four different ways to support a password:
  • On the command line

  • From the keychain, using

    @keychain
  • From an environment variable, using

    @env
  • From stdin

I’m not familiar with the Jenkins runtime environment but I figure that it should support at least one of those, eh?

Oh, one other thing. Modern (Xcode 11) versions of

altool
support the snazzy new
--store-password-in-keychain-item
command. If you run a one-off Jenkins job that stores the password that way, I expect that it’ll put it in a place where future Jenkins jobs can access it.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

While I'm sure the non-keychain options would work fine, I'd like to use the keychain if possible as it is is much more secure way to store the account password.


--store-password-in-keychain-item is how I was attempting to add the item to my keychain. Interestingly, if I try to run this from the build server, I get no errors, but I also don't see any entry for it in the keychain app:


Running as SYSTEM

Building in workspace /Users/Shared/Jenkins/Home/workspace/CredentialTest

[CredentialTest] $ /bin/sh -xe /Users/Shared/Jenkins/tmp/jenkins4349644463532595359.sh

+ xcrun altool --store-password-in-keychain-item someItem -u some@email.com -p '123456'

+ xcrun altool --notarization-info ab123123-123a-123a-123a-123abc123abc -u some@email.com -p @keychain:someItem

2019-11-26 10:51:06.918 altool[5777:830527] *** Error: The keychain returned error code: -25291. Failed to get the password for the keychain item 'someItem'.


If I login as the jenkins user and add the keychain item the same way, I get a success message that I don't see above ("Stored the password in the keychain item...", and I then see it in my local items keychain.


I suspect this has to do with Jenkins being started as a LaunchDaemon.


https://stackoverflow.com/questions/47582989/launchd-not-able-to-access-mac-os-keychains


However, SessionCreate is already set to true in /Library/LaunchDaemons/org.jenkins-ci.plist so I'm really not sure what the issue could be or how to troubleshoot further from here.

As far as I know,

altool
does do anything special with the keychain. I did some poking around with LLDB and, AFAICT, it’s just calling
SecItemCopyMatching
in the obvious way.

The most likely cause of this problem is the way that Jenkins is running its jobs. It’s very common for UNIX-y software to partially switch their execution context, that is, switch the traditional BSD execution context (the UIDs and GIDs, basically) but not the macOS-specific execution context (the Mach bootstrap namespace and the security context). See Technote 2083 Daemons and Agents for more background on this than is likely to be helpful.

As to what you can do about this, I think your first suggestion, using the System keychain, is likely to be the most successful. I tried this here in my office, and it seems to work.

Specifically:

  1. On 10.15 with Xcode 11.1 installed, I used System Preferences to create

    qnotarisation
    user. I made it a standard user (that is, not an admin).
  2. I logged in as that user, just to make sure the account was set up correctly.

  3. I then logged out, doing the rest of this work with the Mac’s GUI setting at the login screen.

  4. I used SSH to log in as an admin user.

  5. I created a small shell script to act as my daemon:

    $ cat /Library/PrivilegedHelperTools/altest.sh 
    #! /bin/sh
    
    
    xcrun altool --list-providers -u uuu -p @keychain:sss

    .

  6. I wrapped that in a

    launchd
    job:
    $ cat /Library/LaunchDaemons/com.example.altest.plist 
    …
    <dict>
      <key>Label</key>
      <string>com.example.altest</string>
      <key>ProgramArguments</key>
      <array>
        <string>/Library/PrivilegedHelperTools/altest.sh</string>
      </array>
      <key>UserName</key>
      <string>qnotarisation</string>
      <key>StandardOutPath</key>
      <string>/Users/qnotarisation/altest-stdout.txt</string>
      <key>StandardErrorPath</key>
      <string>/Users/qnotarisation/altest-stderr.txt</string>
    </dict>
    </plist>

    .

  7. I loaded that as a

    launchd
    daemon:
    $ sudo launchctl load /Library/LaunchDaemons/com.example.altest.plist

    .

  8. I used the

    security
    tool to created a generic password in System keychain:
    $ sudo security add-generic-password -a uuu -s sss -w ppp -A /Library/Keychains/System.keychain

    IMPORTANT This uses the

    -A
    argument to set the ACL wide open. On a real machine, you’ll probably want to use the
    -T
    argument.
  9. I started my launchd job:

    $ sudo launchctl start com.example.altest

    .

  10. After waiting a few seconds, I looked at the job’s output:

    $ cat /Users/qnotarisation/altest-stderr.txt 2019-11-27 02:58:54.910 altool[1538:23135] *** Error: Failed to retrieve providers info: (     "Error Domain=ITunesConnectionOperationErrorDomain Code=-20209 \"This Apple ID has been locked for security reasons. Visit iForgot to reset your account (https://iforgot.apple.com).\" UserInfo={NSLocalizedRecoverySuggestion=This Apple ID has been locked for security reasons. Visit iForgot to reset your account (https://iforgot.apple.com)., NSLocalizedDescription=This Apple ID has been locked for security reasons. Visit iForgot to reset your account (https://iforgot.apple.com)., NSLocalizedFailureReason=App Store operation failed.}" )

    .

As you can see, it found the password and attempting to contact the notary service (that failed, of course, because the account and password are completely bogus).

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

ps DTS is closed 25…29 Nov in observance of the US Thanksgiving holiday.

$ sudo security add-generic-password -a uuu -s sss -w ppp -A /Library/Keychains/System.keychain


This did the trick, thanks. I'm still not sure why I couldn't just copy the keychain entry altool made to the system keychain; if I try it throws an "An error has occurred. Unable to add an item to the current keychain. One or more parameters passed to a function were not valid." error message.

I'm still not sure why I couldn't just copy the keychain entry

altool
made to the system keychain

I believe that’s because

altool
created the item in the Local Items keychain [1], which is the way that Keychain Access shows you the iOS-style keychain. Interoperability between the iOS-style keychain and the old school Mac file-based keychain can be a bit bumpy. Feel free to file bugs about any such problems you encounter.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

ps DTS is closed 25…29 Nov in observance of the US Thanksgiving holiday.

[1] If you have iCloud Keychain turned on, this will be named iCloud.

Is there a particular reason why you can't do the following:


  1. Create an app-specific password for your Jenkins instance.
  2. Use Jenkins' secret text feature and save your app-specific password this way.


By the way, this is the standard way Jenkins used in many software companies. That is, have non-person accounts (i.e. system account or technical user) that are created specifically for a particular purpose and provide the credentials to Jenkins via secret texts. In turn the secret text can be exposed as environment variables to the various build scripts and thus can become command-line parameters that way.


Thanks