Service exited due to SIGTERM | sent by launchd

Getting "Service exited due to SIGTERM | sent by launchd" message for binary running as service under LaunchDaemons.

Scenarios:

  1. I have a binary running as service(service plist contents pasted in at the end), everything works fine.
  2. Now in order to use the newly compiled binary, I unload the service, replace the existing binary with the new one and then load the service.
  3. I expect to see the service running but what I see is that the service was killed by launchd with SIGTERM. Not sure why???
    (snippet of system.log after doing launchctl load with the new binary)
sh-3.2# launchctl load /Library/LaunchDaemons/com.MyPkg.myprogrambinary.plist
sh-3.2#
sh-3.2# tail /var/log/system.log
Nov 22 19:04:51 Nilesh-Mac-mini com.apple.xpc.launchd[1] (com.MyPkg.myprogrambinary): Unknown key for string: ServiceDescription
Nov 22 19:04:51 Nilesh-Mac-mini com.apple.xpc.launchd[1] (com.MyPkg.myprogrambinary): Please switch away from OnDemand to KeepAlive.
Nov 22 19:04:51 Nilesh-Mac-mini com.apple.xpc.launchd[1] (com.MyPkg.myprogrambinary[73878]): removing service since it exited with consistent failure - OS_REASON_EXEC
Nov 22 19:04:51 Nilesh-Mac-mini com.apple.xpc.launchd[1] (com.MyPkg.myprogrambinary): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.
Nov 22 19:05:01 Nilesh-Mac-mini com.apple.xpc.launchd[1] (com.MyPkg.myprogrambinary[73886]): Service exited due to SIGTERM | sent by launchd[1]
Nov 22 19:05:01 Nilesh-Mac-mini com.apple.xpc.launchd[1] (com.MyPkg.myprogrambinary): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.
Nov 22 19:05:11 Nilesh-Mac-mini com.apple.xpc.launchd[1] (com.MyPkg.myprogrambinary[73887]): removing service since it exited with consistent failure - OS_REASON_EXEC
Nov 22 19:05:11 Nilesh-Mac-mini com.apple.xpc.launchd[1] (com.MyPkg.myprogrambinary): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.


BTW, everything works fine with the newly replaced binary if I restart the system. Also, both the binaries and the .pkg are codesigned and notarized.


Service plist:

sh-3.2# ls -l /Library/LaunchDaemons/com.MyPkg.myprogrambinary.plist
-rwxr-xr-x  1 root  wheel  1059 Nov 22 15:18 /Library/LaunchDaemons/com.MyPkg.myprogrambinary.plist
sh-3.2#
sh-3.2# cat /Library/LaunchDaemons/com.MyPkg.myprogrambinary.plist
        
        
        
        
                Label
                com.MyPkg.myprogrambinary
                OnDemand
                
                Program
                /opt/myprogrambinary
                ProgramArguments
                        
                                /opt/config.json
                        
                RunAtLoad
                
                ServiceDescription
                MyPkg
                UserName
                root
                WorkingDirectory
                /opt/
        
        
sh-3.2#


OS details:

sh-3.2# system_profiler SPSoftwareDataType
Software:


    System Software Overview:


      System Version: macOS 10.15 (19A602)
      Kernel Version: Darwin 19.0.0
      Boot Volume: Macintosh HD
      Boot Mode: Booted from installation CD/DVD
      Computer Name: Nilesh-Mac-mini
      User Name: System Administrator (root)
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled
      Time since boot: 4 days 4:01


What is it that I am missing or doing wrong?


PS: Observed that this is only reproducible on macOS 10.15 and not on any other macOS versions.


Thanks in advance!!!


Regards,

Nilesh chate

Accepted Reply

to use the newly compiled binary, I unload the service, replace the existing binary with the new one and then load the service.

How do you replace the binary?

The most common gotcha here is using a program that copies one executable over another. This doesn’t end well because the kernel caches code signing information that’s not flushed when you change the binary out from under that cache. When replacing binaries like this, it’s important to use a mechanism that doesn’t overwrite the binary. The standard approach is to save a temporary copy somewhere and then move it into place.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Replies

to use the newly compiled binary, I unload the service, replace the existing binary with the new one and then load the service.

How do you replace the binary?

The most common gotcha here is using a program that copies one executable over another. This doesn’t end well because the kernel caches code signing information that’s not flushed when you change the binary out from under that cache. When replacing binaries like this, it’s important to use a mechanism that doesn’t overwrite the binary. The standard approach is to save a temporary copy somewhere and then move it into place.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

I am using a program written in golang to replace the existing file. I tried with mv and cp command and it worked fine as mentioned by you.

Perhaps I was wondering if there is a way I could clear codesign cache of the binary before replacing it rather than renaming it and moving the new file or is there any better/correct way to do this.


Thanks in advance!!!

Perhaps I was wondering if there is a way I could clear codesign cache of the binary before replacing it rather than renaming it and moving the new file or is there any better/correct way to do this.

You are overthinking this. The correct way to do this is as I described in my previous response. Don’t try anything more complex.

Indeed, this was discussed on stage at WWDC this year. Check out WWDC 2019 Session 703 All About Notarization, and specifically the section starting on slide 63. This ends with a big bullet saying “Whenever you update a signed file, create a new file”.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"