Network System Extension upgrade issue

We are trying to update the network system extension on macOS12.4. But sysextd is crashing and failing to update the system extension. I am sharing the sysextd logs and crash report here:

sysextd console logs:

sysextd crash report:

I am still working to figure out what is happening right now.

Can somebody help me in resolving this issue?

But sysextd is crashing and failing to update the system extension.

Does this go away if you restart your machine?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hi Eskimo, Thanks for your reply.

We still observe the issue even after restart.

Sharing some more information here:

The current output of 'systemextensionsctl list' looks like this (same as before update too)

admin@admins-MacBook-Pro-2 ~ % systemextensionsctl list                                           

1 extension(s) --- com.apple.system_extension.network_extension

enabled active teamID bundleID  (version) name [state]
'* *   ABC my.sysext.bundleid (4.3.2.123/4.3.2.123) MySystemExtension [activated enabled]

Suppose we are trying to update to 4.3.2.567 and it is failing.

It looks like sysextd is intentionally calling fatalError because it cannot derive the client information for the incoming XPC connection, in this case the container app.

Thread 1 Crashed::   Dispatch queue: sysextd.extension_manager
0   libswiftCore.dylib   0x7ff82a22e244 _assertionFailure(_:_:file:line:flags:) + 420
1   sysextd              0x10d436802    AbstractFrameworkClient.clientInfo.getter + 226

The client information is derived from the code signature. In these test cases, how is the container app signed?

Thanks Matt for the reply. If the code signing is problem, how the installation has been successful? However, for the analysis, I am sharing further information below:

System extension installation location: /Applications/SampleGUI.app/Contents/Applications/SampleSysExtHost.app/Contents/Library/SystemExtensions/my.sysext.bundleid.systemextension

     SampleSysExtHost.app   ===> system extension container app 
     my.sysext.bundleid.systemextension  ===> network system extension.

Please find the container app and system extension code signing and entitlements details in attached file:

Sharing the details again:


**ContainerApp codesign output:**
devs-iMac:Applications dev$ codesign -d --entitlements - SampleSysExtHost.app

Executable=/Applications/SampleGUI.app/Contents/Applications/SampleSysExtHost.app/Contents/MacOS/SampleSysExtHost
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>com.apple.application-identifier</key>
	<string>[TEAM_ID].[APP_ID]</string>
	<key>com.apple.developer.networking.networkextension</key>
	<array>
		<string>app-proxy-provider-systemextension</string>
	</array>
	<key>com.apple.developer.system-extension.install</key>
	<true/>
	<key>com.apple.developer.team-identifier</key>
	<string>[TEAM_ID]</string>
	<key>com.apple.security.get-task-allow</key>
	<false/>
</dict>
</plist>


**ContainerApp provisioning profile output:**

devs-iMac:Contents dev$ security cms -D -i embedded.provisionprofile 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>AppIDName</key>
	<string>[Profile AppID Name]</string>
	<key>ApplicationIdentifierPrefix</key>
	<array>
	<string>[TEAM_ID]</string>
	</array>
	<key>CreationDate</key>
	<date>[DATE]</date>
	<key>Platform</key>
	<array>
		<string>OSX</string>
	</array>
	<key>IsXcodeManaged</key>
	<false/>
	<key>DeveloperCertificates</key>
	<array>
		<data>[DATA]</data>
	</array>								
	<key>Entitlements</key>
	<dict>		
		<key>com.apple.developer.system-extension.install</key>
		<true/>
		<key>com.apple.developer.networking.networkextension</key>
		<array>
				<string>packet-tunnel-provider-systemextension</string>
				<string>app-proxy-provider-systemextension</string>
				<string>content-filter-provider-systemextension</string>
				<string>dns-proxy-systemextension</string>
				<string>dns-settings</string>
		</array>		
		<key>com.apple.application-identifier</key>
		<string>[TEAM_ID].[APP_ID]</string>		
		<key>keychain-access-groups</key>
		<array>
				<string>[TEAM_ID].*</string>
		</array>	
		<key>com.apple.developer.team-identifier</key>
		<string>[TEAM_ID]</string>
	</dict>
	<key>ExpirationDate</key>
	<date>[DATE</date>
	<key>Name</key>
	<string>[PROFILE NAME]</string>
	<key>ProvisionsAllDevices</key>
	<true/>
	<key>TeamIdentifier</key>
	<array>
		<string>[TEAM_ID]</string>
	</array>
	<key>TeamName</key>
	<string>[TEAM_NAME]</string>
	<key>TimeToLive</key>
	<integer>6570</integer>
	<key>UUID</key>
	<string>[UUID]</string>
	<key>Version</key>
	<integer>1</integer>
</dict>
</plist>


**System extension codesign output:**
devs-iMac:SystemExtensions dev$ codesign -d --entitlements - my.sysext.bundleid.systemextension

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>com.apple.application-identifier</key>
	<string>[TEAM_ID].[APP_ID]</string>
	<key>com.apple.developer.networking.networkextension</key>
	<array>
		<string>app-proxy-provider-systemextension</string>
	</array>
	<key>com.apple.developer.team-identifier</key>
	<string>[TEAM_ID]</string>
	<key>com.apple.security.application-groups</key>
	<array>
		<string>[TEAM_ID].com.test.appgroup</string>
	</array>
	<key>com.apple.security.get-task-allow</key>
	<false/>
</dict>
</plist>


**System extension provisioning profile output:**
devs-iMac:Contents dev$ security cms -D -i embedded.provisionprofile

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>AppIDName</key>
	<string>[NAME]</string>
	<key>ApplicationIdentifierPrefix</key>
	<array>
	<string>[TEAM_ID]</string>
	</array>
	<key>CreationDate</key>
	<date>[DATE]</date>
	<key>Platform</key>
	<array>
		<string>OSX</string>
	</array>
	<key>IsXcodeManaged</key>
	<false/>
	<key>DeveloperCertificates</key>
	<array>
		<data>[DATA]</data>
	</array>							
	<key>Entitlements</key>
	<dict>		
		<key>com.apple.developer.networking.networkextension</key>
		<array>
				<string>packet-tunnel-provider-systemextension</string>
				<string>app-proxy-provider-systemextension</string>
				<string>content-filter-provider-systemextension</string>
				<string>dns-proxy-systemextension</string>
				<string>dns-settings</string>
		</array>			
		<key>com.apple.application-identifier</key>
		<string>[TEAM_ID].[APP_ID]</string>		
		<key>keychain-access-groups</key>
		<array>
				<string>[TEAM_ID].*</string>
		</array>				
		<key>com.apple.developer.team-identifier</key>
		<string>[TEAM_ID]</string>
	</dict>
	<key>ExpirationDate</key>
	<date>[DATE]</date>
	<key>Name</key>
	<string>[PROFILE_NAME]</string>
	<key>ProvisionsAllDevices</key>
	<true/>
	<key>TeamIdentifier</key>
	<array>
		<string>[TEAM_ID]</string>
	</array>
	<key>TeamName</key>
	<string>[TEAM_NAME]</string>
	<key>TimeToLive</key>
	<integer>6570</integer>
	<key>UUID</key>
	<string>[UUID]</string>
	<key>Version</key>
	<integer>1</integer>
</dict>
</plist>

Because of this upgrade issue, we are unable to uninstall the system extension. Deactivation request is failing. The only remaining way to uninstall the system extension is disabling SIP which should not be done in general.

Can you please have a look at the above the signing related information and let us know what to do next.

Thank you for sharing this information on how the container app is signed.

What is going on here:

Executable=/Applications/SampleGUI.app/Contents/Applications/SampleSysExtHost.app/Contents/MacOS/SampleSysExtHost

I would expect that the container app bundle structure look like the following:

SampleGUI.app
  Contents/
    MacOS/
      SampleGUI
    _CodeSignature/
    embedded.provisionprofile
    Info.plist
    Library/
      SystemExtensions/
        my.sysext.bundleid.systemextension
    PkgInfo
    Resources/

Having helper apps in other places inside your macOS bundle outside of the designated locations will cause syspolicyd to evaluate the execution of your app / extension as code that is bundled incorrectly. See the link provided the proper location on Helper apps.

Thanks Matt for your point.

If this is the case, I am wondering how the system allowed the system extension to be installed and loaded for the first time. What I mean is, why and how this restriction applies only at the time of upgrade?

My container app looks like this:

EncryptionProxyHost.app
  Contents/
    MacOS/
      SampleGUI
    _CodeSignature/
    embedded.provisionprofile
    Info.plist
    Library/
      SystemExtensions/
        my.sysext.bundleid.systemextension
    PkgInfo
    Resources/

Right now, the above container app is bundled inside other app at /Applications/SampleGUI.app/Contents/Applications/, and system allowed installation and loading of system extension without reporting any issue about location, but failed at the time of upgrade.

So if I move the container app to location /Applications/SampleGUI.app/Contents/Helpers/, will it be allowed for upgrade?

Apart from this, does SystemExtensions framework any API to uninstall/deactivate system extension based on the combination of bundle ID and version?

There was a typo in above container app details. Please find the container app structure below:

SampleSysExtHost.app
  Contents/
    MacOS/
      SampleSysExtHost
    _CodeSignature/
    embedded.provisionprofile
    Info.plist
    Library/
      SystemExtensions/
        my.sysext.bundleid.systemextension
    PkgInfo
    Resources/

So if I move the container app to location /Applications/SampleGUI.app/Contents/Helpers/, will it be allowed for upgrade?

Am I missing something? I do not see the external SampleGUI.app or the Helpers/ directory here at all?

Hi Matt,

SampleGUI.app is not a container app for system extension. Here we bundled SampleSysExtHost.app (container app for extension) inside SampleGUI.app as below:

** /Applications/SampleGUI.app/Contents/Applications/SampleSysExtHost.app ** and container app structure is as below:

SampleSysExtHost.app
  Contents/
    MacOS/
      SampleSysExtHost
    _CodeSignature/
    embedded.provisionprofile
    Info.plist
    Library/
      SystemExtensions/
        my.sysext.bundleid.systemextension
    PkgInfo
    Resources/

Based on your input on bundling issue, I did an experiment where container app is not bundled in SampleGUI.app and distributed it along side SampleGUI.app that will be installed as /Applications/SampleSysExtHost.app. With this change, the issue is not resolved. Experiment details given below:

Scenario:

  1. Installed version 4.3.2.10 as /Applications/SampleSysExtHost.app
  2. Upgraded to 4.3.2.11 . System extension upgrrade failed.

Observations:

On BIG SUR 11.6 and 12.4:

  1. sysexxtd log shows:
default	12:11:04.335985+0530	sysextd   /Applications/SampleSysExtHost.app/Contents/Library/SystemExtensions/my.sysext.bundleid.systemextension: package type not `DEXT`

default	12:11:04.335985+0530	sysextd	/Applications/SampleSysExtHost.app/Contents/Library/SystemExtensions/my.sysext.bundleid.systemextension: entitlement `com.apple.developer.endpoint-security.client` not present or not true

default	12:11:04.336046+0530	sysextd	activateDecision found existing entry of same version: state activated_enabled, ID D888D9B7-53B5-4728-9C16-E3294A73BA3F

default	12:11:04.336103+0530	sysextd	initial activation decision: requestAppReplaceAction()

default	12:11:04.336127+0530	sysextd	notifying client of activation conflict

default	12:11:04.337141+0530	sysextd	client approved continuing upgrade for my.sysext.bundleid

default	12:11:04.337215+0530	sysextd	attempting to realize properties with identifier my.sysext.bundleid

default	12:11:04.337300+0530	sysextd	UNIX error exception: 3

default	12:11:04.340020+0530	sysextd	UNIX error exception: 3

default	12:11:04.344399+0530	sysextd	sysextd/daemon_ipc_nsxpc.swift:16: Fatal error: unable to extract client info from connection

  1. Crashed at same place:
 Thread 1 Crashed:: Dispatch queue: sysextd.extension_manager
0  libswiftCore.dylib      	0x00007fff2c899367 _assertionFailure(_:_:file:line:flags:) + 1767
1  sysextd            	0x000000010aab074c 0x10aa7c000 + 214860
2  sysextd            	0x000000010aab338d 0x10aa7c000 + 226189
3  sysextd            	0x000000010aab0fbf 0x10aa7c000 + 217023
4  sysextd            	0x000000010aab101f 0x10aa7c000 + 217119
5  com.apple.Foundation     	0x00007fff213a0c86 __NSXPCCONNECTION_IS_CALLING_OUT_TO_EXPORTED_OBJECT_S1__ + 10
6  com.apple.Foundation     	0x00007fff2134bb88 -[NSXPCConnection _decodeAndInvokeMessageWithEvent:flags:] + 2271
7  com.apple.Foundation     	0x00007fff21303039 message_handler + 206
8  libxpc.dylib         	0x00007fff20200c24 _xpc_connection_call_event_handler + 56
9  libxpc.dylib         	0x00007fff201ffa9b _xpc_connection_mach_event + 938
10 libdispatch.dylib       	0x00007fff203108a6 _dispatch_client_callout4 + 9
11 libdispatch.dylib       	0x00007fff20327aa0 _dispatch_mach_msg_invoke + 444
12 libdispatch.dylib       	0x00007fff20316493 _dispatch_lane_serial_drain + 263
13 libdispatch.dylib       	0x00007fff203285e2 _dispatch_mach_invoke + 484
14 libdispatch.dylib       	0x00007fff20316493 _dispatch_lane_serial_drain + 263
15 libdispatch.dylib       	0x00007fff203170ad _dispatch_lane_invoke + 366
16 libdispatch.dylib       	0x00007fff20320c0d _dispatch_workloop_worker_thread + 811
17 libsystem_pthread.dylib    	0x00007fff204b745d _pthread_wqthread + 314
18 libsystem_pthread.dylib    	0x00007fff204b642f start_wqthread + 15

Later, On big sur 11.6, I even observed that the new extension is activated with user approval, leading to 2 system extensions with same bundle ID in state activated enabled.

The problems with this state are:

  1. We can not deactivate older version system extension even with system extension API request. The only way is to disable SIP and deactivate using systemextensionsctl reset.

QUES: When SIP dependency will be removed to deactivate system extension using systemextensionsctl ?

QUES: Are you planning to add ability to systemextensionsctl to deactivate system extension based on bundleId and version? It will be helpful to deactivate stale older version system extensions as explained in my scenario.

So the issue still occurs even when container app distributed as separate app => this clarifies no issue with existing bundling..right?

Now I am wondering what is causing this issue? Can any of the following cause this upgrade issue?

  1. Above log says the entitlement com.apple.developer.endpoint-security.client not present or not true for system extension. Can this cause issue while upgrading?

  2. 'systemextensionsctl list' command output shows the system extension version as 4.3.2.123/4.3.2.123. This versioning can cause issue?

Has there been any progress on this issue? I am experiencing the same problem.

Network System Extension upgrade issue
 
 
Q