Thanks Matt and Eskimo for looking into this. I am attaching the full crash report. Please advice if you find anything usual in the report. Thanks. Please change the file extension from '.log' to '.ips'. ie: com.mycompanyclient.MyCompany-Client.productui.Productpkttunnel-2023-08-18-200444_2 copy.log -> com.mycompanyclient.MyCompany-Client.productui.Productpkttunnel-2023-08-18-200444_2 copy.ips com.mycompanyclient.MyCompany-Client.productui.Productpkttunnel-2023-08-18-200444_2 copy.log

Hi @meaton, I was able to to capture below system log from one of the system where this crash was observed. But the same logs were not coming on other setups. Can you please check if you can conculde anything from this? 2023-09-08 15:53:45.945148+0530 0x5c2c55 Fault 0x803806 74546 14 com.mycompany.client.mycompany-Client.productui.productpkttunnel: (Network) [com.apple.network:] nw_hash_table_release_all_objects called with invalid hash table 2023-09-08 15:53:45.945152+0530 0x5c2c55 Activity 0x803806 74546 0 com.mycompany.client.mycompany-Client.productui.productpkttunnel: (libsystem_trace.dylib) Activity for state dumps 2023-09-08 15:53:45.952315+0530 0xe99 Error 0x0 410 0 com.apple.ifdreader: [com.apple.CryptoTokenKit:ccid] Failed to find AppleUSBAlternateServiceRegistryID. 2023-09-08 15:53:45.953383+0530 0xe99 Error 0x0 410 0 com.apple.ifdreader: [com.apple.CryptoTokenKit:ccid] Failed to find AppleUSBAlternateServiceRegistryID. 2023-09-08 15:53:45.954154+0530 0x5cd11f Default 0x0 0 0 kernel: arm64e_plugin_host: running binary "bash" in keys-off mode due to identity: com.apple.bash 2023-09-08 15:53:45.960683+0530 0x5c2c55 Error 0x0 74546 0 com.mycompany.client.mycompany-Client.productui.productpkttunnel: (Network) [com.apple.network:] nw_hash_table_release_all_objects called with invalid hash table, dumping backtrace: [arm64] libnetcore-3100.140.3 0 Network 0x00000001a2a83564 __nw_create_backtrace_string + 192 1 Network 0x00000001a2c78b84 nw_hash_table_release_all_objects + 1164 2 Network 0x00000001a261f8a0 -[NWConcrete_nw_endpoint dealloc] + 500 3 Network 0x00000001a2d546e4 -[NWOSAddressEndpoint dealloc] + 76 4 Network 0x00000001a2b22688 nw_array_dispose + 440 5 Network 0x00000001a280d12c -[OS_nw_array dealloc] + 28 6 Network 0x00000001a2ceeb7c -[NWConcrete_nw_path .cxx_destruct] + 92 7 libobjc.A.dylib 0x000000019bcc840c _ZL27object_cxxDestructFromClassP11objc_objectP10objc_class + 116 8 libobjc.A.dylib 0x000000019bcbfe88 objc_destructInstance + 80 9 libobjc.A.dylib <…> full log Thanks Vishal

Yes if I do stopVPNTunnel from provider app then along with utun interface all routes for utun interface also deletes. But as you said I dont want to do that. Please suggest if there is any way we can handle it inside provider extension itself.

I have files the feedback 13167536 Thanks for the tip on '-cancelTunnelWithError:', with this I am able to cancel the tunnel and catch the notification in provider app to reset the tunnel manager thus the virtual interface and related route table. This clears the route entries for utun3 (in my case). In provider app, I was able to collect the custom error domain and code using -fetchLastDisconnectErrorWithCompletionHandler:. But this API is only available in macOS 13.0 and later. How do I collect Disconnection Error in older macos version (Bigsur and Monterey)? I can see the '_lastDisconnectError' is available in debuger, but is there a way I can collect it in my program? (lldb) p ((NETunnelProviderSession*)(statusChangeNotification.object))->_lastDisconnectError (NSError *) $4 = 0x000060000037d680 domain: "MyTunnelErrorDomain" - code: 1

Thanks Matt. We will keep observing the setup and raise the bug as suggested by you.

Thanks for your response Quinn. You seem to have hard coded user 501. Is that intentional? I had written earlier code for an example. Below is how exactly the code looks like: logged_in_user=`users` for user in ${logged_in_user}; do user_uid=`id -u $user` launchctl enable gui/$user_uid/com.mycompany.client.myproduct echo "myproduct ui launchctl enable returns $?" echo "myproduct ui Launching UI for $user with uid: $user_uid " launchctl bootstrap gui/$user_uid /Library/LaunchAgents/com.mycompany.myproduct.plist echo "myproduct ui launchctl bootstrap returns $?" done Yes it works for 2nd local user created. Even if I create a new user after installing my software with user1 (userid: 501), launchd successfully starts the agent for the user2 (userid: 502) which is created later. But it still doesnt work for the AD user (uid=1677024404)

I list the steps and scenarios I am doing: I have an Apple installer package (.pkg). It lays down a launchd agent property list in /Library/LaunchAgents. Its post-install script loads the agent in the current user’s GUI context with the launchctl bootstrap gui/UID PATH command. I log in to the Mac’s GUI using a local user account. And double click your package to open it in the Installer. I run through the installation process. At the end, the agent is running in the logged in user’s context. I logout the current user and login to a different local user, I found that the agent is still running and listing in 'launchctl list' command. I logout the 2nd local user and login to an AD user, I found that the agent is not running and not listing in 'launchctl list' command. Observation: Surprisingly if I install the .pkg using an AD user and switch to a local user or a different AD user I found the agent running and listing in 'launchctl list' command. If you add a log point to your script inside your for loop, do you see the right value for user_uid? The log is printing the local users ids . In my case there are two local users with user ids 501 and 502 and these two are printed in log when I login using a local user and install the pkg.

Still running? Or running again? Given that your agent is loaded into the Aqua session, I’d expect the first user’s agent to terminate when you log out and the system to start a new instance of the agent for the new user when you log in. @eskimo you are right, i meant a new agent with a new pid is started when a new local user is logged in. Well, that’s weird. I’ve got no good explanation for why that’d be the case. @eskimo do you see any issue with the agent plist file I have loaded? Also I wanted to know how is the launchd system is designed for launch agents? Since an AD user is not locally known, its userid will also be known in advance to the system. Then what is the need of enabling and bootstrapping an agent in a loop like below: for user in ${logged_in_user}; do user_uid=`id -u $user` launchctl enable gui/$user_uid/com.mycompany.client.myproduct launchctl bootstrap gui/$user_uid /Library/LaunchAgents/com.mycompany.myproduct.plist done Another observation: If I add a new local user after installing the pkg (using 1st local user), the service is started when i switch to a newly created local user.

I do this is in this order: Remove property list. Remove helper tool. Remove the job. Hi @quinn, are you sure we need to 'Remove the job' here. I followed the steps you captured here to remove my 'helper tool' using 'sudo launchctl remove /Library/LaunchDaemons/com.my***.Installutil.plist' and I had to and running SMJobBless() didnt work again. I even rebooted my macbook which didnt help. SMJobBless was not giving any error and no errors were captured in console logs. Console log was showing 'backgroundtaskmanagementd' did the job normally: 2024-01-18 21:51:33.286201+0530 0x607f8 Default 0xa7a28 474 0 backgroundtaskmanagementd: [com.apple.backgroundtaskmanagement:main] effectiveItemDisposition: appURL=(null), type=legacy daemon, url=file:///Library/LaunchDaemons/com.my***.Installutil.plist, config={ BTMConfigArguments = ( "/Library/PrivilegedHelperTools/com.my***.Installutil" ); BTMConfigBundleIdentifiers = ( ); BTMConfigExecutablePath = "/Library/PrivilegedHelperTools/com.my***.Installutil"; BTMConfigLabel = "com.my***.Installutil"; } After spending 2 days I thought to doubt the step captured here. To recover the setup I had to call SMJobBless() API which didnt start the helper tool, but copied the plist and helper too, to its location, then I had to run 'sudo launchctl load -w -F /Library/LaunchDaemons/com.my***.Installutil.plist' separately and then it started to work. I am capturing the steps below for others so that they dont run into same problem: How to uninstall a helper tool: Remove property list. Remove helper tool. Note dont run 'launchctl remove' if you wise to reinstall the helpertool again. I still have to try 'SMJobRemove' API and check if that removes the job from the launchctl list. else @quinn can guide how to remove the job from launch list, otherwise as pointed by quinn earlier the next reboot will remove the entery from the list.

This issue was also present on MacOS Ventura, but on Ventura I had an option to revert back my Xcode to 14.x and I was still able to work on Instruments tool for debugging. But yesterday I upgraded my MacOS to Sonoma and I was unaware that Xcode 14 will not work on Sonoma. I was forced to upgrade my Xcode also to 15.2. Now I am stuck and I am not able to use this tool on Sonoma. I have filled a bug in Apple's Feedback Assistant: FB13594413. Please help me prioritising this bug with the relevant team.

If you are able to attach a project with the system extension you are trying to debug This issue is also present with Apple 'SimpleFirewall' project. Below are the steps to reproduce: Download and compile Apple's SimpleFirewall project Before running the Apple is Xcode 15.2, attach 'com.example.apple-samplecode.SimpleFirewall********.SimpleFirewallExtension' in Xcode's 'Debug->Attach to process by Name'. Now run the SimpleFirewall App and allow to install Filter Extension. Above will install, start and attach the Filter Extension in 'Debug Navigator' in Xcode. Now for Filter Extension's memory, try to 'Profile in Instruments'. This action will fail with the error: 'Process No Longer Exists'. Note: I have found a work around for this issue, if I attach the pid of system extension after starting the App at runtime, it allows me to 'Profile memory in Instruments '.

Thanks for the response @eskimo. Can you explain more about your overall goal here? We have two apps which follow different release cycle. 1st app has created an keypair entry in the keychain and already present in field on deployment setups. Now we want to release our 2nd app which wants to access the keychain entry. Since the 2nd app is not whitelisted in the ACL, I want the 2nd app installer to make that entry in ACL. Can it be done somehow programmatically without deleting the existing entry?

Thanks for your response @meaton Matt. Sorry if my original question was not clear. In my case, the UI service which installs and configure the packet tunnel, that itself is not starting. I need this service (Launch Agent) up and running as it does many things including configuring and starting the packet tunnel. So need your help in debugging what could be wrong where that launchd couldnt start the ui service in above mentioned test scenario.

If your app takes the form of a container app with the full presence of a UI then do you have an issue here? yes my app is an UI container app which installs and configures the packet tunnel. if you print out the status from launchctl 'launchctl list' doesn't list my service, there is no entry for my service. Even a system restart didnt start or list the service. I have mentioned the test scenario in my original 1st post. Please refer the test steps to understand the test scenario where it is not working. Also I have attached few logs in the 1st post, please let me know if you see any thing obvious there. Otherwise let me know which service logs I can share from the console which can help in identifying the issue.

If you try to run your container app and Packet Tunnel Provider without MDM or launchd, and just as a normal container app and provider, does everything install correctly? Yes everything works fine in your mentioned scenario. To share more data, my app works as expected (launchd starts the agent) when I install my package on mac machine without MDM and a user logged in. In case of MDM usecase also, things works fine if an user is logged in to the test mac machine. In above case, since an user is logged in 'laucnhctl bootstrap gui/ /Library/LaunchAgents/com.****.****ui.plist' is triggered by the postinstall script.