Documentation bug reported, as FB14929037.
Yes - the offset from the string is quite large, but I just want useful output such as "ThisFunctionWasInvolved" from "ThisFunctionCausedAPanic.cpp, line 455".
Symbolicate the other frames you've got. This is a quick way to differentiate between "something is wrong with THIS symbol" vs "something systemic is wrong".
Yes, this is a “something systemic is wrong” situation. I was trying to reduce the amount of information for readers to wade through, but I am not getting useful results for any efforts at symbolicating the addresses on the back trace or the PC (program counter).
” dwarfdump" will print the contents of a dsym file, which will either show that the symbols
dwarfdump works on dSYM files, but apparently not on debug-builds-with-symbols-in-the-binary. As such, I rebuilt with separating the symbols into a dSYM file. dwarfdump provides what appears to be correct information for the symbols, but for a sanity check here is a blurb from the start of the dump:
KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME(arm64e): file format Mach-O arm64
.debug_info contents:
0x00000000: Compile Unit: length = 0x000001f9, format = DWARF32, version = 0x0004, abbr_offset = 0x0000, addr_size = 0x08 (next unit at 0x000001fd)
0x0000000b: DW_TAG_compile_unit
DW_AT_producer ("Apple clang version 13.1.6 (clang-1316.0.21.2.5)")
DW_AT_language (DW_LANG_C99)
DW_AT_name ("/Users/username/Library/Developer/Xcode/DerivedData/KEXT_NAME-stuff/ArchiveIntermediates/KEXT_NAME/IntermediateBuildFilesPath/KEXT_NAME.build/Release/KEXT_NAME.build/DerivedSources/KEXT_NAME_info.c")
DW_AT_LLVM_sysroot ("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk")
DW_AT_APPLE_sdk ("MacOSX12.3.sdk")
DW_AT_stmt_list (0x00000000)
DW_AT_comp_dir ("/Users/username/sandbox/codedirectory")
DW_AT_APPLE_optimized (true)
Since the binary is different, I triggered a new panic to work with:
Apple Silicon Panic log (sanitized)
The command line utility atos should be helpful as well, but here’s an example of what I get as output from that, when using full dSYM file information:
zsh_prompt% atos -arch arm64e -o KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME -l 0xfffffe0015423fd0 0xfffffe0015426e2c
0x00002e5c (in KEXT_NAME)
If I manually look up the 0x2e5c offset in the output from dwarfdump, I can see the following:
0x00002e5c: DW_TAG_formal_parameter
DW_AT_type (0x0000000000009408 "const IOService *")
DW_AT_artificial (true)
But that's definitely not conveying the one-step-to-symbolication that atos is supposed to be able to perform. It does not appear that I am using it incorrectly, but I am definitely not getting helpful results from it.
Following the idea that perhaps the kernel slide is not being properly dealt with, I followed this link ( https://lists.apple.com/archives/darwin-kernel/2014/Jan/msg00011.html ) for advice on how to deal with kernel slide (below is all commands listed, and lack of useful information provided):
lldb commands and log for dealing with kext slide
So to answer all of the questions:
Systemic issue, I am having trouble interpreting a kernel panic log and getting valid symbols as a result of address lookup for Apple Silicon
dSYM file appears to be built properly
atos is not providing helpful information, despite having proper dSYM file
lldb commands to load images and deal with kernel slides does not yield helpful information
Post
Replies
Boosts
Views
Activity
Kevin Elliot, DTS Engineer, solved this in a private code-level support ticket.
There is yet another offset that was not being accounted for, but can be found using the command line tool otool:
zsh_prompt% otool -arch arm64e -l /Library/Extensions/KEXT_NAME.kext/Contents/MacOS/KEXT_NAME
will give a lot of output information, but a section with this information:
cmd LC_SEGMENT_64
cmdsize 232
segname __TEXT_EXEC
vmaddr 0x0000000000004000
vmsize 0x0000000000010000
Shows that an address of 0x4000 needs to be accounted for when performing the atos lookups.
By adding that value to the load address of the kext when calling atos, we get the correct / expected answer.
zsh_prompt% atos -arch arm64e -o KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME -l <load address + (vmaddr of __TEXT_EXEC section)> 0xfffffe0015426e2c
com_company_kext_name::expectedFunction(OSObject*, void*, void*, void*, void*) (in KEXT_NAME) (SourceFile.cpp:1999)
Such a relief! Kudos to Kevin Elliot for finding a solution to this.
DMA issues were resolved (ordering of hi- and lo-address 32-bit registers was mixed up, resulting in bad addresses).
The steps provided for PCIDriverKit Bus Leader and Memory Space Enable are working properly - the code written above does work.
One undocumented requirement is that the name of the dext must match the dext's bundle identifier. This is most likely a bug, and I have submitted a Feedback Report for it (FB15590713), but in the meantime please make sure that your dext's "PRODUCT_NAME" matches "$(PRODUCT_BUNDLE_IDENTIFIER)" in the dext target's "Build Settings" tab.
This was noticed by the following developer forum thread as well:
https://developer.apple.com/forums/thread/133499?answerId=422125022#422125022
It is documented that the bundle identifier of the dext must be just an extension beyond the bundle identifier of the dext-hosting application.
For example, if the Dext-hosting application's bundle ID is "com.company.DextHostingApp" then the dext's bundle ID must be "com.company.DextHostingApp." where can be whatever you want, but the leading section must match the owning application's bundle ID.
Apple has changed the certificates over the years, the current ones are "Developer ID Application" and "Developer ID Installer" - you shouldn't need "Installer" unless you must run an installer specifically. (Sorry I don't recognize the "ipa" acronym, if that has relevance, or if it's just a typo from "app").
To answer your second question: yes, more than one person can have the Developer ID certificate on their machine to have multiple "build machines."
The initial generation/download of the certificate must come from (I believe) the Team Agent (not an Admin), and there are special "private keys" that you need to make sure you receive as well.
It's been a while since we've done that, but the process is something like Team Agent at developer.apple.com creates the certificate and downloads/installs it, then "exports" it from Keychain Access, and the next developer imports it to their Keychain Access app.
Thank you both for the responses.
We have accomplished the Team Agent request for entitlements for PCI DriverKit - it looks like I am able to perform all other tasks necessary at my "admin" level on the developer portal.
Following ssmith_c's advice, I was able to build, sign, notarize, and deploy a dext within a hosting application, the added export and manual selection seemed to be the difference I needed.
However, I'm still encountering some issues with Xcode 15.2 when trying to make a distributable application that works with the dext (#3 of the software types listed above). One worked, and one seemed to not work. The "not working" one is crashing based on:
Exception Type: EXC_CRASH (SIGKILL (Code Signature Invalid))
Exception Codes: 0x0000000000000000, 0x0000000000000000
Termination Reason: CODESIGNING 1 Taskgated Invalid Signature
Using codesign to inspect the signature, it seems ok, but I likely am just not seeing the issue:
daniek3@MacBook-Pro Project % codesign -dv --verbose=3 AppName.app
Executable=/Users/daniek3/repository/Project/AppName.app/Contents/MacOS/AppName
Identifier=com.company.AppName
Format=app bundle with Mach-O universal (x86_64 arm64)
CodeDirectory v=20500 size=319557 flags=0x10000(runtime) hashes=9975+7 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha256=94ec693fa2c5c43167f1e8005a93d6405361072f
CandidateCDHashFull sha256=94ec693fa2c5c43167f1e8005a93d6405361072fa22fd9750416d50890c41f4e
Hash choices=sha256
CMSDigest=94ec693fa2c5c43167f1e8005a93d6405361072fa22fd9750416d50890c41f4e
CMSDigestType=2
CDHash=94ec693fa2c5c43167f1e8005a93d6405361072f
Signature size=8980
Authority=Developer ID Application: Company (<Team ID>)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Oct 29, 2024 at 2:23:36 PM
Info.plist entries=22
TeamIdentifier=<Team ID>
Runtime Version=14.2.0
Sealed Resources version=2 rules=13 files=3
Internal requirements count=1 size=220
As noted, however, this was in Xcode 15.2, so I'll move to Xcode 16 and give it another try.
Trying the same app that is having difficulty in the last post (above) in Xcode 16.1.
From the Organizer, choosing to distribute app as "Custom->Direct Distribution->Export->Development Team (selected)->Automatically Manage Signing" gives the following two errors:
There is a problem with the request entity - You already have a current Developer ID Application Managed (with Kext) certificate or a pending certificate request.
No profiles for 'com.company.AppName' were found - Xcode couldn't find any Developer ID provisioning profiles matching 'com.company.AppName'.
Having previously developed kexts for our hardware, yes, our developer certificate has the "kext" qualifier. This should be normal for most developers moving from kext to dext development.
There absolutely is a provisioning profile for com.company.AppName - as will be shown in the next step.
If instead I use the choices:
From the Organizer, choosing to distribute app as "Custom->Direct Distribution->Export->Development Team (selected)->Manually Manage Signing" and select "Distribution Certificate: Default (..)" and "AppName.app: " I get a new error that reads:
Profile doesn't match the entitlements file's value for the com.apple.developer.driverkit.userclient-access entitlement.
This error seems poorly articulated as well, since the entitlements file requires I enter the Bundle ID of the dext that the application will communicate with, but there is no place to declare that in the developer portal when making the provisioning profile.
Should I open a technical support incident to get this more closely looked at? This app works when built in debug, but I am not getting the settings correct to be able to distribute it.
Like you, I previously encountered a similar error in Xcode 16.0 beta - entered as FB15239007 (this pertains to the kext-related developer certificate error).
I've additionally submitted FB15634854 to describe the "Profile doesn't match the entitlements file's value for the com.apple.developer.driverkit.userclient-access entitlement." error to Apple.
FWIW, I completely rebuilt the project, thinking perhaps that I had gotten into a bad customization spot with the previous one, but I am still unable to get a combination that works in Xcode 16.1 for this.
Thank you Kevin for the explanations and the tools to distinguish which element contributed to which piece. It's taken a few days, but I've figured out a few things and can hopefully get this all settled.
First, I had changed the bundle ID of my dext to what it "should be", after learning that the bundle ID ought to be an extension of the "owning" application's bundle ID.
It turns out you can't do that from an Admin role. I kept looking at the output of the security command and seeing the older bundle ID showing up for com.apple.developer.driverkit.userclient-access., which was not the updated bundle ID I was developing now.
Backing up to this point of the thread:
That leads to the other two errors, which are actually the same underlying error. First the manual error:
Profile doesn't match the entitlements file's value for the com.apple.developer.driverkit.userclient-access entitlement.
In other words, you pointed Xcode at a provisioning profile ("2") which didn't match apps signing ("1") configuration. You were actually on the right track here;
there is no place to declare that in the developer portal [what the dext bundle ID is] when making the provisioning profile.
Your Team Agent must request DriverKit entitlements and know in advance the bundle ID of the dext you intend to create. That's how the provisioning profile (2) from the developer portal knows the bundle ID of the dext that your application wants to create a user client for. Having this more clearly documented would be helpful.
Next post I'll explain the settings that I am currently using - there are still a few lingering issues which most likely need new forum threads, so I won't initiate those conversations on this thread, but I do want to wrap this up with a clear description so that others don't hit the same walls that I did.
For this explanation, I am using Xcode 15.x. The hosting application will use com.company.HostingApp as its bundle ID and com.company.HostingApp.Driver as the dext's bundle ID, and then a second app with bundle ID com.company.ClientOnlyApp which communicates with the dext but is not the hosting application.
As noted above, the driver bundle ID of com.company.HostingApp.Driver needed to be requested from the Team Agent at the developer portal. Once that's completed, you still need to create the Identifier to be used, by going to Identifiers and creating one that matches.
Dext Identifier settings:
DriverKit Allow Third-Party User Clients ("Capabilities" tab - I need this, perhaps you do not)
DriverKit ("Additional Capabilities" tab)
DriverKit Allow Any UserClient Access ("Additional Capabilities" tab)
DriverKit PCI (Primary Match) ("Additional Capabilities" tab) (I'm making a PCI driver, so I need this)
Hosting App Identifier settings:
DriverKit Communicates with Drivers ("Capabilities" tab)
System Extension ("Capabilities" tab)
DriverKit ("Additional Capabilities" tab)
DriverKit UserClient Access ("Additional Capabilities" tab)
Client Only Identifier settings:
DriverKit Communicates with Drivers ("Capabilities" tab)
DriverKit ("Additional Capabilities" tab)
DriverKit UserClient Access ("Additional Capabilities" tab)
Once these are built, you can move to Profiles and create the provisioning profiles necessary. Since the settings are covered in the Identifiers, creating a Provisioning file only required selecting Distribution - Developer ID, then selecting the App ID (aka Identifier), then under Certificate choose your Developer Team's Certificate, and finally under Additional Entitlements there is a drop-box to select DriverKit and System Extension Template for Mac Dev ID profile / Mac Direct Dist Profile (yours may not be named this, but mine is).
And in the Xcode Project -> Target -> Signing and Capabilities, I have the following selections:
Dext Target Signing and Capabilities
Automatically manage signing checked
Team
Bundle Identifier: com.company.HostingApp.Driver
DriverKit (x2)*
DriverKit (Allow Any UserClient Access)
DriverKit (PCI Primary Match) (x2)*
(*=these are due to Xcode 15's duplication of capabilities)
Hosting App Target Signing and Capabilities
Automatically manage signing checked
Team
Bundle Identifier: com.company.HostingApp
App Sandbox
Hardened Runtime
System Extension
Client Only App Target Signing and Capabilities
Automatically manage signing checked
Team
Bundle Identifier: com.company.ClientOnlyApp
App Sandbox
Hardened Runtime
System Extension
Once all of those settings are done, I can produce distributable dext software and applications that work with the dext.
Order of steps to produce distributable software:
Hosting application's Xcode's Product -> Archive menu option
Xcode's Window -> Organizer, select the hosting application archive that was just built, and press the Distribute App button
Choose Custom
Choose Developer ID - distribute directly to customers
Choose Export - sign and export without notarizing
Choose Manually manage signing
Select Distribution certificate for your Developer Team, Hosting App provisioning file for the hosting app, and Dext provisioning file for the dext.
Click Export
Use notarytool and staple to complete the notarization process.
Repeat these steps for the Client Only application as well.
Notes & Questions:
I know that I do not have DriverKit UserClient Access selected in the Xcode Project -> Target -> Signing and Capabilities selection for either the hosting application or the client only application. This seems to be in keeping with Kevin Elliot's example, and the current situation is working for me, perhaps because the dext is set to "allow any UserClient Access"?
System Extension Redistributable is an entitlement that means your dext (or other system extension) is allowed to be distributed by another Developer Team. This is not a feature that I needed, but perhaps you might.
It is unclear what a Developer Team needs to do when the team must deploy multiple dexts - it seems like the system is built for a single dext authorization (see above post where the Team Agent must register the dext bundle ID) - how would a team select between different dexts?
Hosting applications must be placed in /Applications/, and directly there, not in a sub-directory. Given that requirement, it seems that an installer package is still the best option for distributing the host application, so that it is installed properly on the user's hard drive?
Hi Kevin,
First, thank you for your patience on all of this.
It turns out you can't do that from an Admin role. I kept looking at the output of the security command and seeing the older bundle ID showing up for com.apple.developer.driverkit.userclient-access., which was not the updated bundle ID I was developing now.
FYI, this is one of the pitfalls of manual codesigning, as automatic codesigning will not allow that. That's actually the biggest issue with manual codesigning- it allows you to force configuration that won't actually work, so unless you understand EXACTLY why automatic is failing, you can easily end up replacing an error at signing with a different error somewhere "else".
That's the issue, though. I don't know EXACTLY why automatic codesigning is failing - at least with manual code signing I can look at what settings I am using and try something else. If I just press "automatic" and it doesn't work (it hasn't yet), there does not seem to be anything further I can do to affect the problem - I am just stuck.
Next:
First, I had changed the bundle ID of my dext to what it "should be", after learning that the bundle ID ought to be an extension of the "owning" application's bundle ID.
This is common practice and what Xcode does by default, but I don't believe the system actually requires it, as it needlessly restricts/complicates what's possible without any real benefit. It's been awhile since I've specifically tested this, but it's definitely possible to create an app that doesn't follow this requirement and I believe it will work fine.
Ok, that's helpful information.
Next:
What IS true:
That's how the provisioning profile (2) from the developer portal knows the bundle ID of the dext that your application wants to create a user client for. Having this more clearly documented would be helpful.
...is that the user client entitlement IS tied to a specific bundle ID.
Got it. That failed in the automatic process, though - it kept bringing in the old dext bundle ID, and it was only AFTER involving the Team Agent and registering a new dext bundle ID that I got the correct results (via manual code signing and ignoring automatic).
This next piece might be the a-ha moment we've been looking for:
...finally under Additional Entitlements there is a drop-box to select DriverKit and System Extension Template for Mac Dev ID profile / Mac Direct Dist Profile (yours may not be named this, but mine is).
This step is not necessary and will not exist for many developers. The UI above is the original process used to "attach" managed entitlements to specific bundle IDs and was actually replaced by the "Additional Capabilities" tab you used above. You do not need to do so and it's effect will either:
Neutral, as the system should be smart enough to compare the existing entitlement set (configured through "Identifiers-> Additional Capabilities") and ignore any duplicates.
Harmful, as it will allow you to add entitlements to the profile that aren't configured on the identifier. That will then cause problems later, particularly with automatic codesigning.
OH. Ok, since it was presented, and I only had that option or "default" (which implied to not provide ANY entitlement), I chose the option which looked like it would do something. Thank you for clarifying, and glad I was detailing every step for us to find this.
As for whether or not your own app needs an entitlement to communicate to your own dext:
This part is complicated as there are multiple, different, mechanisms at work. Starting with the DEXT side:
By default, the initial connection system only allows connection requests from processes that are signed by the same Team ID as the DEXT. The "allow any UserClient Access" entitlement overrides that behavior allowing the "request" from any process. If "your DEXT" only talks to "your app/processes", then you don't need any entitlement to allow.
I know I have particular use cases that others will not, and for the moment I'm just trying to establish the correct rules to develop and deploy, outside of the App Store, a dext, it's owning application, and supporting applications.
For this particular dext, I need to allow "any userclient" to connect. It looks like the reason my apps can connect without defining the "userclient access" entitlement is because they share the same TeamID as the dext.
I used the term "request" because the DEXT itself has complete, dynamic control (at runtime) of what it actually accepts connections from. Note that I really do mean "complete and dynamic". If you want to create a DEXT UserClient that only accepts every other request on Tuesdays, that is entirely possible. This is a complicated topic on it's own so if you want to discuss that, lets start a new thread on that subject.
No, I don't need a follow-up thread. This I understand, I have two kinds of user clients and can dynamically control whether a connection is established or not, I have that working already. This thread is just for what codesign/entitlements/notarization is needed to actually deploy a dext outside of the App Store.
I suspect your app isn't sandboxed.
It is sandboxed - I'm in fact having trouble with accessing a user preference file thanks to the sandboxing (and spaces in the directory names), but the connection to the dext has no issues. I think it's due to the above bullet point about the TeamIDs matching, then.
Re: multiple dexts - understood, glad to hear that "com.apple.developer.driverkit.userclient-access" entitlement is an array. Should I be creating that entry in App.entitlements as an array with a single string entry? I was creating it as a string with the bundleID of the dext when I was trying it.
Re: creating an installer package:
Attempt the installation.
If it fails, check its location and tell the user if they need to move the app.
Thank you - this is a cleaner solution that is already functioning, and avoids concerns about installer codesigning/notarization breaking the application's/dext's settings.
I understand the repeated statements that Xcode 16 automatic codesigning, archiving, notarization, and deployment should work - but they have not done so for me to date. I will give it yet another try, now that I know what steps are necessary to get a manual configuration working properly.
I've been able to re-test all of these automatic codesigning/provisioning/notarization suggestions on Xcode 16.1, by modifying all of the bundle IDs, changing to "Automatically Manage Signing" and attempting again with my "Admin" developer portal credentials.
It does not work. I still get all 3 errors of:
There is a problem with the request entity - you already have a current Developer ID Application Managed (With Kext) certificate or pending certificate request.
No profiles for "com.company.HostingApp.Driver" were found - Xcode couldn't find any Developer ID provisioning profiles matching 'com.company.HostingApp.Driver'.
No profiles for com.company.HostingApp" were found - Xcode couldn't find any Developer ID provisioning profiles matching 'com.company.HostingApp'.
It seems you are correct, the Admin level does not have authority to manipulate the developer portal side of the automatic process.
"Tightly controlled account access" does not really work in the modern world of two-factor authentication and remote development teams. The "Team Agent"/"Account Holder" is the CEO, typically - the only one empowered by the company to make legally binding agreements with Apple. The CEOs in most large organizations are not going to distribute their 2FA information to multiple developers just to get tasks like constructing App Identifiers done.
Yes, most companies use something of a "build machine" account that produces the corporate-signed software, however, that account is rarely, if ever, operated by the CEO either.
I understand that my complaint needs to be entered as a feature request, and I will enter a Feedback Assistant report to this effect.
In the meantime, however, the above set of instructions I provided has made it possible for me to complete the task of producing distributable dext software, so I will be marking that as the accepted solution.
Thank you!