No doubt that working with a child process would be the best option. However, it would also be a little bit dangerous for me to use such a technique as I'm not at all familiar with it. Moreover I'm afraid that it could lead to some complex issues with the sandbox which could require a lot of time to solve. But I keep this interesting suggestion in mind for a future update (I've seen a nice example you wrote in Swift and Objective-C, it could be a good starting point). For now, I will go with snapshots and lock free structures which are more familiar to me, and I will install my signal handlers with sigaction instead of dispatch_sources.
Thanks for the great help!
Post
Replies
Boosts
Views
Activity
Thanks for the reply!
You're right, I probably underestimate the difficulty. However I've already read your interesting post about implementing a custom crash report, and many other related articles on the web. From a previous discussion with you, I finally decided to implement a minimal handling of signals and C++ exceptions and to postpone the dialog I was displaying - to inform the user that we're aware of the problem and try to fix it - to the next startup (if any...).
My attempt here is not to test the real instable context of miscellaneous crashes, but to test the mechanism of saving some files on disk while crashing and react to the existence of these files during the next launch. These kind of things is difficult to test, but I recently found that it is possible with UI test thanks to its proxy mechanism.
I'm aware of the strong limitations: to write on disk, I only use open, write and other allowed C functions. The first file saved is just an empty file to mark the session as crashed. Then data written on disk is not a crash report but a short description of the application context at crash time to help understanding what happened. This context is not captured within the handler but before, during the normal usage of the app. This way, the writing is only done with dates, numbers and strings already stored so that no memory is allocated.
Naturally, this is not perfect, but be sure that I really try to do my best and listen your recommandations very seriously.
Finally, about dispatch_source: should I understand that to support SIGABRT and the other signals you mentioned, I should implement a second handler using signal or sigaction? I mean, not only for tests but even for production codes?
Many thanks to the very helpful link! Even if I don't try to implement my own crash reporter, this post contains a lot of useful info. After reading, it seems to me that there is no real benefit in this case to catch signals, or even C++ exceptions. Either to save the application state or to display an alert, I need Objective-C and there is nothing I can do with just C or C++ codes, except printing something in the console. But it wouldn't be helpful here since I'm not in touch with the users whose I receive crash reports via the Xcode Organizer.
What you tell about NSException is surprising to me: from my experience, saving a 'recoverable' session and displaying an alert during an exception handling works correctly. The fact that the file could be corrupted in some cases is handled during loading anyway so, theoretically, it cannot lead to a second crash (the file is just discarded if it's corrupted) .
To terminate the process after the handler, I don't call exit, but I uninstall my handlers and then raise the exception again (or create one for a signal). Probably not the most elegant way, but I remember doing so because the application didn't quit otherwise (I should try again just returning as you recommend).
Finally, it would be nice to know what's going on with Core Audio. I see only two possible scenarios: 1) Core Audio is available but the report lacks some lines in the binary image section 2) Core Audio is unavailable for some reason I cannot imagine... Knowing that could be useful to continue investigating.
Here is a crash report I just try to join to a post two days ago encountering the same error:
2024-03-25_13-04-40.6314.crash
Thanks for your help.
I really don't understand what happened with the uploaded report. Many lines are missing from the main thread, but the version I have here is complete. I probably made a mistake somewhere, sorry for that. I join the file again.
universalExceptionHandler is a C function I call in my app delegate to catch errors from both Signals and NSExceptions. From there, I try to save a recovered file and open a NSAlert. When required, I use a dispatch block to open the alert in the main thread. But in practice, the alert seems only useful for exceptions while it fails with signals (at least some of them, I'm not sure about that...).
About C++, something is weird: all C++ codes used by this software are related to audio units and make an extensive use of Core Audio. However, I don't see any reference to Core Audio in the binary image. I cannot figure out how it is possible... Could it mean that something went wrong during the launch of the application? Or that Core Audio was crashed for some reason?
I wasn't able to post my answer because of the 'This post contains sensitive language. Please revise it in order to continue.' error... What can I do?
I was able to reproduce the issue in a small test project:
It seems to be that some of my custom views return NO from -mouseDownCanMoveWindow to ensure that clicking-dragging them doesn't move the window. But, for some reason, the system applies that request to the full toolbar. This is weird, but fortunately less difficult to work around: as some sub-views of these views are sub-classes of NSControl, they already prevent moving the window, so that the undesired effect remains limited to small areas.
Thanks for the tip, that was very helpful!
So here are the results of my last tests:
I was able to reproduce that bug with several other plugins from different manufacturers which have in common to be authorized with iLok. I don’t describe the symptoms here since they are exactly the same as in my previous post, but they affect dozens of plugins, all from the list of manufacturers coming from this beta-testing session (but it probably exist other cases).
I was also able to reproduce the bug with the VST or VST3 versions of the same plugins by opening them in an application hosting VST plugins loaded on the App Store (n-Track).
More surprisingly, I got many similar errors when opening Audacity, which is not a sandboxed application: each of these plugins triggered the same error message during validation. However, after quitting and relaunching the app, the plugins were finally available and worked normally.
In addition, the Logic Pro question revealed that version 10.7.9 was not sandboxed. Here's what the codesign command returns:
Executable=/Applications/Logic Pro X.app/Contents/MacOS/Logic Pro X
This command was executed on the Monterrey laptop, but I also loaded the last demo version on Ventura, and the version number is exactly the same. So, apparently, the most recent version of Logic Pro is still not sandboxed (which is confirmed by the fact that there is no container with its name in the user's library).
I would have deduced from all this that sandboxing was ultimately not a viable choice for a professional music or audio application, but I also found an iLok-protected plugin that maybe contradicts this idea:
My beta tester had pointed out that the GRM Tools plugins (from INA in Paris) didn’t crashed like the others, despite their iLok protection. I investigated this and, indeed, these plugins work perfectly in a sandboxed application on Ventura. The difference seems in the way they are authorized: whereas all other plugins open an iLok window during the validation process, and crash when they are instantiated, the GRM plugins require the customer to explicitly enter a validation code into the iLok manager after installing the plugin bundle and, after this step, there is no further (visible) interaction with iLok during validation or instantiation, and everything works as expected.
This finally suggests that manufacturers would have a solution for implementing Pace iLok in a way that is supported by a sandboxed application. But, on the other hand, there are so few sandboxed music apps that they’re probably not aware of this problem. We can therefore fear that this situation will never be resolved, especially if the Logic Pro team doesn’t move in the direction of sandboxing. I fully understand why they don’t, but it would be a great motivation (perhaps the only one) for manufacturers to fix their implementation of iLok...
Anyway, I now have many more info to think about the question of an alternative or additional distribution. Thanks a lot for your help!
I was finally able to reproduce the issue relatively quickly on Ventura. Here is a report of my tests (sorry for the length of this post):
Installed Plugin with iLok protection : IRCAM-Trax from Flux Audio (demo version).
OS: macOS 13.2.1,
Hardware: iMac Retina 4K, 21.5-inch, 2019, 3,2 GHz Intel Core i7 6 cores
Sandbox options: Outgoing Connection • Audio Input • User Selected Files: Read-write • Download, Music and Movies folders: Read-write
Test 1: (DEBUG Scheme) SANDBOX, HARDENED RUNTIME (Disable Library Validation)
=> Failed: Exact same behavior as in macOS 12 (except that the error message is presented fourth instead of twice). The logs provided by the console wasn't very helpful, they don't seem related to the plugin crash.
Test 2: (DEBUG Scheme) SANDBOX, HARDENED RUNTIME (Disable Library Validation • Disable Memory Protection • Allow DYLD Environnement Variables • Allow Unsigned Executable Memory • Allo Execution of JIT-compiled Code)
=> Failed: Exactly as the previous.
Test 3: (DEBUG Scheme)
=> Failed: After deactivating the sandbox and Hardened Runtime, no plugin wants to open although all plugins have been validated again (since the Application Support folder where the referenced plugins are stored has been changed). I realized that the Hardened Runtime's ‘Disable Library Validation’ key is required to open any Audio Unit plugin, even in a non-sandboxed application (probably except those from Apple ? I didn’t test it).
Test 4: (DEBUG Scheme) HARDENED RUNTIME (Disable Library Validation)
=> Failed: Here the iLok manager finally complained about the fact that it detected a debugger when I tried to open Trax… impressive.
Test 5: (RELEASE Scheme) HARDENED RUNTIME (Disable Library Validation)
=> Crashed: Strangely, with the release scheme, the application crashed when I tried to open Trax, I join the report below:
Anonymous-08:09:2023.crash
Test 6: (RELEASE Scheme) HARDENED RUNTIME (Disable Library Validation • Disable Memory Protection )
=> Success: Trax did open and all work as expected!
I was happy to finally find a solution, but the bad news is that this solution involves not using a Sandbox, which is not an option for me since I would like to distribute on the AppStore.
I still wondered how Logic and Garage Band do to work around this issue, so I tested again with Garage Band (on Monterey, the test was done with a licensed version of Logic Pro 10.7, which I think is sandboxed, and it had worked normally). But, finally, Garage Band 10.4.8 behave exactly the same way than my application:
First this message this message was displayed 4 times when I tried to open Trax:
Then, the error 100001 (reproduced in a previous post) was displayed.
Next, this more informative message was displayed:
Finally, you can see the content of the Garage Band’s temporary folder on this last screenshot: each error produces a file with another name and don’t overwrite the previous one. I hope this folder is cleared regularly because, depending on the size of the bundle, and how many times the user gets this error, this could consume some hard drive space unnecessarily...
So, it finally appears that there is a real conflict, at least from macOS 12, but it could earlier, between some plugins authorized via iLok and the OS (with the collection of my beta-tester, it was something like 75% of the iLok protected-plugins). The only remaining doubt is about the try we made with success on Logic Pro 10.7.9 and Monterey (but I have no prove that this version is sandboxed, the simplest explanation would be that it’s not...). Do you think there is something else I can do to find a solution ? It seems that these problems are far removed from my application and that I don't have much control over them. Thanks!
Following your suggestions, I try to reproduce the context of the crash as much as possible. I started with macOS 13 because I realized that the test I'd already done on Ventura was inconclusive because I was using a free demo plugin that wasn't authorized with iLok. Fortunately, other demo versions using iLok exist, so I’ll try this way. If the bug is not reproductible on Ventura, I'll consider downgrading to macOS 12 or installing it on an external drive. All this is time consuming, but hopefully I'll have more info in a few days. If so, I'll share them here.
Anyway, thanks for your help!
Oups, sorry for that ! (once opened in a text editor, such a report becomes difficult to read...)
Ok, since the same beta tester was here today, we tried again the whole sequence of operations and I have some more precise results:
Actually, the most common bug I describe in my first message don't produce any crash report. They just produce a big binary file (that seems to be an image of the bundle, I could upload it if you want) saved in the temp folder of the host application, display a first warning described in my previous message, and finally this one:
Even if the application didn't crash, we were not able to open any of these plugins with my application (practically they are unusable). I don't know if it could help, but it appears that all of these plugins are protected by iLok keys (https://www.ilok.com). Moreover they all open without problem in Logic Pro and Ableton Live.
On the other hand, the partial crash report I joined was related to an issue having some specificities: it also crashed the host application so that it produced a crash report. This second category seems related to less plugins, all related to the same manufacturer. It could be a problem of the plugin itself, however we were also able to open such plugins in Logic Pro without encountering the same issue.
I join a new report produced with a plugin of the latter type:
anonymous.crash
Yes it's the name of the plugin (an emulation of a famous Solid State Logic mixer), not something from my application.
I join the full report, anonymized as indicated in the instructions. Initially it's a file with the .ips extension (coming from the Console app), but because it was not possible to join it, I changed the extension to .crash.
Thanks!
Anonymous Crash Report.crash
The warning appears when trying to create an instance of any audio unit provided by some specific manufacturers and to open it in its editor (both things being merged as a single operation). The message body was followed by another sentence :
macOS cannot verify that this app does not contain any malicious software
and
created this file today at 15:32
The file in question was a file called NSCreatedObjectFileImageFromMemory-p47UEwps stored in the temporary directory (-[NSURL temporaryDirectory]) of the host application.
The panel buttons proposed either to trash the file or to cancel. Whatever the button clicked, a second dialog appears with an exclamation mark in a red icon and the Fatal Error 100001 message, and the plugin never opened (but the host application didn't crash).
Here is a part of the crash report (if you want, I can send the full report to your email address):
Date/Time: 2023-08-24 15:04:45.7209 +0200
OS Version: macOS 12.5.1 (21G83)
Report Version: 12
Bridge OS Version: 6.6 (19P6066)
Anonymous UUID: 0495D4E2-3107-8006-D560-7301560EB3CE
Sleep/Wake UUID: 2CFC336A-EB4A-4493-BB08-8CCCBA20E12F
Time Awake Since Boot: 73000 seconds
Time Since Wake: 596 seconds
System Integrity Protection: enabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0x0000000137c7cc34
Exception Codes: 0x0000000000000002, 0x0000000137c7cc34
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: Namespace SIGNAL, Code 10 Bus error: 10
Terminating Process: exc handler [41378]
VM Region Info: 0x137c7cc34 is in 0x137bb1000-0x137cc9000; bytes after start: 834612 bytes before end: 312267
REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL
mapped file 135c56000-137773000 [ 27.1M] r--/rwx SM=COW ...t_id=d8860195
GAP OF 0x43e000 BYTES
---> __TEXT 137bb1000-137cc9000 [ 1120K] r-x/rwx SM=COW ...le SSL 4000 G
__DATA 137cc9000-137cf5000 [ 176K] rw-/rwx SM=COW ...le SSL 4000 G
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 bx_console SSL 4000 G 0x137bdf476 0x137bb1000 + 189558
1 bx_console SSL 4000 G 0x137c7e739 AUV2AdapterEntry + 439641
2 dyld 0x1148ca334 dyld4::RuntimeState::notifyObjCInit(dyld4::Loader const*) + 170
3 dyld 0x1148cff03 dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 167
4 dyld 0x1148cffb2 dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 108
5 dyld 0x1148dee00 dyld4::APIs::dlopen_from(char const*, int, void*) + 592
6 CoreFoundation 0x7ff815cd6994 _CFBundleDlfcnLoadBundle + 149
7 CoreFoundation 0x7ff815d5094f _CFBundleLoadExecutableAndReturnError + 484
8 AudioToolboxCore 0x7ff81712efa5 APComponent_FromBundle_Loadable::ResolveFactoryFunction() + 67
9 AudioToolboxCore 0x7ff81712ded2 APComponent::newInstance(unsigned int, bool, void (OpaqueAudioComponentInstance*, int) block_pointer) + 126
10 AudioToolboxCore 0x7ff817223c90 instantiate(OpaqueAudioComponent*, unsigned int, bool, void (OpaqueAudioComponentInstance*, int) block_pointer) + 311
11 AudioToolboxCore 0x7ff81722402c __AudioComponentInstanceNew_block_invoke + 88
12 AudioToolboxCore 0x7ff8170b5479 Synchronously + 87
13 AudioToolboxCore 0x7ff817223ee2 AudioComponentInstanceNew + 183
14 <AppName> 0x10f9274e3 mx::EditableDevice::instanciate(AudioComponentDescription*, unsigned short, unsigned char) + 77
15 <AppName> 0x10f9273cf mx::EditableDevice::EditableDevice(mx::MixerBase*, AudioComponentDescription*, unsigned short, bool) + 101
16 <AppName> 0x10f9087ab mx::Insert::Insert(mx::MixerBase*, AudioComponentDescription*, unsigned char, unsigned char, unsigned char) + 41
17 <AppName> 0x10f90e957 mx::Track::insertEffectAtIndex(unsigned long long, unsigned char, unsigned char, unsigned char, unsigned char) + 559
Thanks for you reply. I tested one of these plugins on an intel iMac running macOS 10.15 without problem. Today, I also tested the same plugin on an intel iMac running macOS 13.2.1 and it opened normally. So, no, I wasn't able to reproduce the bug. But it could be due to the fact that the systems I used was different (unfortunately I don't have a machine running macOS 12 here). For Catalina, the result is probably related to the fact the system is less secure than Monterey, but for Ventura it's unclear if it's related to the plugin, the hardware or the system.
I also forgot to say that, during this beta testing session, we tried to disable Gate Keeper on the beta tester machine using the 'sudo spctl --master-disable' command, and that was successful. This seems to confirm your assumption. Knowing that, do you think there is a possible workaround using some other entitlements ?
dp