Some more information: This can pass on one run and fail, with no code changes on the next, however as Jens explains the failure has the hardened flag set but not the hard and kill flags We noticed the script we used in Xcode to build the test targets from the xctest app was passing on its environment and hence our test hardened apps had XCTest framework included, we removed this ( env -i xcodebuild ...) but still get the intermittent failures The test signed/hardened apps are command line apps (as Jens said, no bundles) only the hardening bit (0x10000) is set in the flags, and the dynamic status always has the same 0x10000 set but is 50/50 missing the 0x300 values: Our log line shows us trying to AND the 0x301 with the dynamic status flags but failing to get the 0x300 matching, also not too sure what the high 0x22 bits are? code for pid 15481 does not have required status : 0x22010001 does not match 0x00000301 Explicitly setting the hard and kill flags (and hardened) in codesign works fine, but it seems manual work compared to asking for hardened only Our test command line app (the file create can be watched for by the launcher show we know the app is in main before we tests it's flags) #import <Foundation/Foundation.h> int main(int argc, const char * argv[]) { // if we have a filepath on the command line create a file so the caller knows we have started if (argc == 2) { // get the filepath NSString* filename = [NSString stringWithUTF8String:argv[1]]; // fill with any data, we just use the filepath NSData* data = [filename dataUsingEncoding:NSUTF8StringEncoding]; // write to a file [data writeToFile:filename atomically:true]; } // Wait for ever, the caller will call terminate when they are done BOOL shouldKeepRunning = YES; NSRunLoop *runLoop = [NSRunLoop currentRunLoop]; while (shouldKeepRunning && [runLoop runMode:NSDefaultRunLoopMode beforeDate:[NSDate distantFuture]]) { sleep(1); } }

With Quinn's test code: For pages app I got 0x12000 For failing cs-validation-test-hardened I got 0x10000 For passing cs-validation-test-hardened I got 0x10000 For Info: kSecCodeSignatureLibraryValidation = 0x2000 kSecCodeSignatureRuntime = 0x10000 All had runtime set, and I thought setting ENABLE_HARDENED_RUNTIME would also include library checking too?

The MS Azure provides the build VM and we request macOS-Latest (which is currently 12.7.1) When we get 12.7.1 the hardening test works as expected :-) 50:50 we get 12.6.9 (which is strange since MS lists that we would get 12.7.1?) We then git fetch all the code and do an xcodebuild test xxxx On 12.6.9 we get the following: Fault with time server (we fixed this with an extra sync command) The hardening flags are wrong, ie we ask for hardened and the dynamic test says we do NOT have the hard and kill flags set So yes we are running the test app and the xcodebuild test (which uses the app) on a 12.6.9 VM