Xcode 15 performance tests ignore baseline

I'm trying to write performance tests with Xcode 15, but it's not working: even though setting a baseline seems to work (baseline data files are created), Xcode seems to ignore that baseline altogether whenever the test is run. As a result, performance tests never fail, no matter how bad performance gets.

The source editor persistently displays a “No baseline average for Time” message.

Has anyone else encountered this? Figured out how to sidestep the issue?

Of note, the issue is easily reproduced: create a new (macOS in my case) app project with tests, add a dummy performance test; run the test once and save its baseline, and despite this subsequent tests will always succeed, even if you significantly slow the dummy test down.

(I've filed this as FB13330139)

Yes, I also see this in Xcode 15.1 (15C65) running on 14.2.1.

Same result here with Xcode 15.1

I had the same issue. The baseline is now working on Xcode 15.2

The baseline is partially working for me on Xcode 15.2. It doesn't appear performance tests with the measure(metrics:block:) method are taking their baseline into account. Also Xcode is not suggesting setting а baseline for such a test. Tests with the measure(_:) method are working fine.

in addition to my previous comment

Seeing the same issue in Xcode 15.3

Same results in Xcode Version 15.3 (15E204a)

Same issue in Xcode 15.4

same issue in Xcode 14.2 (I have both 15.2 and 14.2). Is there any workaround for this? Using measure { } works for measuring time, but I need to measure memory metrics too and it's currently not possible.

Xcode 15 performance tests ignore baseline
 
 
Q