Hi,
I'm writing performance tests using facilities provided by XCTest with Xcode 10.2:
func testPerformanceExample() {
self.measure {
...
}
}My issue is that the code I measure is supposed to be very fast (around 1ms) and I want the test to verify that no regression is introduced. However making the code much slower currently does not trigger a test failure, although Xcode clearly notices that it is slower as shown in these screenshots:
https://artoverflow.io/downloads/worse.png
https://artoverflow.io/downloads/performance_result.png
The test logs contain this:
Test Case '-[DummyTests.DrawingDrawableTests testPerformanceExample]' started.
[...]/DrawingDrawableTests.swift:132: Test Case '-[DummyTests.DrawingDrawableTests testPerformanceExample]' measured [Time, seconds] average: 0.006, relative standard deviation: 14.419%, values: [0.008725, 0.005750, 0.005917, 0.005780, 0.005808, 0.005798, 0.005741, 0.005819, 0.005837, 0.005759], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "Local Baseline", baselineAverage: 0.002, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[DummyTests.DrawingDrawableTests testPerformanceExample]' passed (0.471 seconds).I notice that they contain "maxRegression: 0.100" which looks to be an absolute threshold in seconds. And indeed, making my block take more than 0.1s actually makes the performance test fail. But this makes XCTestCase.measure() pretty useless for anyone that wants to run really optimized code. In current case this is for real time rendering and I want the test to detect when the implementation is not able to reach 60 fps.
Of course I could manually run the app and check, but I want to reduce manual testing time. That's where XCTest is supposed to help.
Is there any way to configure this maxRegression or more generally to make XCTestCase.measure() usable for fast blocks?
At the moment the only workaround I have is to artificially increase the amount of work being done in the measure blocks, but this has 2 very annoying drawbacks:
- the test is much slower than needed
- this arbitrarily chosen amount of work only allows detecting regression on my own hardware. If the test is run on faster hardware, the amount of measured work needs to be increased again, so this makes my test code not future-proof at all.
