xctest measure not failing when performance bounds are crossed

Hello everyone,


I had a question around the XCTest measure https://developer.apple.com/documentation/xctest/xctestcase/1496290-measure API. My test does not fail, even if the standard deviation is not met.


measured [Time, seconds] average: 0.054, relative standard deviation: 10.368%, 
values: [0.059608, 0.053853, 0.048230, 0.066531, 0.049239, 0.054683, 0.047571, 0.056887, 0.051896, 0.050362], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "Local Baseline", baselineAverage: 0.000, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 0.000%, maxRegression: 0.100, maxStandardDeviation: 0.100


Is there something wrong with the setup? Would appreciate any insights.


Thanks,

Sudeep.

Up vote post of bhowmisu Down vote post of bhowmisu
1.4k views
Posted by
bhowmisu
Reply
Add a Comment

Replies

As you can see in your log, there is also an absolute time threshold:

maxRegression: 0.100


As your times are below this threshold, performance test won't fail. I have the same issue and didn't find any way to customize this threshold. For now the only solution I have is increasing the amount of work done in measured block so that it takes at least 0.1s in optimized scenario. This way, regressions are detected when running tests.

Posted by
Ceylo
Post not yet marked as solved Up vote reply of Ceylo Down vote reply of Ceylo
Add a Comment