How to write a performance test case

Question

iamharrypotter OP

Created Apr ’20

Replies 1

Boosts 0

Views 1.7k

Participants 2

Currently I am facing some issues on writing a idea performance test case.

Problem which I am facing could be lack of documentation.

Method under test.

    func addStrings() {
        let maxCount = 100000
        var array:Array = Array()
        for _ in 0..            array.append("Test")
        }
    }

Performance test case.

func testPerformanceExample() throws {
        // This is an example of a performance test case.
        let strings = StringsCreator()
        self.measure {
            // Put the code you want to measure the time of here.
            strings.addStrings()
        }

How do i set the ideal paramters for testing?

What is the relation between MAX STDDEV , RESULT , AVERAGE and BASELINE.

Looking for a descriptive answer with example. Thanks in advance.

Boost

Answer 1

Developer Tools Engineer OP

Apple

Jun ’20

Hi,

The measure method runs the block of code a number of times to measure its performance. This results in a number of gathered measurements. Since each of these measurements can vary to some extent they are averaged out to produce a single value that's representative of the list of measurements. This is the "AVERAGE" value.

If you've run this test before you can save the average value from one test execution as the value to compare future test executions against. This saved value is the "BASELINE".

If a performance test has a saved baseline value, future test executions will compare their average values to this baseline and compute how much faster to slower the code has become. This value is the "RESULT".

Each of these values is the average of a number of measurements. The average value is taken to be representative of the list but depending on the how similar or different the values are to each other, the average may be more or less representative of the list. To give a sense of how much variation there is in list of measurements, their "Standard deviation" is computed.

A high standard deviation could mean that the performance test is having noisy data and that the results could vary from run to run even, even if the code under test doesn't change. To account for this, some amount of variation is allowed when comparing two performance results but there's also a max standard deviation that's allowed in the indication of a pass or a fail. This value is the "MAX STDDEV".

If you find that your performance test varies too much even when you didn't change the code or that its results are unreliable, then you can configure the measurement to run more times. This will produce more measurements for each test execution which statistically should produce a more representative average. You can do this by creating and configuring a XCTMeasureOptions object that you pass to the measure method:

Code Block swiftlet options = XCTMeasureOptions()
options.iterationCount = /* Tweak this value on a test-by-test basis */ 
measure(options: options) {
		/* The code that's being measured */
}

There is no configuration that's perfect for every situation. The default configuration is good in most of situations but if the code that's being tested runs very quickly you may want to experiment with a higher iterationCount value in an attempts to average out some of the noise and get a more representative average value.

4