Useless XCTMetrics (XCTMemoryMetric & XCTCPUMetric) when Measuring XCUIApplicaiton

Question

Created Mar ’21

Replies 4

Boosts 1

Views 2.4k

Participants 6

I've noticed that XCTMemoryMetric & XCTCPUMetric seem to record empty or nonsensical data when running a UI test flow for an XCUIApplication.

I'm attempting to test the memory and CPU footprint for a SwiftUI iOS app using the following code:

Code Block Swiftfunc testBasicFlowMemory() throws{
	let app = XCUIApplication()
	app.launch()
	var metrics:[XCTMetric] = []
	metrics.append( XCTClockMetric() )
	metrics.append( XCTMemoryMetric(application: app) )
	metrics.append( XCTCPUMetric(application: app) )
	self.measure(metrics: metrics){
		/*Method which uses XCUI API to test the app instance*/
		self.runBasicFlowTest(app: app)
	}
}

When I run the test above, I notice runBasicFlowTest is executed 6 times (even though the metics only record 5 values.

Of the three metrics I wanted to track only XCTClockMetric returned meaningful data:

[Clock Monotonic Time, s] values: [114.728229, 114.944770, 121.813337, 116.394432, 117.491242]

XCTMemoryMetric mostly recorded 0.0 or nonsense data:

[Memory Physical, kB] values: [3596.288000, 0.000000, 0.000000, 0.000000, 0.000000]
[Memory Peak Physical, kB] values: [0.000000, 0.000000, 0.000000, 0.000000, 0.000000]

XCTCPUMetric likewise recorded 0.0 or nonsense data:

[CPU Instructions Retired, kI] values: [0.000000, 206223944.266000, 0.000000, 0.000000, 211895544.471000]
[CPU Cycles, kC] values: [0.000000, 252096240.472000, 0.000000, 0.000000, 257352232.305000],
[CPU Time, s] values: [0.000000, 86.585296, 0.000000, 0.000000, 0.000000]

I'm on Xcode Version 12.4 (12D4e), and my app is targeting iOS 14.4 on a simulated iPhone 11 Pro.

Has anyone had any luck using XCTMetrics with UI Tests?

Boost

Answer 1

raffis OP

Mar ’21

Hi,

I've tried performance testing with UI tests in Xcode 11 and I came to the same conclusion regarding XCTCPUMetric and XCTMemoryMetric... Useless. The data collected does not make sense or I also see a lot of "0" in the values recorded.

I can't read anything about it online, no bug reports or complains on SO. I'm glad someone else is experiencing the same issue.

However since Xcode 12, I can't see the test report popup and can't set the baseline for XCTStorageMetric, XCTCPUMetric or XCTMemoryMetric. The performance tests just always pass. Have you experienced something similar?

It would be great if someone from the XCTest team could respond.

1

Answer 2

cwiedemann OP

Apr ’21

I'm having the same problem. It has rendered my XCTStorageMetric
, XCTCPUMetric and XCTMemoryMetric
useless.

1

Answer 3

paultas2 OP

Sep ’21

I'm experiencing the same issue

1

Answer 4

mschnaserpcc OP

Oct ’22

Similar problem here, except I'm looking to get one memory peak value that isn't zero. This is using the sample/default project template:

final class MemoryMeasuringSampleAppUITests: XCTestCase {

  func testLaunchPerformance() throws {
    measure(metrics: [XCTApplicationLaunchMetric()]) {
      XCUIApplication().launch()
    }
  }

  func testLaunchPerformanceMemory() throws {
    measure(metrics: [XCTMemoryMetric()]) {
      XCUIApplication().launch()
    }
  }
}

What's interesting is, I also see this message printed in my console several times while my test is running:

2022-10-17 09:42:22.344941-0500 MemoryMeasuringSampleAppUITests-Runner[21734:14123818] [Default] Unable to reset the peak interval footprint of pid 21734`

Any ideas?

0