Problem using test plans with local Swift Packages

Question

mul OP

Created May ’20

Replies 3

Boosts 0

Views 4.8k

Participants 2

I'm using Xcode 11.5.

I'm using test plans.

I have an app that uses several local Swift Packages.

When I test the app, I'd like to run the tests for the packages.

In the current scheme I've tried putting a checkmark for "SomePackageTests" under the Test column in the Build action.

The test plan, "SomeTestPlan", seems to show up as its own target in the Build action. It has a disclosure icon to its left. When I expand, I see the test targets for the app and for all local packages. The expanded test targets don't have their own checkboxes. I've also tried putting a checkmark for "SomeTestPlan" itself under the Test column in the Build action.

When I edit the test plan, I see the test targets for the app and for all local packages. I've tried enabling the package tests there too.

The app tests (unit and UI) build and run successfully. For example, they show up in the test navigator. I can right-click on them, select "Run", and they build and run successfully.

The package tests are also listed in the test navigator. They build successfully—I see a "Build Succeeded" message pop up—but they don't run successfully. The error looks like this:

2020-05-26 11:03:33.947959-0400 xctest[55965:34366669] Failed to create a bundle instance representing '/Users/someuser/Library/Developer/Xcode/DerivedData/SomeApp-eecekbbecwouyeagzirvwewncpem/Build/Products/Alpha-Debug-iphonesimulator/SomePackageTests.xctest'. Check that the bundle exists on disk.

Program ended with exit code: 1

(Note that "Alpha-Debug" is the name of both a build configuration and a test plan. I assume the above is a reference to the test plan, but I could be wrong.)

I *think* the problem is that Xcode is looking in the wrong place—but only for the package tests, not the app tests.

It's correct that there's no directory at that location. There is, however, a directory by that name in this location:

/Users/someuser/Library/Developer/Xcode/DerivedData/SomeApp-eecekbbecwouyeagzirvwewncpem/Build/Products/Variant-UBSan/Alpha-Debug-iphonesimulator/SomePackageTests.xctest/

Here's the key difference.

The first path has ".../Products/Alpha-Debug...".

The second path has ".../Products/Variant-UBSan/Alpha-Debug...".

"UBSan" is the name of my default (and one-and-only) test configuration in the test plan.

So: it seems like building builds to a test configuration dir, but running looks in a non-test configuration dir. At least for packages.

Is this an Xcode bug, or did I misunderstand something?

Again, the goal is to run all of my package tests when I test my app.

PS I *think* this was working before I switched to test plans, but I'm not absolutely certain.

Answered by mul in 421933022

I was able to resolve this. As far as I can tell, it's an Xcode bug.

The short version is that Xcode seems to assume that at least one test configuration will use the shared (default) test configuration, unmodified. That assumption can of course be wrong; it's possible that no test configuration exactly matches the defaults. The mistake can prevent Xcode from finding the test target bundles for local Swift Packages.

The long version...

* A test plan has a shared (default) test configuration.

* A test plan has one or more actual test configurations.

* Xcode places build products in directories whose names are based on their differences from the shared (default) configuration.

* The build products for test configurations that don't have any meaningful differences are placed under the root, e.g.: "/Products/***/...".

* The build products for test configurations that do have meaningful differences are placed under subirectories whose names are derived from the differences, e.g.: "/Products/Variant-ASan-UBSan/***/...", "/Products/Variant-UBSan/***/...".

* Some part of Xcode assumes that at least one test configuration will match the shared defaults. In other words, some part of Xcode assumes that a test bundle will always exist under the root, in "/Products/***/...".

* In my case that wasn't true; my one and only test configuration enabled UBSan. That caused its build products to be placed in "/Products/Variant-UBSan/***/...". So, at least for my local Swift Packages, Xcode was looking in the wrong place: it was looking in the root rather than in a variant subdirectory under the root.

* The workaround was to create a test configuration that didn't differ from the default.

===

As an aside, I mentioned that the local Swift Package targets were listed in the scheme. I eventually realized that that was a hold over from my pre-test plan configuration; that's how I was getting Xcode to test all packages together with the main app before converting. With the test plan in place I no longer need them listed directly in the scheme. Unfortunately, removing them didn't fix my actual problem.

Boost

Answer 1

mul OP

May ’20

Accepted Answer

I was able to resolve this. As far as I can tell, it's an Xcode bug.

The short version is that Xcode seems to assume that at least one test configuration will use the shared (default) test configuration, unmodified. That assumption can of course be wrong; it's possible that no test configuration exactly matches the defaults. The mistake can prevent Xcode from finding the test target bundles for local Swift Packages.

The long version...

* A test plan has a shared (default) test configuration.

* A test plan has one or more actual test configurations.

* Xcode places build products in directories whose names are based on their differences from the shared (default) configuration.

* The build products for test configurations that don't have any meaningful differences are placed under the root, e.g.: "/Products/***/...".

* The build products for test configurations that do have meaningful differences are placed under subirectories whose names are derived from the differences, e.g.: "/Products/Variant-ASan-UBSan/***/...", "/Products/Variant-UBSan/***/...".

* Some part of Xcode assumes that at least one test configuration will match the shared defaults. In other words, some part of Xcode assumes that a test bundle will always exist under the root, in "/Products/***/...".

* In my case that wasn't true; my one and only test configuration enabled UBSan. That caused its build products to be placed in "/Products/Variant-UBSan/***/...". So, at least for my local Swift Packages, Xcode was looking in the wrong place: it was looking in the root rather than in a variant subdirectory under the root.

* The workaround was to create a test configuration that didn't differ from the default.

===

As an aside, I mentioned that the local Swift Package targets were listed in the scheme. I eventually realized that that was a hold over from my pre-test plan configuration; that's how I was getting Xcode to test all packages together with the main app before converting. With the test plan in place I no longer need them listed directly in the scheme. Unfortunately, removing them didn't fix my actual problem.

4

Answer 2

Engineer OP

Apple

Jun ’20

Thanks for the question, yes this appears to be a bug which we will investigate further (63677197).

A temporary workaround you can try is to disable all Sanitizers (e.g. ASan, TSan, UBSan) in your test plan configuration(s), which should allow the build product to be in the expected location. Other, non-sanitizer settings in the test plan may still be varied.

0

Answer 3

mul OP

Jul ’20

I think I've run into another variation of this (now in Xcode 11.6). I think there are two additional problems.

I have two local Swift packages that reference two remote Swift packages. (The remote packages are Apple's SwiftProtobuf and RevenueCat.)
When I build for testing, both of the local Swift packages generate errors.
The error for the package that references SwiftProtobuf is a compiler error:

Code Block Command CompileSwift failed with a nonzero exit code

The error for the package that references RevenueCat is a linker error:

Code Block Undefined symbols for architecture x86_64:
  "_ubsan_handle_nonnull_arg", referenced from:
      -[RCPurchaserInfo setUpDateFormatter] in Purchases.o
      -[RCPurchases dispatch:] in Purchases.o
  "_ubsan_handle_float_cast_overflow", referenced from:
      -[NSDate(RCExtensions) millisecondsSince1970] in Purchases.o
  "_ubsan_handle_load_invalid_value", referenced from:
      _RCSetShowDebugLogs in Purchases.o
      _RCShowDebugLogs in Purchases.o
      _RCDebugLog in Purchases.o
      _RCErrorLog in Purchases.o
      -[RCSystemInfo initWithPlatformFlavor:platformFlavorVersion:finishTransactions:] in Purchases.o
      -[RCSystemInfo finishTransactions] in Purchases.o
      -[RCSystemInfo setFinishTransactions:] in Purchases.o
      ...
  "_ubsan_handle_pointer_overflow", referenced from:
      _32-[NSData(RCExtensions) asString]_block_invoke in Purchases.o
  "_ubsan_handle_type_mismatch_v1", referenced from:
      -[RCInMemoryCachedObject setCacheDurationInSeconds:] in Purchases.o
      ___32-[NSData(RCExtensions) asString]_block_invoke in Purchases.o
      -[RCSystemInfo setFinishTransactions:] in Purchases.o
      -[RCHTTPClient setSystemInfo:] in Purchases.o
      -[RCIntroEligibility setStatus:] in Purchases.o
      -[RCPurchaserInfo setUpDateFormatter] in Purchases.o
      -[RCPurchases allowSharingAppStoreAccount] in Purchases.o
      ...
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Here's the first issue: Both errors seem to be caused by the fact that one of my test configurations uses UBSan.
As per the original post, my scheme has two configurations, a default one and one that enables UBSan.
Here's the second issue: The errors persist even if I disable the UBSan test configuration.
If I turn off UBSan in the (disabled) UBSan test configuration, everything builds and runs successfully.

1