Accessing Window Server and other GUI services when running tests via SSH vs LaunchAgent

So what’s you actual goal here? To automate UI tests using xcodebuild? Or are you asking about “UI testing” in a more general sense?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

The end goal is to automate UI testing in the general sense, in a way that does not result in false positives/negatives due to running in a slightly different environment than what an end user would.

Most CI/CD tools (GitHub, GitLab, CircleCI, Jenkins, etc) provide some sort of runner/agent that acts as the parent process of the test suites being run (which can be implemented via xcodebuild, CTest, GoogleTest, etc.).

Sometimes these runners/agents are launched as LaunchAgents, and sometimes they are running on a separate machine (host), and use SSH to run the command on the "worker node" (VM).

The latter (using SSH) has the advantage that it simplifies provisioning and CI integration quite a bit. It also matches the situation when a developer is investigating a CI test failure, as they will typically SSH into the CI node and run the test there. Having the two situations run the test in a similar fashion is advantageous.

So if running UI tests via SSH is close enough to what would be the environment when run via a LaunchAgent that would be great.

If that's not the case, then some details on why this is problematic (in practice) would be appreciated, to be able to analyze the impact and importance of moving existing CI solutions over to a different approach.

Thanks!

Of course the responsible process of the test suite will be different in the two cases, either ci-agent-runner if launched via a Launch Agent, or sshd-keygen-wrapper if launched via SSH.

Let's assume for this topic that the test suite is launched disclaimed, making it its own responsible process regardless of the parent process hierarchy.

I see two factors in play here:

• Type of product — If you were building an actual product to deploy widely, doing this stuff from an agent is the only option that I’m confortable with. However, your specific use case offers more wiggle room, because you’re only deploying to a limit number of fairly knowledgeable users.

• UI testing — The question here is whether xcodebuild has enough smarts to run a UI test reliably even though it’s been started in an SSH session.

Unfortunately, I don’t have an answer for the second question. My focus in mainly on products rather than tools, and I’ve never dug into what, if any, smarts that xcodebuild applies to this problem.

Lemme go back to your original question:

Does modern macOS versions expose GUI services to the global or non-GUI per-session bootstrap namespace … ?

The answer to that is “No.” The rules haven’t changed in that regard.

Does modern macOS versions expose … otherwise enable UI testing from SSH sessions

The answer to that is “No, but also yes.”

Running GUI code in an SSH session is always a bad idea. However, modern versions of macOS [1] provide the infrastructure that makes it possible to cross sessions, that is, starting code in the GUI session from the SSH session.

The obvious follow-on question is whether xcodebuild uses that, and I don’t have an answer for you there.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] By “modern” I mean macOS 10.10 or later, which saw the introduction of launchd ‘2.0’.

Thanks Quinn!

The use-case here is UI testing, but not necessarily though xcodebuild, so any magic that xcodebuild does internally to handle being launched from a SSH session would not help in the general case.

The general case would be a suite of app bundles or standalone executables, launched via e.g. CTest. For example test-window-positioning.app which would be launched by executing test-window-positioning.app/Contents/MacOS/test-window-positioning and checking its exit code.

Running GUI code in an SSH session is always a bad idea.

The original Technical Note TN2083 lists some cases of why running GUI code from an SSH session is a bad idea (such as not being able to interact with the Dock), but these cases seem to work fine in macOS 12-15. Do you know of any concrete cases where this will be an issue? Knowing these will help analyze the potential risk of current solutions based on SSH sessions.

However, modern versions of macOS [1] provide the infrastructure that makes it possible to cross sessions, that is, starting code in the GUI session from the SSH session.

What infrastructure would this be? I know of open, but are there others?

What infrastructure would this be? I know of open, but are there others?

I tried to launch an executable with open --stdout /tmp/stdout --stderr /tmp/stderr -W ./foo --args bar baz but that doesn't seem to give the expected result:

1. The executable is launched via Terminal.app, instead of directly (like an application bundle would have)
2. The --args are not propagated to the executable. Presumably they are passed to Terminal.app, which ignores them (?)
3. The executable's stdout/stderr is not written to the files. Presumably it's the output of Terminal.app that gets written to these files (?).

Is there a way to launch an executable via the Launch Services APIs or otherwise, that:

1. Passes on arguments
2. Propagates stdout/stderr
3. Waits for the executable to end

? Perhaps via launchctl bsexec? Or registering another application as the default handler of public.unix-executable instead of Terminal, which does the above?

The use-case here is UI testing, but not necessarily though xcodebuild, so any magic that xcodebuild does internally to handle being launched from a SSH session would not help in the general case.

OK, that makes this more of a product than a tool, so my advice is gonna be more risk averse.

The original Technical Note TN2083 lists some cases of why running GUI code from an SSH session is a bad idea (such as not being able to interact with the Dock)

Right. When I wrote that I included the Dock example because it was a quick way to demonstrate the issue. You shouldn’t put any stock in the fact that this example no longer works. If you’re building a product I recommend that you follow the rules outlined in that technote rather than rely on these implementation details. Specifically, have your code start in the right context rather than try to switch contexts.

I recently had almost the same discussion with a different developer. Have a read through Session, Desktops and login screen. I’m especially proud of “the Five Stages of Mac Screen Sharing Developer Grief™” (-:

I know of open, but are there others?

launchd ‘2.0’ added a bunch of commands to launchctl and those commands can target specific domains. If you really want to make this work, use bootstrap to load your agent into the gui/<UID> domain and then start the executable from that agent. That way your executable will start in the right context.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"