Swift Process with Psuedo Terminal (PTY)

[I’m responding here, rather than on your Swift Forums thread, because this is somewhat off-topic for Swift Forums.]

You mentioned Xcode so I’m presuming that this is on the Mac; let me know otherwise.

Pseudo terminals are a royal pain in… any language (-: If you’re trying to handle one specific case, I recommend that you do your puppetstringing via expect, which takes care of all this silliness for you. See its man page for details.

If that doesn’t meet your needs then post back and I’ll take a proper look at your Process code.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@eskimo, you were the one I was hoping would reply since you had some thoughts for the original thread I saw, thanks for getting back to me!

You mentioned Xcode so I’m presuming that this is on the Mac; let me know otherwise.

Correct, we're building a CLI tool in Swift. It should ideally run on both macOS and Linux if at all possible (we have some plans to use the cli on a server at some point)

 I recommend that you do your puppetstringing via expect, which takes care of all this silliness for you.

Took a look at the expect manpage and while I can see it being useful to automate some aspects of this, or to add a special case for this particular tool were running, I'm really looking to solve this on a more generic basis.

For some context: As I mentioned, we're building a CLI tool in Swift, that among other things can automate some of the iOS developer tasks we have at work. To do this, we leverage existing tools for example: git, swiftlint, etc and use the binaries from those programs to create an interface in the CLI that abstracts their usage. This has been working fine until we ran into issues using TSCBasic.Process to run the xcodes binary as it requires some user interactivity to enter a userID and password. The recommendation in that Swift bug was to move to Foundation.Process which were trying to do now. However, now I'm stuck with not being able to run thexcodes binary using Foundation.Process because of the issue posted in this thread.

Any insight on how to get this working with Foundation.Process would be great! Essentially, I want to be able to execute some arbitrary binary and have it run as if it were run via a users terminal.

to run the xcodes binary as it requires some user interactivity to enter a userID and password.

I’m not sure what xcodes is. Did you mean xcodebuild?

If so, I don’t really understand this requirement:

It should ideally run on both macOS and Linux

xcodebuild isn’t a Linux thing.

Moreover, if the goal is to run xcodebuild then you’d be much better off special casing that. Again, writing generic pseudo terminal code to puppetstring arbitrary command-line tools is hard.

As to the question of how hard it is, there are three issues:

• Dealing with pseudo terminals in general (A).

• Integrating your pseudo terminal with Process (B).

• Actually puppetstringing the final process (C).

With regards A, I’m going to recommend that you read generic Unix references on that subject. Terminal devices have a lot of history on Unix-y systems, and thus there’s a lot to learn. Most of this isn’t covered by Apple documentation. Moreover, you’ll encounter subtle differences between the various Unix flavours.

My go-to reference for this sort of thing is Advanced Programming in the Unix Environment.

With regards C, this is very specific to the tools that you’re trying to puppetstring, so I’m going to leave that to you (-:

With regards B, I can certainly help you with that. My advice is that you start by trying to puppetstring a test tool that you control. The advantage of doing that in that you ‘own’ both sides of the communication channel, so you can tell whether the problem is on the send side or the receive side. Once you get your test tool working, you can start targeting the real tool with the knowledge that your core code is correct and any remaining problems are specific to how that tool works.

Apropos that, let me introduce you to the the readpassphrase function (-: You can call it from Swift like so:

func readPassPhrase() -> String? {
    var buffer = [CChar](repeating: 0, count: 256)
    return buffer.withUnsafeBufferPointer { buf -> String? in
        guard let p = readpassphrase("Enter password: ", &buffer, buffer.count, RPP_REQUIRE_TTY) else {
            return nil
        }
        return String(cString: p)
    }
}

IMPORTANT readpassphrase returns a pointer to the buffer you pass in, so you have to be careful that this buffer persists after the call. That’s why I used withUnsafeBufferPointer(…) rather than an ampersand. See The Peril of the Ampersand for more on this.

I wrapped this in a tool like so:

import Foundation

…

func main() {
    let p = readPassPhrase() ?? "-"
    print("passphrase: \(p)")
}

main()

Running it from Terminal works as expected:

% ./Test688534
Enter password:             <<< Type "opendoor" and press Return.
passphrase: opendoor

But look at this:

% ./Test688534 < /dev/null
Enter password: 

If stdin is set to /dev/null, why is this prompting for a passphrase? The answer is that it works by opening /dev/tty. And that brings us back to point A. So far your attempts to run the tool via Process have assumed that overriding stdin is sufficient, but that’s simply not the case )-:

Finally, if the platform you’re targeting doesn’t support readpassphrase, try getpass instead. Note that, at least on macOS, this is implemented using readpassphrase but it’s more forgiving because it doesn’t use the RPP_REQUIRE_TTY.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I’m not sure what xcodes is. Did you mean xcodebuild?

Nope, this xcodes is an external tool that manages downloading different xcode versions. As part of this it reads apple id from the terminal and the password (both of which are causing issues with the standard Process setup I had, hence needing to look into PTYs)

Apropos that, let me introduce you to the the readpassphrase function (-: You can call it from Swift like so

Thanks for this! It's a good test bed to work from as it demonstrates the issue I'm facing. I created a small binary from the code you posted and ran it in my terminal by itself, and get the same result you do (it prompts, reads the password and displays it).

I then tried to call that binary from within my CLI using the PTY stuff I have above and get essentially no output:

clide (luis/foundation-process) > swift run clide
[0/0] Build complete!
password  <<<<< This was me entering input on my terminal 
password  <<<<< This was displayed after I hit enter

I can see what you mean by "overriding stdin is sufficient, but that’s simply not the case" as it doesn't seem to be working here. In my code I read stdin and then write to the parent PTY handle, from my understanding that should then cause the child handle to interpret that as input, so I'm just confused why that doesn't behave like that? Or why I don't see the "Enter password" prompt at the beginning.

Thanks again for following up, I'll check out the book but I'm hoping to be able to get this done in a reasonable amount of time. Currently, I'm just stuck because I'm not sure where I've gone wrong in my setup.

OK, so, two things:

• In addition to having a test tool that calls readpassphrase, you should create another test tool that deals with stdin and stdout in the usual way. That allows you to verify that your Process code is handling the basics correctly.

• Earlier I wrote:

  The answer is that it works by opening /dev/tty.

  Sadly, I’m not an expert in Unix terminal management but I did a little more digging and I think the key term you need to research is controlling terminal. See the so-named section in the termios man page.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@eskimo,

Reading up on "controlling terminal" was helpful to understand more about whats happening when spawning these processes, I'm still not sure how to actually apply this to my usecase as I've tried to use ioctl(childDescriptor, TIOCSCTTY) in my code to become the controlling terminal with the child part of the PTY and this returns back -1 indicating a failure. Reading online sources have been difficult to grasp as most of them seem be using fork and then ioctl but, from the compiler error I got, it seems fork is not supported in Swift.

In addition to having a test tool that calls readpassphrase, you should create another test tool that deals with stdin and stdout in the usual way. 

Took your advice here and created a simple program that just reads input and outputs some input:

func main() {
    print("What is your name: ", terminator: "")
    let name = readLine()
    print("Hi, \(name ?? "Unknown")")
}

main()

Going to work on getting this to function via Process as its a simpler use case and hopefully solving this can solve the readpassphrase issue. As to what I have so far, I have a Process setup:

let process = Process()
process.executableURL = executablePath.asURL // path to my basic name collection program
process.arguments = arguments.values
process.run()

When launching this process I get:

clide (remotes/origin/luis/foundation-process) > swift run clide
[3/3] Build complete!
What is your name:

Which is great! However, when I input something and press enter, the program does nothing. It just adds a new line in my terminal and continues executing with no additional output. I'm really not sure why this is the case, to me it feels like all this complicated Unix stuff is what a higher-level API like Process is supposed to handle so it's a little confusing when this just doesn't work?

Reading the docs for Process seem to indicate it should just setup stdin to be whatever the parent process is:

If this method isn’t used, the standard input is inherited from the process that created the receiver. This method raises an NSInvalidArgumentException if the receiver has already been launched.

But when actually running the program it doesn't seem like this is the case...

Looking at other threads (like this one). Other developers seem to be running into the same issues here. I'm genuinely stuck here and having spent the last week reading up on this I'm not sure I'm any closer to solving this issue in Swift. It really feels like this should be easier to accomplish, but maybe I'm missing something.

from the compiler error I got, it seems fork is not supported in Swift.

Correct. That’s because it’s not safe to use fork from Swift [1]. However, most of this stuff can be done via posix_spawn, or Process, both of which you can use from Swift.

However, when I input something and press enter, the program does nothing.

Are you running this in Terminal? Or in Xcode?

I'm genuinely stuck here

Yeah, I understand your frustration. Unfortunately there’s a limit to the amount of time I can spend helping you out here on DevForums. I think it’d be best for you to open a DTS tech support incident so I can allocate more time to this.

If you do this, please reference this thread for context.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] The child side of a fork runs in a very constrained environment, one that’s compatible with the Swift runtime. It’s not compatible with the Objective-C runtime either, but Objective-C lets you call fork and assumes that you’ll follow the rules on the child side (that is, restrict yourself to a very limited set of C).

@eskimo,

Thanks for the DTS suggestion. Just used one up to get support for this topic. Will reply back if we come to a resolution, thanks again!

Earlier I wrote:

The child side of a fork runs in a very constrained environment, one that’s compatible with the Swift runtime.

Someone asked me about this in a separate context, but I decided to reply here so that others could benefit.

The theoretical problem here is that:

• Swift code depends on the Swift runtime.

• That depends on malloc.

• Calling malloc between fork and exec* is problematic if the process has more than one thread.

Specifically, imagine this sequence:

1. You have a process with two threads, A and B.

2. Thread A calls malloc, which locks a mutex.

3. While malloc is working, thread B calls fork.

4. In the child process, only thread B exists. It calls malloc.

This deadlocks because the child process has inherited the memory state of the parent, but not the threads of the parent. So the mutex is locked in memory but there's nothing to unlock it.

Now, in theory malloc can solve this problem with a pthread_atfork. However, that doesn’t really solve the general problem:

• pthread_atfork is notoriously tricky.

• If you’re writing portably code, you now depend on all implementations of malloc doing this.

• malloc isn’t the only problem here. You might, for example, find that other mutexes in the Swift runtime cause you grief. I haven’t checked on this myself, but I have seen exactly this with Objective-C runtime.

The other way around this is to write your Swift code such that it doesn’t call malloc, or do anything else that might involve locking a mutex. Unfortunately there’s no official way to do that [1].

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Although there are certainly efforts in this space, including the unsupported @_noLocks attribute and the Embedded Swift effort.