I was able to figure this out. It turn out that posix_openpt() is broken on macos, the file descriptor it returns isn't useful for much. If you use openpty(), that file descriptor works ok. So it appears to be a bug in macos with posix_openpt().

Thanks a bunch for looking at this. It's the client open that's needed before anything will work, just the grantpt() and unlockpt() is not enough. POSIX is kind of vague, but I would interpret it as saying that whatever is returned from posix_openpt() is a normal file descriptor that can be used for normal purposes. But it's on the margins. I would say it's a bug, but that's me :). The way it works in MacOS is kind of unfortunate, really. There are reasons to create a pty that don't require the creating program to open the client. There are programs that create a pty so so that other programs that can only talk to serial ports can be used without a serial port, for instance. But at least I (and the rest of the world) know now. -corey