Interaction between iPhone and Mac/PC via USB - How?

There isn’t a good way to do what you want here:

• There are no low-level USB APIs on iOS [1].

• MFi won’t help because you can’t turn the Mac into an MFi accessory.

• If you search the ’net you’ll find lots of references to usbmuxd. This technique is not supported and I strongly recommend that you avoid it.

IMO the best way forward here is to do this over the network, but that does require you to be running code on both ends.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

[1] You wrote:

It appears that developers are not able to send raw USB
protocol/messages/packets unless the developer is a part of MFI, is
that correct?

It’s correct in summary but wrong in detail. If you create an MFi accessory then your app talks to it via the ExternalAccessory framework. That does not give you access to “raw USB”.

Thank's eskimo

So for the apps that do achieve this [1][2], are you implying they are almost certainly using usbmuxd?

If this method isn't supported, how does it get past app store review?

Also, on the chance that part of my post hasn't come through clearly: I would be Ok with the connection between the iPhone and Mac wasn't to use raw/native USB - but the preference is that it happens over a cabled connection..

I'm interested in:

• Ability to send video and audio from the iPhone to the Mac over a cabled connection

• Ability to send commands from the Mac to the iPhone that the app will use to make adjustments to the video/audio stream

If it's not possible over a raw USB protocol/mechanism - is there any other mechanism that would allow this over a cabled connection? Preference here is USB (utilising the users existing iPhone cables) but if that's not possible would any other types of cables allow something like this?

The idea is to use the iPhone camera for live streaming, and allowing the user to adjust settings (quality, zoom, etc) from their computer without needing to get up and touch the iPhone.. With a user likely streaming via their network (WiFi), I'd prefer to not send the video/audio stream over WiFi

[1] https://apps.apple.com/us/app/id1352834008
[2] https://apps.apple.com/us/app/id1524933155

So for the apps that do achieve this … are you implying they are
almost certainly using usbmuxd?

I’ve not looked at these app and so can’t comment.

If this method isn't supported, how does it get past app store review?

I also can’t comment about App Review policies.

Ability to send video and audio from the iPhone to the Mac over a
cabled connection

If cables are the sticking point here, how about Ethernet?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

Would you be comfortable giving a rough estimate based on just what you know about them (even if that is simply seeing their App Store page and reading this thread) combined with your knowledge on iOS development technologies and what you know?

Alternatively, would you be comfortable giving a rough estimate about how likely any app would be using usbmuxd if that app was able to communicate with a companion Mac app via USB (ie in a general sense - not in reference to any particular app)

eg. very likely, likely, maybe, not sure, unlikely, very unlikely?

I'm not looking for anything that's a sure thing, but if you feel you're comfortable to give a rough estimate, and you were also ok to give one, please do. I understand if you'd prefer not to though.

Based on your posts in this thread, it seems like there really isn't another way to do this other than usbmuxd. So it seems either likely or very likely that's what they are using?

I've only used one of the apps and it only works when you open the companion app on Mac (and PC).

Using an Ethernet cable could be a way forward - yep.. Using an Ethernet cable with an iPhone is just like plugging it into your Mac - as in, there is nothing different an application would need to (just deal with sending network packets) - the OS handles everything else?

Or, is there any way a user would be able to tell? ie any commands I can run on my Mac to output some info that might tell me, or anything I can look for in the logs? I know how to increase the logging for each of the 'subsystems' (is that the right term?) - if the logs that might show this info are not output by default..

Using an Ethernet cable with an iPhone is just like plugging it into
your Mac - as in, there is nothing different an application would need
to (just deal with sending network packets) - the OS handles
everything else?

Pretty much. You need one of these and one of these (or a third-party equivalent) and a power supply.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

This is old but let me fill the gap for the benefit of future readers with the same question.

1. the other products definitely use usbmuxd, there's no other way.
2. connecting iPhone to ethernet via a powered adaptor or two looks very odd and not ergonomic for end users.
3. the cables themselves are not quite ergonomic either, if you can use wifi - go for it.
4. cables still have their use, I wish Apple to introduce a supported API or enhance one of existing API's (e.g. Network framework) to support connectivity over USB.

I wish Apple to introduce a supported API or enhance one of existing API's (e.g. Network framework) to support connectivity over USB.

AFAIK this just works on recent systems.

I tried this out today here in my office:

1. Using Xcode 14.3 on macOS 13.2.1 with an iPhone running iOS 16.4.1 connected via USB…

2. On the iPhone, in Settings, turn off Wi-Fi, Mobile Data, and Bluetooth [1].

3. In Xcode, create a new test project with the code shown below and wire startStop() up to a button.

4. Add the following to the Info.plist to allow it to use Bonjour:

   <key>NSBonjourServices</key>
   <array>
   	<string>_ssh._tcp</string>
   </array>
   

5. Run it on the iPhone.

6. On the iPhone, tap the button to start the server.

7. On the Mac, in Terminal, choose Shell > New Remote Connection.

8. The iOS device’s SSH ‘server’ shows up in the list. Select it and click Connect.

9. In Xcode, you’ll see the iOS app log the following:

   listener will start
   listener did change state, new: ready
   listener did receive connection, from: Optional(fe80::7c41:91ff:fe4f:8e95%anpi0.52866)
   

In short, the Mac was able to connect to a Bonjour service running on the iPhone, even though the iPhone has all network interfaces disabled.

IMPORTANT It’s possible that this only works because of developer bits (the Mac having Xcode installed or the iPhone being in Developer Mode). I don’t have time to test this with a non-developer configuration. I encourage you try that for yourself.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

---

[1] We disabled TCP/IP over Bluetooth many moons ago, so I’m doing this just for good measure, not because I think it’ll actually change things.

var listenerQ: NWListener? = nil

func start() -> NWListener {
    print("listener will start")
    let listener = try! NWListener(using: .tcp)
    listener.service = .init(type: "_ssh._tcp")
    listener.stateUpdateHandler = { newState in
        print("listener did change state, new: \(newState)")
    }
    listener.newConnectionHandler = { connection in
        let remotePeer = connection.currentPath?.remoteEndpoint
        print("listener did receive connection, from: \(remotePeer)")
        connection.cancel()
    }
    listener.start(queue: .main)
    return listener
}

func stop(listener: NWListener) {
    print("listener will stop")
    listener.stateUpdateHandler = nil
    listener.cancel()
}

func startStop() {
    if let listener = self.listenerQ {
        self.listenerQ = nil
        self.stop(listener: listener)
    } else {
        self.listenerQ = self.start()
    }
}

Nice. And if you don't want using ssh from Mac for some reason (e.g. you are paranoid and think that it might work over ssh as part of Apple magic but that doesn't mean it will work with other protocols) you can do the same for the blower part and launch it on mac:

class Model: ObservableObject {
    @Published var text: String = ""
    private var listenerQ: NWListener?
    private var browserQ: NWBrowser?

    func log(_ string: String) {
        print(string)
        text.append(string + "\n")
    }

    func startBrowser() -> NWBrowser {
        log("browser will start")
        let browser = NWBrowser(for: .bonjour(type: "_ssh._tcp", domain: nil), using: .tcp)
        browser.stateUpdateHandler = { newState in
            self.log("brower did change state, new: \(newState)")
        }
        browser.browseResultsChangedHandler = { results, changes in
            self.log("brower did change results, new: \(results) \(changes)")
        }
        browser.start(queue: .main)
        return browser
    }
    
    func stopBrowser(browser: NWBrowser) {
        log("browser will stop")
        browser.stateUpdateHandler = nil
        browser.cancel()
    }

    func startStopBrowser() {
        if let browser = browserQ {
            browserQ = nil
            stopBrowser(browser: browser)
        } else {
            browserQ = startBrowser()
        }
    }

    var isBrowserStarted: Bool {
        browserQ != nil
    }

    var isListenerStarted: Bool {
        listenerQ != nil
    }   ...
}

Change the "_ssh._tcp" appropriately and don't forget to change the corresponding string in "Bonjour services" array in the plist of your iOS app.

I also used log instead and in addition to bare print so iPhone app can be self contained and show the log without Xcode:

struct ContentView: View {
    @StateObject private var model = Model()
    
    var body: some View {
        VStack(alignment: .leading) {
            Button(model.isListenerStarted ? "Stop Listener" : "Start Listener") {
                model.startStopListener()
            }.padding()
            Button(model.isBrowserStarted ? "Stop Browser" : "Start Browser") {
                model.startStopBrowser()
            }.padding()
            HStack {
                Text(model.text)
                    .font(.caption2.monospaced())
                Spacer()
            }
            Spacer()
        }
        .padding()
    }
}

IMPORTANT It’s possible that this only works because of developer bits (the Mac having Xcode installed or the iPhone being in Developer Mode). I don’t have time to test this with a non-developer configuration. I encourage you try that for yourself.

This is very important indeed unless of course you want to make an app that works just for you or for people who wouldn't mind installing relevant components in case they are important. Just to check if it works in clean environment is somewhat a challenge, you'd either need to enable developer mode on the phone, install the app via Xcode and then disable developer mode, making sure it cleanly disabled all dev bits and pieces, or you'd need to make a TestFlight version of this tester app. Also make sure you do the actual test on the Mac that didn't have Xcode installed.

The open questions at this point:

• does this work in "clean environment" (without developer bits on Mac and/or iPhone)? To be tested.
• assuming it does (or it is not important to have environment clean), does it still work over USB if I leave wifi / bluetooth enabled? (This feature might unwanted, e.g. for security reasons. Or, equally, it could be desired).
• assuming working over wifi/bluetooth is undesired for some reason (e.g. security), could we make it stop working when wifi/BT are enabled and I unplugged the USB cable?

(in other words set an option to disable all wireless connectivity similar to how it is possible disabling cellular connectivity in URLSession).

I have an error when I try to do this. My code is based on eskimo's. My version of it is:

import Foundation
import Network

class SSH_Session: ObservableObject{
    
    public private(set) var listener: NWListener? = nil    
    public private(set) var ButtonText: String = ""
    @Published public private(set) var isListening: Bool = false
    
    init(){
        setButtonText( false )
    }
    
    public func start(){
        
        print("listener will start")
               
       listener = (try? NWListener(using: .tcp) ) ?? nil
        if (listener == nil) {
            print( "Failed listener acquisition." )
            setButtonText( false )
        }else {

            listener?.stateUpdateHandler = { newState in
                print("listener did change state, new: \(newState)", "" )
                if( (self.listener?.debugDescription != nil) && (self.listener?.debugDescription != "") ) {
                    print( " " + self.listener!.debugDescription )
                }else{
                    print( "\n" )
                }
            }
            
            listener?.service = .init(type: "_ssh._tcp")
            
            listener?.newConnectionHandler = { connection in
                let remotePeer = connection.currentPath?.remoteEndpoint
                print("listener rejected a receive connection, from: \(remotePeer as Optional)")
                connection.cancel()
            }
            
            listener?.start(queue: .main)
            setButtonText( true )
        }
        return
    }
    
    public func stop() {
        print("listener will stop")
        self.listener?.stateUpdateHandler = nil
        self.listener?.cancel()
        setButtonText( false )
     }
    
    public func StartStop() {
        if( self.isListening ) {
            self.stop()
        } else {
            self.start()
        }
    }
    
    private func setButtonText( _ isListening: Bool )
    {   self.isListening = isListening
        ButtonText = isListening ? "Close SSH Connection" : "Open SSH Connection"
    }
}

In my version of stateUpdateHandler I included a verbose diagnostic.

Xcode is installed in the iPhone 14, and it is in developer mode. I turned off Bluetooth, WiFi, and cell service even though it is not connected to a cell service.

Info.plist is not automatically added to a project since Xcode 13. I am using Xcode 14.3. When the project was created I did not know to add one. So I used the properties GUI. In the "Build Settings" tab I added the key value pair shown in this screenshot:

The screenshot includes the error I got. I copy and paste its text here:

listener will start
listener did change state, new: ready 
 [L1 failed, local endpoint: <NULL>, parameters: tcp, local: ::.49728, definite, attribution: developer, server, port: 49728, path satisfied (Path is satisfied), interface: en0[802.11], ipv4, dns, service: <NULL>._ssh._tcp.<NULL> txtLength:0]
listener did change state, new: failed(-65555: NoAuth) 
 [L1 failed, local endpoint: <NULL>, parameters: tcp, local: ::.49728, definite, attribution: developer, server, port: 49728, path satisfied (Path is satisfied), interface: en0[802.11], ipv4, dns, service: <NULL>._ssh._tcp.<NULL> txtLength:0]

The terminal program in the MacStudio does not show any SSH servers are available iPhone. The iPhone's SSH service appears not to be advertised.

I do not know what to do about this error. Any suggestions would be appreciated.

Notice the -65555 in that log message. That translates to kDNSServiceErr_NoAuth, indicating that Network framework failed to advertise the listener with Bonjour because it wasn’t authorised to do so.

Setting NSBonjourServices as a custom build setting won’t include it in your Info.plist. Rather, you must set it in the Info tab of the target editor. When you go to add a key, choose Bonjour services.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

The delete key removed the Terminal's "New Remote Connection" list in the Build Settings tab.

I added the Info.plist file by adding a key value pair I did find a way to add. This was in "URL Types". As soon as I added this an Info.plist file appeared in the project. I then removed the URL type I added. I then edited the newly created Info.plist file to add the key, value, pair: "Bonjour services", "_ssh._tcp" to it. This worked in the iPhone 14 simulator running in the MacStudio. The MacStudio did show up in Terminal's "New Remote Connection" list.

But when the program was downloaded into a real iPhone 14 over a lightning cable I still got the 65555 NoAuth error, and the real iPhone 14 did not appear in the Terminal's "New Remote Connection" list. It is as if the key, value, pair: "Bonjour services", "_ssh._tcp" in the Info.plist file did not make it into the real iPhone 14. This screen shot shows the setup:

I changed the listener initialization line to:

listener?.service = .init(name: "iPhone1", type: "_ssh._tcp", domain: "BRDF" )

That got rid of of the NoAuth error. The message I get now is:

listener will start
listener did change state, new: ready 
 [L1 ready, local endpoint: <NULL>, parameters: tcp, local: ::.62862, definite, attribution: developer, server, port: 62862, path satisfied (Path is satisfied), interface: en0[802.11], ipv4, dns, service: iPhone1._ssh._tcp.BRDF txtLength:0]

But advertisement still fails. It does not show up as a server in the Terminal program. The message shows local endpoint is still: <NULL>. Could that have something to do with it?

Is there a tutorial somewhere about how to do this by using Bluetooth instead of the Lighting Cable USB? Plan B is to use Bluetooth.

For future reference, the general solution above also applies to two iOS/iPadOS devices, even if they both have Lightning ports (and you don't even need to supply power to the Lightning-to-USB adaptor!)

https://developer.apple.com/forums/thread/744559

Anyone have any hints on how to get a Windows PC to connect to a Bonjour listener/server?