AppProxy breaks existing connections

Is it expected behavior?

This is odd behavior and I would have not expected this. Do you have any more information to share on this issue such as your NENetworkRules, any logs that you are seeing in your flow provider, and do you have the flow copying process in place to pickup and handle these connections?


Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

For example, the rule can be like this:

Code Block language
NENetworkRule(remoteNetwork: nil, remotePrefix: 0, localNetwork: nil, localPrefix: 0, protocol: .TCP, direction: NETrafficDirection.outbound)

It's very easy to reproduce. Just open a persistent TCP connection like SSH. Run a
simple transparent proxy. Observe that SSH connection is broken.

do you have the flow copying process in place to pickup and handle these connections?

Yes, but this logic applies to the new connection only. Correct? How is it possible to do this for
the existing ones?

For SSH try using a NENetworkRule like this:

Code Block swift
settings.includedNetworkRules = [
    NENetworkRule(remoteNetwork: NWHostEndpoint(hostname: "0.0.0.0", port: "22"),
                  remotePrefix: 0,
                  localNetwork: nil,
                  localPrefix: 0,
                  protocol:.TCP,
                  direction: .outbound)
]

I was able to turn on a Transparent Proxy and make a SSH connection to a remote test machine.

Code Block text
% ssh user@xx.x.x.155
...
Welcome to Ubuntu 18.04.3 LTS ...
  System information as of Fri Sep 18 13:58:57 UTC 2020
  System load:  0.0                Processes:           90
  Memory usage: 20%                IP address for eth0: x.x.x.155
  Swap usage:   0%

On the Transparent Proxy log:

Code Block 
2020-09-18 06:58:47.997426-0700 provider will handle new flow, flow: 0x7f8d1d4151d0 TCP com.apple.openssh[{length = 20, bytes = x}] remote: x.x.x.155:22
2020-09-18 06:58:48.002330-0700 new flow with bundle id com.apple.openssh
2020-09-18 06:58:48.002391-0700 ### (TCP) ### provider will handle new NEAppProxyTCPFlow, flow: 0x7f8d1d4151d0 - TCP com.apple.openssh[{length = 20, bytes = x}] remote: x.x.x.155:22
2020-09-18 06:58:48.002444-0700 provider core will start new TCP flow, flow: 0x7f8d1d4151d0
2020-09-18 06:58:48.002513-0700 ##### Copier - Ready 4 #####

Yes, but this logic applies to the new connection only. Correct? How is it possible to do this for the existing ones?

Yes, existing connection may see interruptions and may have to be restarted. Is that what you are seeing too?

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Yes, existing connection may see interruptions and may
have to be restarted. Is that what you are seeing too?

Yes, this is what we see and this is a concern. So for example, if you establish SSH connection before AppProxy is activated, this existing SSH connection gets disconnected. Is this expected?
If so, this is not optimal.

I was also able to reproduce an unresponsive SSH connection while starting a system Transparent Proxy in the middle of my SSH session. Note that this is specific to the NENetworkRules matching the SSH session. It looks like you are running into this situation because your system starts one flow responsbile for handling the SSH session pre-proxy and then when the Transparent Proxy is started you have introduced a completely new flow handling strategy for handling your SSH session. So you can imagine where this leads to an interruption because a new local and remote connection is now opened to handle this TCP flow for your SSH session. This can be resolved by restarting the SSH session after the system Transparent Proxy starts.


Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

OK, so this is the expected behavior. That's bad because there are a lot of scenarios when connections should stay alive when AppProxy starts. I used SSH just as an example.

Is it possible to submit this as a feature request to Apple?