NetworkExtension breaks Safari

Safari throwing these errors due to an altered origin IP does make sense if the client application in Safari has configured their site in this way. A few questions on how your remoteEndpoint is delivered:

• Do you have anything in place that would alter the remoteEndpoint delivered by the system for NEAppProxyFlow? For example a system level proxy or a PAC proxy for certain endpoints that would have already altered the remoteEndpoint?

• These are flows that you are instructing your proxy to handle, correct?

• Do you have any other filtering rules in place that would cause the remoteEndpoint to show up as 0.0.0.0?

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Safari throwing these errors due to an altered origin IP does make sense if the client application in Safari has configured their site in this way.

The problem is that when an extension processes Safari’s flows, IP address does not appear anywhere in NEAppProxyFlow API at all.
From remoteEndpoint docs:
https://developer.apple.com/documentation/networkextension/neappproxytcpflow/1406233-remoteendpoint

If the flow’s corresponding socket was created using one of the high-level networking APIs such as URLSession or NSURLConnection, then the hostname property of the remoteEndpoint object contains the DNS name of the remote host. If the flow’s corresponding socket was created using the sockets API directly, then the hostname property of the remoteEndpoint object contains the IP address of the remote host.

So, in our case it is DNS name only, there is no IP address and we cannot set or alter it in the extension.

As for your questions,

Do you have anything in place that would alter the remoteEndpoint delivered by the system for NEAppProxyFlow?  For example a system level proxy or a PAC proxy for certain endpoints that would have already altered the remoteEndpoint?

No, we test on a clean system.

These are flows that you are instructing your proxy to handle, correct?

For the test scenario, the extension processes all TCP flows on the system and just "forwards" them to the original destinations. It acts like a transparent proxy and there is no an actual proxy sever involved.

Do you have any other filtering rules in place that would cause the remoteEndpoint to show up as 0.0.0.0?

No.

I went in and captured a few NEAppProxyTCPFlow objects as a baseline to sort out what is happening here. Here are the Safari and regular CURL flows I captured.

Code Block text
/* Safari */
2020-09-02 07:55:40.794496-0700 0x142d18   Debug  0x0  40214  0 [com.example.apple-samplecode.TransparentProxyTestBed.TransparentProxy:provider] provider will handle new NEAppProxyTCPFlow, flow: TCP com.apple.Safari[{length = 20, bytes = 0x...}] remote: www.apple.com:443
2020-09-02 07:55:24.245073-0700 0x142d19   Debug  0x0  40214  0 [com.example.apple-samplecode.TransparentProxyTestBed.TransparentProxy:provider] provider will handle new flow, flow: TCP com.apple.Safari[{length = 20, bytes = 0x...}] remote: developer.apple.com:443
/* CURL */
2020-09-02 07:55:31.345675-0700 0x142d19   Debug  0x0 40214  0  [com.example.apple-samplecode.TransparentProxyTestBed.TransparentProxy:provider] provider will handle new NEAppProxyTCPFlow, flow: TCP com.apple.curl[{length = 20, bytes = 0x...}] remote: x.x.x.x:443

Addressing your concerns:

The problem is that when an extension processes Safari’s flows, IP address does not appear anywhere in NEAppProxyFlow API at all.

Right, in this case it's handing a NWHostEndpoint endpoint for NWConnection to use when setting up the remote side of the connection.

So, in our case it is DNS name only, there is no IP address and we cannot set or alter it in the extension.

This goes along with my point above about using the NWHostEndpoint in the remote side of the connection. Are you unable to use this here?

If you use the hostname provided by NEAppProxyTCPFlow do you still receive these errors in Safari:

Code Block text
[Error] Origin <URL> is not allowed by Access-Control-Allow-Origin.
[Error] XMLHttpRequest cannot load <URL> due to access control checks.

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Thanks for your reply.

This goes along with my point above about using the NWHostEndpoint in the remote side of the connection. Are you unable to use this here?

We use NWHostEndpoint to connect to the remote site without a problem (socket API: gethostbyname and connect). The problem is that this breaks Safari somehow.

If you use the hostname provided by NEAppProxyTCPFlow do you still receive these errors in Safari:

Exactly. That's the whole point!

Can you please try to make your TransparentProxy sample to capture all flows from Safari and open a YouTube video? Safari console should be full of such errors.

By the way, is there a publicly available sample of NEAppProxyProvider? We were unable to find anything.

By the way, is there a publicly available sample of NEAppProxyProvider? We were unable to find anything.

No, there is publicly available sample for this. I used one of my test bed projects for running the logs posted in the previous thread.

Can you please try to make your TransparentProxy sample to capture all flows from Safari and open a YouTube video? Safari console should be full of such errors.

For further analysis and investigation I would recommend opening a TSI so I can dedicate some time in looking deeper into this.


Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Done!
TSI Follow-up: 745748241

Hello,

We are having the exact same issue.

When filtering all TCP connections most of the sites seem to work fine, but YouTube's videoplayback requests fail because of Access-Control-Allow-Origin and the rest of the page is fine (the other connections seem to work fine). The same thing also happens on Twitch when playing a video / live stream.

Our test application is injecting back all the data without modifying it. The problem only occurs with Safari when the extension is running. The issue does not appear on Chorme and Firefox with the extension running.

From what we tested this problem occurs only when we filter all TCP connections. When we tried filtering only the ports used by the most common protocols, including 443, the videos seem to play fine.

The way we filter all connections is by calling setTunnelNetworkSettings:completionHandler: with this settings:

Code Block objective-c
NETransparentProxyNetworkSettings* settings = [[NETransparentProxyNetworkSettings alloc] initWithTunnelRemoteAddress:@"127.0.0.1"];
NENetworkRule* rule = [[NENetworkRule alloc] initWithRemoteNetwork:nil remotePrefix:0 localNetwork:nil localPrefix:0 protocol:NENetworkRuleProtocolTCP direction:NETrafficDirectionOutbound];
settings.includedNetworkRules = @[rule];

and we create the connection using createTCPConnectionToEndpoint:enableTLS:TLSParameters:delegate: with the NWEndpoint received in handleNewFlow.

In my research it has been concluded that this is a known bug with the way a Transparent Proxy or App Proxy Provider processes cross origin requests in Safari for web sites that load video content with XHR. Specifically, any video web application that loads pieces of video content from a cross origin XHR request while an App Proxy Provider is installed with a NENetworkRule that looks like this will be affected:

Code Block swift
settings.includedNetworkRules = [
    NENetworkRule(remoteNetwork: nil, remotePrefix: 0, localNetwork: nil, localPrefix: 0, protocol: .TCP, direction: .outbound)
]

The issue here is that the host based flows that are passing through the proxy will not pass through the proxy in Safari for cross origin XHR video loading. Same site XHR request seem to work fine.

Using a filter like the follow will capture cross origin XHR requests via IP and port and allow the video sections to be loaded as normal:

Code Block swift
settings.includedNetworkRules = [
    NENetworkRule(remoteNetwork: NWHostEndpoint(hostname: "0.0.0.0", port: "443"),
                  remotePrefix: 0,
                  localNetwork: nil,
                  localPrefix: 0,
                  protocol:.TCP,
                  direction: .outbound)
]

Also note that if a video web application uses the HTML <video> tag, this will not be affected and the video will load as normal.

This is an issue that we are aware of, but I do not have a timeline for a fix.


Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Please retry this with the update to macOS Big Sur Beta 10.


Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Please retry this with the update to macOS Big Sur Beta 10.

I confirm the problem is solved now!
Thanks!

Hi, for me issue is resolved if I include only IPv4 traffic in transparent proxy. If I add IPv6 rule, issue is still present. For example:

let remoteIPv6Network = NWHostEndpoint(hostname: "2001:4860:4860::8888", port: "0")
     let networkIPv6Rule = NENetworkRule(remoteNetwork: remoteIPv6Network,
                                           remotePrefix: 0,
                                            localNetwork: nil,
                                            localPrefix: 0,
                                            protocol: .any,
                                            direction: .outbound)

Is there a way to fix this?