Which NetworkExtension API to use

App & System Services Core OS
Optaget OP
Created Jan ’21
Replies 5
Boosts 0
Views 1.3k
Participants 3
I am currently examining which of the available NetworkExtensions would be most suitable for a project I am working on.

I would like something like the folllowing:

  • Intercept and modify all DNS traffic (port 53)

  • Intercept and modify TCP and UDP flows going to a specific subnet.

Further, I need to intercept traffic from all applications, not just a single app.

An NEAppProxyProvider seems like an obvious choice since you can setup various rules for what you want to intercept. For example that you want to intercept traffic going to specific subnets and everything going to port 53. Also, it intercepts TCP flows rather than raw IP packets which is simpler. However:
If I cannot use an NEAppProxyProvider to intercept traffic from all applications, I guess my only option is to create a PacketTunnel and then deal with the raw IP packets? And in this case I can only use destination IP subnets to specify which packets I get, right? So in particular:

  • I cannot intercept all traffic to port 53 unless I intercept all subnets.

  • If there is some traffic to the intercepted destination IP subnets which I don’t want to intercept anyway, I will have create my own sockets and send the data from the raw TCP and UDP packets on them (which may be challenging as the local ports and source IPs may change when doing this)

I guess a partial solution could be to make both a PacketTunnel and a DNSProxy. The user would in this case have to accept one prompt for allowing the DNSProxy and another prompt for allowing the PacketTunnel to run.
Answered by Systems Engineer in 657925022
There is a lot to unpack here. Let me start but trying to address some of the questions that were presented here:

There is a (to me) somewhat mysterious class NETransparentProxyManager (https://developer.apple.com/documentation/networkextension/netransparentproxymanager ). It is not clear to me how NETransparentProxyManager is different from NEAppProxyProviderManager. But I assume that NETransparentProxyManager can also only be used to intercept traffic per-app?

One of the differences between NEAppProxyProviderManager and NETransparentProxyManager is that NEAppProxyProviderManager can be used with NEAppRule's to configure how the proxy is enabled and which app's activate the network configuration. NETransparentProxyManager does not have this capability as it's intention is to be more general purpose.

One thing that puzzles me is that if NEAppProxyProviderManager and NETransparentProxyManager can only be used to intercept traffic on a per App basis, then why does NEAppProxyProvider (https://developer.apple.com/documentation/networkextension/neappproxyprovider ) have a routingMethod member (https://developer.apple.com/documentation/networkextension/netunnelprovider/1405979-routingmethod )? Shouldn’t this value always be ‘sourceApplication’? Perhaps it is because it is in the super class NETunnelProvider which is shared with NEPacketTunnelProvider.

NETransparentProxyManager should be able to claim traffic for many different applications on the system using NENetworkRule's with NETransparentProxyNetworkSettings. As to why NEAppProxyProvider has a property of routingMethod, yes, you are correct that this is because this class shares a base class of NETunnelProvider, just like NEPacketTunnelProvider does.

If I cannot use an NEAppProxyProvider to intercept traffic from all applications, I guess my only option is to create a PacketTunnel and then deal with the raw IP packets?

First, as I always do, I recommend that you do not use a packet tunnel as a means to capture and re-route all DNS traffic. If you find yourself in a situation where you are trying to capture a large amount of domains in your match domains list in your packet tunnel NEDNSSettings then there are mechanisms for doing this on macOS and iOS, such as NEDNSProxyProvider. (Note that for iOS NEDNSProxyProvider requires MDM) Packet Tunnels were not built as a means to claim and reroute all DNS traffic and you will most likely run into endless edge cases trying to get this right.

If you are not constrained by version of macOS you want to support then I would recommend using NETransparentProxyProvider to fulfill both of your TCP and UDP needs stated at the top of your post. If you need to support TCP / UDP flows earlier than macOS 11 then you can still use NEAppProxyProvider / NETransparentProxyManager but you will need to proxy all of your flows and will not have the ability to return false and let the OS handle the flow you do not wish to proxy.



Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com
Replies  5
Boosts  0
Views  1.3k
Participants  3
Systems Engineer OP
Apple
Jan ’21
Accepted Answer
There is a lot to unpack here. Let me start but trying to address some of the questions that were presented here:

There is a (to me) somewhat mysterious class NETransparentProxyManager (https://developer.apple.com/documentation/networkextension/netransparentproxymanager ). It is not clear to me how NETransparentProxyManager is different from NEAppProxyProviderManager. But I assume that NETransparentProxyManager can also only be used to intercept traffic per-app?

One of the differences between NEAppProxyProviderManager and NETransparentProxyManager is that NEAppProxyProviderManager can be used with NEAppRule's to configure how the proxy is enabled and which app's activate the network configuration. NETransparentProxyManager does not have this capability as it's intention is to be more general purpose.

One thing that puzzles me is that if NEAppProxyProviderManager and NETransparentProxyManager can only be used to intercept traffic on a per App basis, then why does NEAppProxyProvider (https://developer.apple.com/documentation/networkextension/neappproxyprovider ) have a routingMethod member (https://developer.apple.com/documentation/networkextension/netunnelprovider/1405979-routingmethod )? Shouldn’t this value always be ‘sourceApplication’? Perhaps it is because it is in the super class NETunnelProvider which is shared with NEPacketTunnelProvider.

NETransparentProxyManager should be able to claim traffic for many different applications on the system using NENetworkRule's with NETransparentProxyNetworkSettings. As to why NEAppProxyProvider has a property of routingMethod, yes, you are correct that this is because this class shares a base class of NETunnelProvider, just like NEPacketTunnelProvider does.

If I cannot use an NEAppProxyProvider to intercept traffic from all applications, I guess my only option is to create a PacketTunnel and then deal with the raw IP packets?

First, as I always do, I recommend that you do not use a packet tunnel as a means to capture and re-route all DNS traffic. If you find yourself in a situation where you are trying to capture a large amount of domains in your match domains list in your packet tunnel NEDNSSettings then there are mechanisms for doing this on macOS and iOS, such as NEDNSProxyProvider. (Note that for iOS NEDNSProxyProvider requires MDM) Packet Tunnels were not built as a means to claim and reroute all DNS traffic and you will most likely run into endless edge cases trying to get this right.

If you are not constrained by version of macOS you want to support then I would recommend using NETransparentProxyProvider to fulfill both of your TCP and UDP needs stated at the top of your post. If you need to support TCP / UDP flows earlier than macOS 11 then you can still use NEAppProxyProvider / NETransparentProxyManager but you will need to proxy all of your flows and will not have the ability to return false and let the OS handle the flow you do not wish to proxy.



Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com
1
Optaget OP
Jan ’21
Thank you very much for your quick and thorough answer to my long post! It was very helpful: What I did not realise first when reading the API documentation is that if you use for example an NETransparentProxyManager, then you need to use NETransparentProxyNetworkSettings when calling setTunnelNetworkSettings in the provider. I figured this out from your reply, so I have now successfully written an AppProxyProvider that intercepts traffic based on NENetworkRules by using an NETransparentProxyManager. I was able to intercept both DNS traffic and also traffic to a subnet as I wanted. And it did seem to capture traffic from all apps.

I do still have two follow up questions 

First question:

You write that:

“… you can still use NEAppProxyProvider / NETransparentProxyManager but you will need to proxy all of your flows and will not have the ability to return false and let the OS handle the flow you do not wish to proxy.”

Is that the different between using NEAppProxyProvider and NETransparentProxyProvider? That false returns from handleNewFlow means “block” for NEAppProxyProvider and “pass” for NETransparentProxyProvider? I cannot see any differences in the API documentation. I cannot test this as I currently only have access to MacOS 10.15. The target MacOS version for the project I work on is still up for discussion.

Second question:

What is the difference between using NEDNSProxyManager and NETransparentProxyManager with a rule of:

NENetworkRule(destinationHost: NWHostEndpoint(hostname: "", port: "53")

It seems to me like I get significantly more stuff captured with the NETransparentProxyManager approach. In particular a lot of requests going to an address on 10.0.0.0/8 which is not an address on my network and also to fe80::1. So maybe NETransparentProxyManager captures more DNS traffic than NEDNSProxyManager?
0
Systems Engineer OP
Apple
Jan ’21

Thank you very much for your quick and thorough answer to my long post! It was very helpful: What I did not realise first when reading the API documentation is that if you use for example an NETransparentProxyManager, then you need to use NETransparentProxyNetworkSettings when calling setTunnelNetworkSettings in the provider. I figured this out from your reply, so I have now successfully written an AppProxyProvider that intercepts traffic based on NENetworkRules by using an NETransparentProxyManager. I was able to intercept both DNS traffic and also traffic to a subnet as I wanted. And it did seem to capture traffic from all apps.

That is great news!

Regarding:

Is that the different between using NEAppProxyProvider and NETransparentProxyProvider? That false returns from handleNewFlow means “block” for NEAppProxyProvider and “pass” for NETransparentProxyProvider? I cannot see any differences in the API documentation. I cannot test this as I currently only have access to MacOS 10.15. The target MacOS version for the project I work on is still up for discussion.

Yes. For NEAppProxyProvider on macOS 10.15, if you return false from handleNewFlow the flow will be discarded by the OS because it is in a closed state. On macOS 11.0, for NETransparentProxyProvider if you return false then the OS will handle the flow for you just like the proxy was not involved. It's on my list of things to do get that documented.

Regarding:

What is the difference between using NEDNSProxyManager and NETransparentProxyManager with a rule of:

NENetworkRule(destinationHost: NWHostEndpoint(hostname: "", port: "53")

It seems to me like I get significantly more stuff captured with the NETransparentProxyManager approach. In particular a lot of requests going to an address on 10.0.0.0/8 which is not an address on my network and also to fe80::1. So maybe NETransparentProxyManager captures more DNS traffic than NEDNSProxyManager?

The main difference is the ability to configure your NETransparentProxyNetworkSettings with includedNetworkRules, excludedNetworkRules, and proxySettings. NEDNSProxyManager is not setup to create configurations like this and is aimed specifically at handling DNS traffic.


Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com
1
EoinN OP
Mar ’21
meaton you mentioned that NEDNSProxyProvider needs to be supervised on iOS, Is that not the case on macOS?

Also, I think you mentioned some time ago that there was some documents forthcoming on this.
0
Systems Engineer OP
Apple
Mar ’21

meaton you mentioned that NEDNSProxyProvider needs to be supervised on iOS, Is that not the case on macOS?

NEDNSProxyProvider can be developed and distributed on macOS via Developer ID without MDM. You will just need to Developer ID sign and Notarize these builds before distribution.

Also, I think you mentioned some time ago that there was some documents forthcoming on this.

Yes, I am hoping to get this done soon.


Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com
0
Which NetworkExtension API to use
First post date Last post date
 
 
Q