PacketTunnelProvider - Issue with inbound TCP traffic

Hi,

MacOS Version- 10.15.6

I have a remote access VPN application which made use of KEXT earlier and now I have migrated to NetworkExtension framework. I am running PacketTunnelProvider as SystemExtension.
I maintain two threads, one for reading packets from utun and other for writing packets to utun.
Here is a gist of my packet handler threads.
  1. Reading packets from utun

Code Block
read_packets_from_utun(){
[tun_device.packetFlow readPacketsWithCompletionHandler:^(NSArray<NSData *> * _Nonnull packets, NSArray<NSNumber *> * _Nonnull protocols) {
    send_packets(packets, protocols);
     
  }];
}
send_packets(packets_array){
pcount = packets_array.count;
        for(int i=0; i<pcount; i++){
// Encrypt and send packet to server
           encrypt_and_send(packets_array[i]);
        }
// Read next packets
}

The 'send_packets' function then encrypts each packet in a loop, sends it to the server and then calls 'readPacketsWithCompletionHandler' again in order to keep receiving packets.

2. Writing packets to utun
Code Block
NSArray<NSData*>* packet_array = [NSArray arrayWithObject:packet];
    NSArray<NSNumber*>* proto_arr = [NSArray arrayWithObject:proto_num];
    [tun_device.packetFlow writePackets:packet_array withProtocols:proto_arr ];

After decrypting each packet, I write each packet back to utun via writePackets call.

Question. Is this the correct way of using writePackets call?

I ran some performance tests using iperf with the system extension and observed the following. (I also compared the performance numbers with that of older application with KEXT).
  1. Performance for UDP traffic (bidirectional) is observed to be the same for both the SystemExtension and KEXT.

  2. Performance for outbound TCP traffic (Macbook sending data to remote server) is found to be the same for both SystemExtension as well as KEXT.

  3. There is performance degradation seen for inbound TCP traffic (remote server sending data to macbook).

The performance is degraded to approximately 10% of that of older app(KEXT).
I also tried buffering packets before writing to utun, but there was no luck :(

Am I missing something while handling packets?

If that's not the case, Is there any known issue with TCP traffic with PacketTunnelProvider?

Thanks.




After decrypting each packet, I write each packet back to utun via writePackets call.

Is this after you have received the packets back from the server?

The performance is degraded to approximately 10% of that of older app(KEXT).
I also tried buffering packets before writing to utun, but there was no luck :(

Buffering packets here can get you into problems if you run into poor network conditions. I would either declare an upper bound of buffered packets that you can keep in memory before reading more from your server or only read from the remote/server side after the local write has taken place. Overall though, if you are reading from the utun interface, and then sending the packets to your server, then reading the packets from your server, and writing the packets to the utun interface it sounds like you are handling this correctly.

If that's not the case, Is there any known issue with TCP traffic with
PacketTunnelProvider?

Not that I am aware of.


Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com
PacketTunnelProvider - Issue with inbound TCP traffic
 
 
Q