Understanding peekBytes in handle inbound/outbound flows

I am trying to make sense of some numbers in network extension.

Background:
I have a NEFilterDataProvider on Catalina MacOS working, mostly it is an extension of existing SimpleFirewall except that it attempts to scan all out going and incoming TCP data.
The objective is to
  1. Scan as much stream as possible in a performant manner, this is being achieved by, returning

Code Block
let userVerdict: NEFilterDataVerdict = .init(passBytes: data.count /*total bytes indicated*/ , peekBytes: Self.peekBytes)

in the inbound/outbound data handler

2. If 1. is not feasible then try to atleast sample offsets

3. Meter the inbound and outBound TCP flow

To achieve 3. I think the only possible way is to scan all the data?

Code Block
/* This is what the data handlers looks like */
 override func handleInboundData(from flow: NEFilterFlow, readBytesStartOffset offset: Int, readBytes: Data) -> NEFilterDataVerdict {
    return .init(passBytes: readBytes.count, peekBytes: Self.peekBytes)
}

To test how our network extension, which simply requests for more data in its data handlers(in/out - bound) will fair in these scenarios we ran some tests on a VmWare fusion environment which had a vmnet network, using a tool called iperf(v2). Another machine acted as server and a macOS vm worked as client. The test run on the macOS looked as follows

Code Block
iperf -c 172.16.1.11 -t 300 -r -b 200M #limit throughput to 200Mbps

Code Block
iperf -c 172.16.1.11 -t 300 -r #saturate the network


While running these tests we saw huge variations in CPU utilization and Network throughput simply by changing the peekBytes value in the FilterDataVerdict. For large peekbytes things were more performant and for lower peekbytes they weren't.

However with physical interfaces over internet the sweet spot for peekbytes was at 1400 bytes.


I have shared what we observed here, I know links are not allowed so you will have to append https

imgur.com/a/E1yCaVs

I hope what I am saying makes sense?
The ask is if there is any guidance around peekBytes for NetworkExtensions.

Many thanks!