latency of NEFilterPacketProvider

I have ran some latency tests (measuring download and upload from the internet).

When filtering via a NEFilterData, the speed stays almost the same (about a 5% decrease).

However, when filtering via a NEFilterPacket, even when doing only this:

override func startFilter(completionHandler: @escaping (Error?) -> Void) {

        packetHandler = { (context:NEFilterPacketContext, interface:nw_interface_t, direction:NETrafficDirection, packetBytes:UnsafeRawPointer, packetLength:Int) in
            return .allow
        }
        completionHandler(nil)
    }

Then the speed is about 40% of the speed without filtering.

Is that expected? Is there anything I can do to prevent this?

I tried to set

filterManager.grade = .inspector

but that did not help either.