Filter large list of IP addresses

For an NEDataFilter, I am trying to filter a large list of IP addresses (computed from the domain names). I can see 2 ways of doing this:


1) I create many rules NEFilterRule, and apply them via NEFilterSettings, and select .drop

2) I apple the NEFilterSettings with no rule, change a default action to .fitlerData, and then do the lookup of the IP addresses inside the handleNewFlow function.


What would be the best solution, latency wise ? The 2nd seems faster to me, as I can implement O(log(n)) lookup, but are there useful tricks that 1 is doing, that means I should consider it ?

I am trying to filter a large list of IP addresses

Define “large”?

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

50,000 right now.

So option 1 would involve 50,000 rules. I’ve seen folks try to do this sort of thing on iOS and they run into memory limits way before then. macOS is a different platform, so it might cope, but it’s definitely a concern.

There is a third option here, namely, process the IP addresses into a smaller set of rules that has some false negatives, and then deal with those false negatives in

-handleNewFlow:
.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks for your response.


I am surprised that this would create issues on the memory side. NEFilterRule has a size on the heap of 24 bytes. So 50,000 objects of this size would be about 1.14 MB. That's low even for an iPhone ?


I am not entirely sure what you mean by processing the IP addresses into a smaller set of rules. I am not clear how that could be done.


From the quick and dirty measurement I made, by doing (2) and organizing the ip addresses into as set, the lookup takes about 200 us (0.2 milliseconds). I was wondering if doing (1) could beat that, but from your answer you do not seem to be bullish on (1).

NEFilterRule
has a size on the heap of 24 bytes.

You have misunderstood how these sizes are calculated.

NEFilterRule
is a small object but it has references to lots of other objects. Moreover, it’s not just the in-memory cost you have to consider. The system has to move these rules between processes.

One way to get a rough understanding of this cost is to archive the resulting rules. For example:

let rules = (0..<50000).map { i -> NEFilterRule in
    let b3 = i & 0xff
    let b2 = (i >> 8) & 0xff
    let endpoint = NWHostEndpoint.init(hostname: "1.2.\(b2).\(b3)", port: "0")
    let networkRule = NENetworkRule(destinationNetwork: endpoint, prefix: 24, protocol: .any)
    let filterRule = NEFilterRule(networkRule: networkRule, action: .filterData)
    return filterRule
}
let settings = NEFilterSettings(rules: rules, defaultAction: .allow)
let start = Date()
let data = try! NSKeyedArchiver.archivedData(withRootObject: settings, requiringSecureCoding: true)
let end = Date()
print(end.timeIntervalSince(start))
print(data.count)

On my main work Mac (a 2016 MacBook Pro) it takes roughly 1.5 seconds to archive the rules and the resulting archive is roughly 16 MiB.

I am not entirely sure what you mean by processing the IP addresses into a smaller set of rules. I am not clear how that could be done.

My suggestion is that you group addresses by their common prefix. For example, if you have 1.2.3.4 and 1.2.3.13, you could create a rule that matches 1.2.3.0/28 (that is 1.2.3.0 through 1.2.3.15).

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

I see, that makes sense. Thanks for the explanation, I'll look into that.

Filter large list of IP addresses
 
 
Q