I am looking into this same topic. To me it seems as if some magic is applied when an NENetworkRule for UDP port 53 for a specific host (e.g. www.example.com) is entered.
Prior to that rule, a DNS request for www.example.com would go to my standard DNS server (let's say 192.168.0.1:53) and not to www.example.com port 53, so one could assume that the entered NENetworkRule does not match.
However, it does seem to match indeed - at least for applications that are using a form of gethostbyname (e.g. if I ping www.example.com from the terminal) then the rule seems to match and sends the DNS request to the Transparent Proxy.
But if an application does its own DNS implementation (such as nslookup), then this still generates traffic for 192.168.0.1 port 53 and that does not match the rule so that the Transparent Proxy doesn't see that traffic with the above rule.
In that way, the implementation of such a NENetworkRule behaves similarly to how DNS requests are redirected if I use the DNSSettings network extension.
Is that understanding correct?