We are building a macOS client where we make a web socket connection to our server using NWConnection. The code to to create NWParameters is:
let options = NWProtocolTLS.Options()
let securityProtocolOptions = options.securityProtocolOptions
//....configure security options for server and client cert validation...
let parameters = NWParameters(tls: options)
let wsOptions = NWProtocolWebSocket.Options()
wsOptions.autoReplyPing = true
wsOptions.setAdditionalHeaders(additionalHeaders.map { ($0.key, $0.value) } )
parameters.defaultProtocolStack.applicationProtocols.insert(options, at: 0)
With these parameters and an NWEndpoint object, we create a connection which connects and transfers data well from both sides. However, whenever server gracefully closes the connection, the client remains oblivious to this and does not close the connection. It only detects a timeout when trying to send some data over the connection after the server disconnect. We looked into Wireshark and we do not see any FIN,ACK or RST packets being received from server.
However, in our windows client, when same exact server closes the connection, we are seeing FIN, ACK and connection immediately closes on client side as well.
We also tried to test same behaviour in golang with a small snippet created by our team member running on Mac itself. This snippet also receives FIN,ACK from server and closes the connection immediately. Only NWConnection in our Mac client does not receive close connection.
So, the question arises, why is NWConnection not receiving FIN,ACK and not closing the connection when a windows as well as a golang client on Mac is able to. Is there any extra configuration required for NWConnection or NWParameter? Is the NWParameter creation code correct? We already checked and we are continuously calling receiveMessage on the NWConnection object. So, missing read is not the issue here.
Also, I do not see any connection timeout option in NWProtocolWebSocket but it exists in NWProtocolTCP. So, is there a way to set connection timeout for web socket connection using NWConnection?
I am developing an NE based System Extension on macOS Catalina 10.15.7 using Xcode 12.4. I have everything in place and my extension is running fine. However, I am not able to attach debugger to my extension from Xcode. I can go to terminal and as root user, I am able to attach lldb to the running system extension. But I am not able to attach the debugger from Xcode using Debug - Attach to Process by PID or Name - Debug Process As - root. Xcode just keeps on Waiting to Attach. If the extension is already running, then I am able to attach by PID but never by name. I would like to be able to start debugger using process name and launch the extension and be able to debug from first line of code in extension.
Also, I really like Xcode's contextual debugging where I can see most of the variables and information without any extra effort. With terminal the debugging is comparatively hectic.
So, is there any way I can attach to System Extensions by name in Xcode? I have tried using target name as well as extension bundle identifier and several other combinations. From terminal I can attach using bundle identifier of the extension itself.
Post not yet marked as solved
We have created a PacketTunnel provider using SystemExtnesion for macOS. We are using Network.framework for creating a web-socket connection. Most of the time everything works fine but sometimes, after system restart, our Packet tunnel provider fails to make a connection to our web-socket server with below error:
POSIXErrorCode(rawValue: 50): Network is down
while internet in every other app is working fine. We do not see any packets being sent in Wireshark. When looking into Console logs I noticed below logs:
default 14:38:55.831499+0530 mynetworkextension [TunnelConnection.swift:44] ["Connect to gateway MyGatewayName"]
default 14:38:55.832142+0530 mynetworkextension [WebSocketClient.swift:36] ["Initialize webscoket for gateway ca5d85cf-0e67-42f3-91d4-67fa56260788, URL wss://my-wss-server"]
default 14:38:55.833280+0530 mynetworkextension CSSM Exception: -2147413736 CSSMERR_DL_DATASTORE_ALREADY_EXISTS
default 14:38:55.838783+0530 mynetworkextension CSSM Exception: -2147413736 CSSMERR_DL_DATASTORE_ALREADY_EXISTS
default 14:38:55.842398+0530 kernel Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default 14:38:55.843311+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 1
default 14:38:55.843553+0530 kernel Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default 14:38:55.843859+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 4
default 14:38:55.844413+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 7A FF 74 1C ED DF 86 82 1B 26 F6 51 47 F7 87 CB
default 14:38:55.844691+0530 kernel Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default 14:38:55.844926+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 0B FD C7 2A 4F 7F F4 61 AA 01 55 9B 36 BC 5E C3
default 14:38:55.845427+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 11 3A F2 96 8D 82 72 1F 8A 7D 9F A4 5F F4 40 60
default 14:38:55.845833+0530 kernel Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default 14:38:55.845936+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 00 A2 4B A8 95 3F 9A 7F 0A 91 22 84 C5 ED 77 8B C8
default 14:38:55.846430+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 47 9B 21 74 59 64 82 84 F4 B5 94 02 46 A2 41 EA
default 14:38:55.846968+0530 kernel Unable to send CoreAnalytics event. Giving up.
default 14:38:55.846987+0530 kernel Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default 14:38:55.847052+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 00 9E 55 C4 13 F2 37 DA 16 04 62 AF A7 46 57 A3 D9
default 14:38:55.847630+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 00 A9 83 7E FA 54 6E D7 C5 28 7B 6A 38 D6 6A E6 CA
default 14:38:55.848126+0530 kernel Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default 14:38:55.848393+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 39 DC 33 FA 9B 7C D8 34 DE 2E 15 04 3B B7 6F 35
default 14:38:55.849277+0530 kernel Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default 14:38:55.850313+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 0E 78 87 DD 47 05 59 72 FC 7E 56 05 F5 9B 4C F6
default 14:38:55.850436+0530 kernel Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default 14:38:55.850898+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 00 FB C2 B2 5B CF 52 FE F0 DD 6C 7A 90 C5 5A E0 A1
default 14:38:55.851356+0530 mynetworkextension subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 00 F3 80 EF 7E B5 19 51 E7 95 AE C3 13 20 39 CB B1
default 14:38:55.851588+0530 kernel Unable to send CoreAnalytics event. Giving up.
default 14:38:55.851742+0530 mynetworkextension Failed to talk to secd after 4 attempts.
default 14:38:55.853388+0530 mynetworkextension [C221 873998CB-9BC7-4C6E-8218-9BD5A4346538 Hostname#8c901c3c:443 tcp, url hash: 5ffa153d, tls, attribution: developer, context: Default Network Context (private), proc: BD2C7D53-7F93-3CA0-8003-4B8C86F705C9] start
default 14:38:55.853487+0530 mynetworkextension [C221 Hostname#8c901c3c:443 initial parent-flow ((null))] event: path:start @0.000s
default 14:38:55.853753+0530 mynetworkextension [C221 Hostname#8c901c3c:443 waiting parent-flow (unsatisfied (Path was denied by NECP policy), interface: en0, ipv4)] event: path:unsatisfied @0.000s, uuid: FA51A7E3-9B79-495C-AB80-0919C89CECCA
default 14:38:55.853790+0530 mynetworkextension nw_connection_report_state_with_handler_on_nw_queue [C221] reporting state waiting
default 14:38:55.854007+0530 mynetworkextension [C221 Hostname#8c901c3c:443 in_progress parent-flow (unsatisfied (Path was denied by NECP policy), interface: en0, ipv4)] event: flow:start_child @0.000s
default 14:38:55.854043+0530 mynetworkextension nw_connection_report_state_with_handler_on_nw_queue [C221] reporting state preparing
default 14:38:55.854156+0530 mynetworkextension [C221.1 Hostname#8c901c3c:443 initial path ((null))] event: path:start @0.000s
default 14:38:55.854198+0530 mynetworkextension [WebSocketClient.swift:332] ["Connection state changed to waiting(POSIXErrorCode(rawValue: 50): Network is down)"]
and below log seems to be the cause of failure:
default 14:38:55.853753+0530 mynetworkextension [C221 Hostname#8c901c3c:443 waiting parent-flow (unsatisfied (Path was denied by NECP policy), interface: en0, ipv4)] event: path:unsatisfied @0.000s, uuid: FA51A7E3-9B79-495C-AB80-0919C89CECCA
So far my understanding was the NECP is mainly applicable to iOS applications because apps can be denied access to local or cellular network. But did not expect to see this on macOS. Is it a valid failure on macOS? If yes, what would be the cases where a path on Mac would become unsatisfied for an application/SysEx?
Please note, we are not setting any required or prohibited interface.
Also, the weird part is that this issue disappears if we restart the machine and we are not able to reproduce it consistently. So, my question is:
When would macOS deny network access to an app or extension on macOS?
Since its possible for Content filter to deny network access, is it possible that we are receiving this path unsatisfied error due to a third part content filter? Would content filter deny verdict would result into "Network is down" error?
If not because of any content filter denying access to our extension, how can we debug further and fix this issue?
Post not yet marked as solved
In our macOS SystemExtension, we are using Network.framework for creating web socket connection to our remote WSS server. We provide authentication token in cookies of the connection and websocket server validates it before upgrading the connection from HTTP to WebSocket. If the cookie is invalid, server returns 403 HTTP status code and closes the connection.
When server returns 403, in Big Sur and Monterey, we get state update of failed(let error) where error is NWError.posix(.ECONNABORTED). However, in Ventura, we are getting state waiting(let error) where error is NWError.posix(.ECONNABORTED).
As per documentation, waiting state should be received if there is any network error in establishing the connection and connection goes into waiting for network path change. But in this case, TCP connection to server is established, HTTP headers are received and validated by server and then get rejected. So shouldn't this result into connection failed instead of waiting?
This behaviour has only changed on Ventura. On Big Sur and Monterey we still get failed with error.
Post not yet marked as solved
We have a PacketTunnelProvider as a SystemExtension supporting macOS Big Sur and above. We supply below config as part of tunnel settings to bring up the tunnel:
Remote Address: 240.0.0.2
DNS Servers: [240.0.0.3]
Match Domains: [A list of FQDNs]
IPv4 Address: 240.0.0.1 and subnet mask 255.255.255.255
IPv4 Include Routes: 240.0.0.1 with subnet mask 255.192.0.0
Once we provide these settings, a utun is created (utun4), the routes are setup as in below screenshot and tunnel is connected.
My first question is, why is there duplicate entry for my tunnel address? AFAIK, tunnel address is by default part of exclude routes.
Next when I access an FQDN, we get the DNS request in our provider and respond to it with an IP of 240.0.0.5 and start getting data packets. If we check route table after this, there is an entry for the new IP we have responded with:
So my next question is, why is the new IP (240.0.0.5) setup against our utun4 when there is already 240.0.0/10 setup against our utun4 covering this IP? Also, if you notice, the duplicate entry for tunnel address is gone, not sure why. Nothing changed in tunnel settings.
Next, I tried to disconnect and reconnect the provider from Network Preferences and tunnel came up with same tunnel settings but a new utun (utun7) got created and route setup looked like below screenshot:
and here are the utuns from ifconfig
So the next question arises is, does PacketTunnelProvider create new utun every time it gets disconnected and reconnected? And if it does, why is not the previous stale utun destroyed? Why are my IP settings still pointing to old utun in route table?
Next, I tried to access the FQDN again, resulting in a DNS request which was responded with IP 240.0.0.5. After that, the route table looks like this:
Now, we can see a duplicate entry for 240.0.0.5 on utun4 as well as utun7. Why? Couldn't that cause confusion or even route conflicts?
Please note, so far we have not had any functional issues due to confusing or even conflicting route setups. After disconnect and reconnect, all the traffic goes correctly via utun7 even though there is old and stale utun4 with conflicting routes.
Another note: All the route outputs are generated using netstat -r -f inet.
We have a PakcetTunnelProvider in SystemExtension with split tunnelling. We are setting up a private range of IP address (240.0.0.1/10) as include routes and a few match domains using NEPacketTunnelNetworkSettings.
On fresh install, everything works fine. We are able to setup tunnel settings and receive DNS as well as data traffic as per our rules.
But during upgrade of our app from one version to another, the tunnel settings are not applied correctly. Only match domains are being setup but include route fails and all our traffic goes to en0 interface. setTunnelNetworkSettings api returns no error but in console log we see below logs:
2022-10-05 17:47:06.698948+0530 0x315 Default 0x0 135 0 configd: [com.apple.SystemConfiguration:IPMonitor] failed to add route, File exists:
2022-10-05 17:47:06.698955+0530 0x315 Default 0x0 135 0 configd: [com.apple.SystemConfiguration:IPMonitor] Net 240.0.0.1/10 Ifp utun5 Ifa 240.0.0.1 [last] [force]
This log is printed about 32 ms after we receive callback from setTunnelNetworkSettings method without error. If we do netstat -r, we don't see the route listed there at all. So, we are sure that no other interface is claiming that route. If we apply those settings again, it succeeds and starts working again.
So, we do not understand why route add is failing and why is NE API not returning this error to our extension?
The update path is:
We have extension up and running
We stop app and replace with new one
New app triggers SysEx replacement which stops the NE
We try to clear tunnel settings during stop but it gets ignored (console log -> ESMVPNSession[Primary Tunnel:MyVPN:E7DBA018-7D8F-4D6E-9DCE-141D000EA5CC:(null)] in state NESMVPNSessionStateStopping: plugin disconnecting, ignoring clear configuration request)
The replacement completes and NE Tunnel is started again
It creates new tunnel settings with match domains and hard coded include routes and sets the configuration.
We get success in setTunnelNetworkSettings callback and start reading packets.
configd prints the log that add route has failed.
Our tunnel gets DNS request as per match domains but does not receive any IP packets based on provided include route. All those packets are sent to en0 interface.
Post not yet marked as solved
We are developing a VPN app using SystemExtension and PacketTunnel provider. Everything is working fine in the SystemExtension but the information displayed Network Preferences and VPN menu item are missing or inconsistent or even duplicates.
In the below screenshot, you can see time mismatch between VPN status and menu item.
In below screenshot, you can see duplicate Disconnect and show time options for single configured VPN:
and in below screenshot, you can see the show time and status options completely missing even though VPN is running for some time:
We see this issue on macOS Big Sur as well as Monterey.
Is there any way we can control or make this behaviour consistent?
Post not yet marked as solved
We are trying to evaluate certificate trust chain in our macOS app. We are setting the certificate chain (Root and two Intermediate CA certificates) using SecTrustSetAnchorCertificates and then calling SecTrustEvaluateWithError. The result is success.
Next time, we are calling SecTrustSetAnchorCertificates with one intermediate CA certificate missing in the certificate chain and then calling SecTrustEvaluateWithError for our server trust. The result is still success.
Next, we are calling SecTrustSetAnchorCertificates with all intermediate certificates but missing Root CA in certificate chain and then calling SecTrustEvaluateWithError for our server trust. The result is false/unsuccessful.
The first and third scenarios are expected. But how is trust evaluation successful when one of intermediate CA certificate is missing? Is macOS caching the intermediate CA certificates we have provided to SecTrustSetAnchorCertificates some other time and using it the next time when it is missing one of intermediate CA certificates since the documentation says intermediate CA certificates are looked up in different location including
Among any certificates you previously provided by calling SecTrustSetAnchorCertificates(_:_:)
but not the Root CA?
If caching is the reason, is there a way we can clear cached intermediate CA certificates so that it only uses the certificate chain I provide in most recent call to SecTrustSetAnchorCertificates? I have already tried passing nil to SecTrustSetAnchorCertificates and then passing the certificate chain in subsequent call. The result is still a success.
Note: All our Root and intermediate CA certificates are custom certificates and not available outside. We have also tried to set false in SecTrustGetNetworkFetchAllowed and result is still the same.
In recent versions of macOS Big Sur and Monterey, SystemExtnesions are getting replaced with same version.
We have an application with a system extension and UI in the app displays status of the system extension to remind users to approve it in case they have not. Since there is no way to query the status of a SystemExtnesion, to display the status in UI, we would submit the SystemExtension request every time UI is displayed and update the status as per the delegate callback. Earlier, if SystemExtension is already approved, with would immediately get .completed result.
However, since recently we are noticing that whenever the app submits the extension request, if there is already an approved SystemExtension, then replacement delegate callback is triggered for same extension version. If we return .replace action, the existing SystemExtension is getting disabled and replaced with SystemExtension of same version. This is contradictory to the documentation of replacement delegate callback which indicates that replacement request will only be called when an extension of different version is found.
From the documentation:
The manager calls this method when it encounters an existing extension with the same team and bundle identifiers, but with different version identifiers. It uses the CFBundleVersion and CFBundleShortVersionString identifiers to determine if the existing and new versions differ. The delegate must make a decision on whether to replace the existing extension.
This issue is noticed in development as well as production signed version of our application and in the same run of the same application process.
This unexpected behaviour is causing issues in our app. For example, when the existing SystemExtension is being disabled, all our network extensions are being stopped. This is resulting in loss of functionality.
Here are some relevant console logs, attached.
logs.txt
Post not yet marked as solved
In our Mac application, we are creating a web-socket connection using NWConnection and we are able to successfully establish the connection and read/write data from both sides. We have auth tokens which are sent in headers of NWProtocolWebSocket.Options to the server. If token is good, server accepts the web-socket connection. As per RFC 6455, if server does not want to accept the connection for any reason during web-socket handshake, it returns 403 status code. In our case, if cookies are not valid, server returns 403 during web-socket handshake.
However, we could not find a way to read this status code in Network.framework. We are only getting failed state with NWErrorwhich is .posix(53) but there is no indication of the status code 403. We tried looking into protocol metadata on NWConnection object and they are nil.
We tested the same using URLSessionWebSocketTask where in failure callback method, we could see 403 status code on task.response which means client is getting the code correctly from server.
So, is there a way to read the HTTP status code returned by server during web-socket handshake using Network.framework?
Post not yet marked as solved
We are developing a PacketTunnel provider based System Extension that creates a utun device when tunnel settings are supplied and everything works fine.
Now, when we try to bring down the utun created by our extension using ifconfig utun down, the extension does not receive packets anymore. So the first question is, is there a way for extension to know that its been marked down? We would want to display a status to our UI to users in case they forget about it.
Next is, when we mark the utun device up next using ifconfig utun up, it still does not receive any packets. It stays down. The only way to start reading the packets is to start the tunnel again where we re-supply the tunnel settings. Is this expected behaviour? Is there anything missing here?
We have a Transparent Proxy and a Content filter as System Extensions. We are now developing a Packet Tunnel provider System extension. Since these are three different extensions, we require users to approve these extensions thrice. We were wondering whether it's possible to combine all three into a single system extension to avoid too many uses approval pop-ups. If that's possible,
How are principal classes for each network extension to be configured?
Will the three extensions run as one process or separate processes, when combined into a single system extension?
Post not yet marked as solved
One of our team members reported very high memory usage for our Network System Extension in Activity Monitor. Therefore, to check for leaks and accumulating memory, I plugged it into Instruments which shows quite low usage (~4MB) compared to Activity Monitor (40MB) at same moment. I am instrumenting release version of my extension (and App). So, my questions are:
Why such a huge difference in these two tools?
Which one should be considered more authentic/valid memory usage
Is there any upper limit for Network System Extension? From my iOS experience, I remember iOS has 15MB limit for Network Extensions. Is there any such limit for Network System Extensions?
I am a bit confused about usage of searchDomains of NEDNSSettings and how different is it from matchDomains. I understand that I can add few domains in matchDomains which will be used to redirect DNS requests for provided domains and their subdomains to my NetworkExtension. So, what is use of searchDomains?
From docs it appears that it will work same as matchDomains but only for exact FQDNs. And DNS requests for any subdomains would not redirected to NE but rather to system DNS resolver. Is that understanding correct?
If that understanding is correct, then what's the use of matchDomainsNoSearch? Any domain in match domain would be a superset of same domain appended in searchDomains? So why even append it?
If my understanding is not correct, then what's the use of searchDomains?
Post not yet marked as solved
I am building a PacketTunnel using SystemExtensions. I have a local FQDN that does not quite resolve using DNS server. Therefore, I have an entry in /etc/hosts which points to the IP of the server. I am able to access this server via FQDN in Safari as well as able to ping it in Terminal.
However, my SystemExtension completely ignores the entry in /etc/hosts file and tries to resolve it using default DNS server which results in NXDOMAIN. I can see its DNS query in Wireshark. So, why would the SystemExtension not follow other apps and use the entries from hosts file for FQDN resolution?
I have already tried to flush dnscache, adding a synthesised IPv6 address and many other suggestion on internet but SystemExtension just refuses to resolve using hosts file. Is there any reason for this behaviour or I could be doing something wrong?
dispatchMain.