Network is down error received in SystemExtension while Internet is working in other Apps like Safari

Question

Created Sep ’22

Replies 5

Boosts 0

Views 3.4k

Participants 3

We have created a PacketTunnel provider using SystemExtnesion for macOS. We are using Network.framework for creating a web-socket connection. Most of the time everything works fine but sometimes, after system restart, our Packet tunnel provider fails to make a connection to our web-socket server with below error:

POSIXErrorCode(rawValue: 50): Network is down

while internet in every other app is working fine. We do not see any packets being sent in Wireshark. When looking into Console logs I noticed below logs:

default	14:38:55.831499+0530	mynetworkextension	[TunnelConnection.swift:44]  ["Connect to gateway MyGatewayName"]
default	14:38:55.832142+0530	mynetworkextension	[WebSocketClient.swift:36]  ["Initialize webscoket for gateway ca5d85cf-0e67-42f3-91d4-67fa56260788, URL wss://my-wss-server"]
default	14:38:55.833280+0530	mynetworkextension	CSSM Exception: -2147413736 CSSMERR_DL_DATASTORE_ALREADY_EXISTS
default	14:38:55.838783+0530	mynetworkextension	CSSM Exception: -2147413736 CSSMERR_DL_DATASTORE_ALREADY_EXISTS
default	14:38:55.842398+0530	kernel	Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default	14:38:55.843311+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 1
default	14:38:55.843553+0530	kernel	Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default	14:38:55.843859+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 4
default	14:38:55.844413+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 7A FF 74 1C ED DF 86 82 1B 26 F6 51 47 F7 87 CB
default	14:38:55.844691+0530	kernel	Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default	14:38:55.844926+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 0B FD C7 2A 4F 7F F4 61 AA 01 55 9B 36 BC 5E C3
default	14:38:55.845427+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 11 3A F2 96 8D 82 72 1F 8A 7D 9F A4 5F F4 40 60
default	14:38:55.845833+0530	kernel	Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default	14:38:55.845936+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 00 A2 4B A8 95 3F 9A 7F 0A 91 22 84 C5 ED 77 8B C8
default	14:38:55.846430+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 47 9B 21 74 59 64 82 84 F4 B5 94 02 46 A2 41 EA
default	14:38:55.846968+0530	kernel	Unable to send CoreAnalytics event. Giving up.
default	14:38:55.846987+0530	kernel	Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default	14:38:55.847052+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 00 9E 55 C4 13 F2 37 DA 16 04 62 AF A7 46 57 A3 D9
default	14:38:55.847630+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 00 A9 83 7E FA 54 6E D7 C5 28 7B 6A 38 D6 6A E6 CA
default	14:38:55.848126+0530	kernel	Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default	14:38:55.848393+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 39 DC 33 FA 9B 7C D8 34 DE 2E 15 04 3B B7 6F 35
default	14:38:55.849277+0530	kernel	Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default	14:38:55.850313+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 0E 78 87 DD 47 05 59 72 FC 7E 56 05 F5 9B 4C F6
default	14:38:55.850436+0530	kernel	Unable to send CoreAnalytics event. Delaying for 1000 u.s. to see if the queue drains.
default	14:38:55.850898+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 00 FB C2 B2 5B CF 52 FE F0 DD 6C 7A 90 C5 5A E0 A1
default	14:38:55.851356+0530	mynetworkextension	subjectInfo[2.16.840.1.113741.2.1.1.1.3] = 00 F3 80 EF 7E B5 19 51 E7 95 AE C3 13 20 39 CB B1
default	14:38:55.851588+0530	kernel	Unable to send CoreAnalytics event. Giving up.
default	14:38:55.851742+0530	mynetworkextension	Failed to talk to secd after 4 attempts.
default	14:38:55.853388+0530	mynetworkextension	[C221 873998CB-9BC7-4C6E-8218-9BD5A4346538 Hostname#8c901c3c:443 tcp, url hash: 5ffa153d, tls, attribution: developer, context: Default Network Context (private), proc: BD2C7D53-7F93-3CA0-8003-4B8C86F705C9] start
default	14:38:55.853487+0530	mynetworkextension	[C221 Hostname#8c901c3c:443 initial parent-flow ((null))] event: path:start @0.000s
default	14:38:55.853753+0530	mynetworkextension	[C221 Hostname#8c901c3c:443 waiting parent-flow (unsatisfied (Path was denied by NECP policy), interface: en0, ipv4)] event: path:unsatisfied @0.000s, uuid: FA51A7E3-9B79-495C-AB80-0919C89CECCA
default	14:38:55.853790+0530	mynetworkextension	nw_connection_report_state_with_handler_on_nw_queue [C221] reporting state waiting
default	14:38:55.854007+0530	mynetworkextension	[C221 Hostname#8c901c3c:443 in_progress parent-flow (unsatisfied (Path was denied by NECP policy), interface: en0, ipv4)] event: flow:start_child @0.000s
default	14:38:55.854043+0530	mynetworkextension	nw_connection_report_state_with_handler_on_nw_queue [C221] reporting state preparing
default	14:38:55.854156+0530	mynetworkextension	[C221.1 Hostname#8c901c3c:443 initial path ((null))] event: path:start @0.000s
default	14:38:55.854198+0530	mynetworkextension	[WebSocketClient.swift:332]  ["Connection state changed to waiting(POSIXErrorCode(rawValue: 50): Network is down)"]

and below log seems to be the cause of failure:

default 14:38:55.853753+0530 mynetworkextension [C221 Hostname#8c901c3c:443 waiting parent-flow (unsatisfied (Path was denied by NECP policy), interface: en0, ipv4)] event: path:unsatisfied @0.000s, uuid: FA51A7E3-9B79-495C-AB80-0919C89CECCA

So far my understanding was the NECP is mainly applicable to iOS applications because apps can be denied access to local or cellular network. But did not expect to see this on macOS. Is it a valid failure on macOS? If yes, what would be the cases where a path on Mac would become unsatisfied for an application/SysEx? Please note, we are not setting any required or prohibited interface.

Also, the weird part is that this issue disappears if we restart the machine and we are not able to reproduce it consistently. So, my question is:

When would macOS deny network access to an app or extension on macOS?
Since its possible for Content filter to deny network access, is it possible that we are receiving this path unsatisfied error due to a third part content filter? Would content filter deny verdict would result into "Network is down" error?
If not because of any content filter denying access to our extension, how can we debug further and fix this issue?

Boost

Answer 1

DTS Engineer OP

Apple

Sep ’22

So far my understanding was the NECP is mainly applicable to iOS applications because apps can be denied access to local or cellular network.

You are correct that this is the most common cause of NECP restrictions but it’s certainly not the only one. See this post for more.

our Packet tunnel provider fails to make a connection to our web-socket server

Is this WebSocket server local to the Mac? Or something that you’d expect to be accessible via the default hardware interface?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 2

dispatchMain OP

Sep ’22

Thanks @eskimo, for the link.

Is this WebSocket server local to the Mac? Or something that you’d expect to be accessible via the default hardware interface?

No, its a remote WebSocket-server.
Just in case it helps,

We have split tunnel with only a fixed private range of IPv4 addresses in include routes. So, there should not be any cause of the loop in the provider.
We also have a transparent proxy provider in same system extension. That also is seeing "Network is down" error on NWConnection, when PacketTunnel provider receives it.

What are other macOS NECP policies that I should look out for? Since the issue happens at random time and goes away upon machine restart (restarting just the PacketTunnelProvider does not help), that makes it look like its some setting misconfiguration, but I'm not sure what. We have seen this issue with people in different network and geographies.

We also have a NWPathMonitor running which reports that none of wifi, wiredEthernet, cellular or other interface type is connected or path is unsatisfied, when this issue starts happening.

0

Answer 3

DTS Engineer OP

Apple

Sep ’22

What are other macOS NECP policies that I should look out for?

I don’t have a comprehensive list, alas.

[our] transparent proxy provider … also [sees] "Network is down" error on NWConnection

…

We also have a NWPathMonitor running which reports that none of .wifi, .wiredEthernet, .cellular or other interface type is connected or path is .unsatisfied, when this issue starts happening.

Yeah, that all points at NECP.

I recommend that you grab a sysdiagnose log while the problem is happening and then file a bug with that log. Please post your bug number here, just for the record.

I realise that getting a sysdiagnose log can be tricky in situations like this, where the problem appears seemingly at random. If you want to increase the odds of that happening, add code that detects the problem and, on seeing it, requests that the user trigger a sysdiagnose log and send it your way. Enable this code in the builds you send to helpful users — perhaps your beta test pool? — and see if that gets you the results you need.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 4

dispatchMain OP

Sep ’22

Finally one of our beta testers reported this issue and we were able to get sysdiagnose from their machine. A bug has been submitted: FB11594187

0

Answer 5

smaryus OP

Mar ’23

Hi,

Did you've managed to find a solution for this problem?

I don't know if it helps, but we had the same issue when includeAllNetworks is used. We've saw that adding a delay at setTunnelNetworkSettings "fixes" the problem with sockets. We are using the C API, socket(). As far as I saw this is not working with NWConnection, but it works with createTCPConnection too. And for us was reproduced only starting with macos 13.X.

The idea is something like:

setTunnelNetworkSettings(networkSettings) { error in
      DispatchQueue.main.asyncAfter(deadline: .now() + 0.5) {  &lt;--- without this 0.5 delay is not working
                        completeBlockFromStartTunelImpl(error)
     }
}

This is the link with our bug: https://developer.apple.com/forums/thread/723314

Another issue that we still have is that after reboot we cannot connect, even with the above delay. So the other "fix" is to delete the VPN profile from system. And in this case, until reboot everything works fine, until it breaks again :).

Hope it helps.

0