Combining Bonjour and QUIC multiplex group using Network.framework

Does NWConnectionGroup not support connecting to Bonjour endpoints?

I don’t have an immediate answer to that, but I’d like you to run a test. If you switch to a .hostPort(…) endpoint, does everything work?

You don’t need to do the full resolve thing to test this, just hard code the service’s local DNS name. For example, if the service name is Guy Smiley then the local DNS name is gonna be guy-smiley.local..

The point of this test is that it’ll determine you whether this issue is specific to your use of a .service(…) endpoint, and not some other oddity. Oh, and if the .hostPort(…) endpoint works then that’s good evidence that your proposed workaround is feasible.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thank you for the quick reply.

Unfortunately I don't quite understand how I would do this. I assume I would still use a NWListener with a NWListener.Service to advertise my service over Bonjour? But even if I don't need to resolve the hostname and instead hardcode it (in my case as "_aircam._udp.local."), I'd still need to specify a port to use .hostPort(), right? And NWListener doesn't allow me to choose my own port as far as I can tell.

I assume nw_group_descriptor_allows_endpoint is a closed source function?

I did (try to) implement the workaround of using Bonjour just for device discovery, resolving the hostname and then trying to establish a QUIC tunnel (multiplexed NWConnectionGroup) using that hostname and a hardcoded port. Unfortunately I did not get very far before hitting another wall.

Creating the NWConnectionGroup using my QUIC parameters and a .hostPort() endpoint worked without getting an error. On the other device I am creating a NWListener with QUIC parameters and the hardcoded port. Its newConnectionGroupHandler is called, and after accepting it (by calling start(queue:..)) the NWConnectionGroups on both devices enter the .ready state. Great!! 🥳

Next I am getting a QUIC stream (NWConnection) from the group by calling .extract(), and call .start(queue:..) on it. Unfortunately this is where I hit a wall again. The logs show the following errors:

nw_endpoint_flow_setup_cloned_protocols [C4 fe80::1c46:3f2a:b0a6:c276%en0.7934@en0 in_progress channel-flow (satisfied (Path is satisfied), interface: en0[802.11], scoped, ipv4, ipv6, dns, uses wifi)] could not find protocol to join in existing protocol stack

nw_endpoint_flow_failed_with_error [C4 fe80::1c46:3f2a:b0a6:c276%en0.7934@en0 in_progress channel-flow (satisfied (Path is satisfied), interface: en0[802.11], scoped, ipv4, ipv6, dns, uses wifi)] failed to clone from flow, moving directly to failed state

and the connection ends up .cancelled.

The exact same error messages were being discussed in this older thread on QUIC multiplexed groups. However in that thread the issue ended up being different QUIC options being used for establishing the tunnel and the individual streams. In my case I am letting the stream inherit the QUIC options from the group/tunnel (though I also tried passing them in explicitly). I am using the same QUIC options that had been working for establishing a single NWConnection.

Hmmmm, I’m not sure why this is failing. I have a test project that uses a QUIC connection group with .hostPort(…) and I didn’t encounter this issue.

Hmmm, looking at my code I’m not using extract(connectionTo:using:). Rather, I’m creating a new connection with the NWConnection.init(from:to:using:) initialiser. Wanna give that a try?

ISTR that I’ve tested this from both the client and server side and it worked in both cases.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thank you, I tried it but it didn't make any difference.

However I just figured it out: The issue was that I was adding my custom protocol framer options into the NWParameters in addition to the QUIC options when establishing the tunnel / connection group. It seems that tripped up nw_endpoint_flow_setup_cloned_protocols.

Removing the custom framer allows me to now successfully create a NWConnection from the group.

Next I'll have to figure out how to get my custom framer added back into the individual NWConnections. It seems that even though the .parameters property is a constant, I can still modify it because it is a reference type. So hopefully that will work. 🤞

I filed enhancement request FB15647226 for the initial issue of NWConnectionGroup not supporting connections to Bonjour service endpoints.

Note that it’s better to reply as a reply rather than in the comments; see Quinn’s Top Ten DevForums Tips.

Unfortunately I could not get it to work.

Bummer. I’d appreciate you filing another bug about that. As before, please post your bug number.

While I was here I decided to check on the status of FB15647226. I don’t have any info to share as to when the fix will ship, but your bug is definitely with the right folks and we understand why it’s failing.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thanks, yes, I much prefer replies over (the very hidden) comments. I just thought using a reply to reply to a (kind of invisible) comment would be weird/wrong.

I did just file this as FB15694053

Thank you for your update on the other FB, much appreciated!

My workaround was to rip out the (very basic) custom protocol framer and implement the same packaging logic on top of NWConnection's byte stream send and receive methods. Which is actually much simpler code, but maybe less performant? (Because of an additional data copy?)

Thanks for filing FB15694053.

My workaround was to rip out the (very basic) custom protocol framer and implement the same packaging logic on top of NWConnection's byte stream send and receive methods. Which is actually much simpler code

Cool.

but maybe less performant? (Because of an additional data copy?)

Possibly. I’ve found that it’s hard to make predictions about performance based solely on intuition. If it matters, you need to measure. However, you can’t measure in this case because one of the options doesn’t work )-:

Taking a step back, I suspect that one of the reasons I’ve not heard about this framer problem before is that most folks using QUIC don’t use a framer. Rather, they rely on the fact that QUIC can efficiently multiple streams over the connection group. If you have time to compare the performance of that option against your new ‘user space’ framing code, I’d be most interested in the results.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Interesting. So people use a new stream even for sending tiny control messages? I guess that makes sense if creating the stream is basically free. Though I assume you lose all guarantees about the order in which messages will be received?

In the end I think performance of the current approach should be fine (of course I'll profile it). Even if there is an additional memory copy (not sure there is), I doubt it would make a meaningful difference.

So people use a new stream even for sending tiny control messages?

Well, it kinda depends on the protocol. However, the canonical example of this is HTTP/3, which each HTTP request/response pair goes via its own stream. Those aren’t tiny, but they can be pretty small.

I assume you lose all guarantees about the order in which messages will be received?

Yes. Although in HTTP/3 that’s a feature, not a bug, in that it avoids head-of-line blocking.

I doubt it would make a meaningful difference.

That’s my intuition as well. IME, unless your dealing with gigabit-ish speeds, memory copies are just noise.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"