Weird behaviour with Network Framework and QUIC Multiplex Groups.

Hey all here is an example you can try out: https://github.com/paxsonsa/quic-swift-demo

I am prototype a QUIC base application system with a client and server. My server is a simple test to experiment with QUIC and Network Framework but I am see some odd behaviour.

Selecting Stream Direction for new streams

In the example below, we are creating a new multiplexed QUIC connection and establish a new stream once the group connection is ready. In some cases, I want to be able to use a different stream kind (uni/bi).

By specifying the options, I get an error in Xcode console like so:

running....
group state: waiting(POSIXErrorCode(rawValue: 50): Network is down)
group state: ready
Connected using QUIC!
nw_endpoint_flow_setup_cloned_protocols [C3 127.0.0.1:4567 in_progress socket-flow (satisfied (Path is satisfied), viable, interface: lo0)] could not find protocol to join in existing protocol stack
nw_endpoint_flow_failed_with_error [C3 127.0.0.1:4567 in_progress socket-flow (satisfied (Path is satisfied), viable, interface: lo0)] failed to clone from flow, moving directly to failed state
Main Connection State: failed(POSIXErrorCode(rawValue: 50): Network is down)
quic_recovery_pto PTO fired after validation

Here is my swift code:

//
//  main.swift
//  QuicTool
//
//  Created by Andrew Paxson on 2024-01-14.
//

import Foundation
import Network

/// Helper function to create a message frame.
func createMessage(version: UInt8, messageType: UInt8, message: String) -> Data {
    let messageData = message.data(using: .utf8) ?? Data()
    let length = UInt32(messageData.count)

    var data = Data()
    data.append(version)
    data.append(messageType)

    // Convert length to 4 bytes and append (big-endian format)
    let bigEndianLength = length.bigEndian
    data.append(contentsOf: withUnsafeBytes(of: bigEndianLength) { Array($0) })

    // Append 2 bytes of padding for 8-byte alignment
    data.append(Data(repeating: 0, count: 2))

    // Add Message Data.
    data.append(messageData)
    return data
}

// Queue for QUIC things.
let queue = DispatchQueue(label: "quic", qos: .userInteractive)

// Create Inital Options for the tunnel.
// This is using an insecure connection as this operation is meant to be local network.
let endpoint = NWEndpoint.hostPort(host: "127.0.0.1", port: .init(integerLiteral: 4567))
let options =  NWProtocolQUIC.Options(alpn: ["demo"])

// Set the initial stream to bidirectional.
options.direction = .bidirectional

sec_protocol_options_set_verify_block(options.securityProtocolOptions, { (sec_protocol_metadata, sec_trust, sec_protocol_verify_complete) in
    sec_protocol_verify_complete(true)
}, queue)

let parameters = NWParameters(quic: options)

// 1) Create a new multiplexed connection
let descriptor = NWMultiplexGroup(to: endpoint)
let group = NWConnectionGroup(with: descriptor, using: parameters)
var mainConn: NWConnection? = nil

// Here we are establishing a state handler for when the connection to the
// the server is neogiated and "ready". Once its ready we want to establish a
// stream using the group with the options set.
//
// This is the main location of the issue we are seeing where the stream is
// established and the data is sent but never updated.
group.stateUpdateHandler = { newState in
    print("group state: \(newState)")
    switch newState {

    // Once the tunnel is established, create a new stream with bidirectional parameters.
    case .ready:
        print("Connected using QUIC!")

        // 2) In normal application I may want to open different kinds of streams in providing
        // new options. Is there a better way to select the stream kind for subsequent streams?
        let options =  NWProtocolQUIC.Options(alpn: ["demo"])
        options.direction = .bidirectional

        // When providing unique options the stream will fail. Removeing the using argument works.
        mainConn = group.extract()! // force unwrap

        mainConn?.stateUpdateHandler = { state in
                print("Main Connection State: \(state)")
            switch state {
            case .ready:

                // Once the connection is ready, lets send some sweet data sauce.
                //
                // By establishing this new stream and sending data, on the server this causes the inital
                // stream with no handle to be open.
                let version: UInt8 = 1
                let messageType: UInt8 = 1
                let message = "hello, I am from the multiplex group ready."
                let messageData = createMessage(version: version, messageType: messageType, message: message)

                mainConn?.send(content: messageData, isComplete: true, completion: .contentProcessed({ sendError in
                    if let error = sendError {
                        print("There was an error sending data: \(error)")
                    } else {
                        print("Data was sent successfully from Main Connection.")
                    }
                }))

            default:
                break
            }
        }
        // Don't forget to start the connection.
        mainConn?.start(queue: queue)
    default:
        break
    }
}


// Receive new incoming streams initiated by the remote endpoint
// this is not used for this example.
group.newConnectionHandler = { conn in
    print("New Connection: \(conn)")

  // Set state update handler on incoming stream
    conn.stateUpdateHandler = { newState in
      print("newState: \(newState) for \(conn)")
      switch newState {
      case .ready:
          print("got a new stream!")
      default:
          break
      }
  }
  // Start the incoming stream
    conn.start(queue: queue)
}

// Start the group with callback queue
group.start(queue: queue)

print("running....")
// We iterate trying to send data on the new stream we created after the
// connection is established.
while true {
    switch mainConn?.state {
    case .ready:
        // Once the connection is ready, lets send some sweet data sauce.
        let version: UInt8 = 1
        let messageType: UInt8 = 1
        let message = "hello, im from the main loop"
        let messageData = createMessage(version: version, messageType: messageType, message: message)
        print("Local Stream Send: \(messageData)")

        mainConn?.send(content: messageData, completion: .contentProcessed({ sendError in
            if let error = sendError {
                print("There was an error sending data: \(error)")
            }
        }))

        sleep(1)
    default:
        continue
    }
}

Accepted Reply

Am I able to create new streams with different directions in this API? Or do I have to choice between bidirectional or unidirectional for the entire group?

With different options, no. The options must be derived from the base options. What happens is that new streams are derived from the initial QUIC connection that were used to setup the tunnel, so typically those new streams would take on the initial setup options of the main tunnel (or groups) connection. And this makes sense because otherwise you would have streams that would have different connection parameters (and behavior) than the parent connection / tunnel. And if thats happens then a case could be made for an entirely new connection or tunnel.

But back to your error, if you pass in options that are different from the main tunnels options then the protocol stack will be searched and the options will not be found and so an error will be returned.

If you want to extract via an endpoint you can with something like:

mainConn = group.extract(connectionTo: endpoint)

And this should give you:

2024-01-23 06:48:19.950474-0800 0xe525e5   Debug       0x0                  52666  0    TestQUIC: (libquic.dylib) [com.apple.network:quic] quic_stream_alloc_new_id allocated new stream id 4 for bidirectional stream
  • May I query where you got that log output, I am using CFNETWORK_DIAGNOSTICS=3 on my product scheme, are you attaching via Console.app? It'll help me with my other problem which which I will take to another thread.

Add a Comment

Replies

Additionally, there seems to be some weird issues with a hidden stream being created but there is no way to gain access to the handle:

Hidden Streams

My server is simple, it accepts a single connect and then a single stream initiated by the client. In my swift code I am looking to establish a connection and create a bidirectional stream after the multiplex group is ready.

After the group is ready I extract a new stream from the group and attempt to send data once it is ready. However, this new stream is never accepted but on the server side, the hidden stream is accepted. Upon a deeper dive into the logging, I can see that there are two stream open by the end of the program, the newest one is the one I created by extracting it from the group, the first is one I did not create and do not have a handle too. If I change my server to accept twice, the stream I create after the group is ready is usable.

This is odd behaviour, I don't have a handle to the first hidden stream on the client side and attempt to send data with the group handle just creates new streams.

Try setting up the listening side with Network Framework also using NWListener with something similar to the following:

import Foundation
import Network


/// Helper function to create a message frame.
func createMessage(version: UInt8, messageType: UInt8, message: String) -> Data {
    let messageData = message.data(using: .utf8) ?? Data()
    let length = UInt32(messageData.count)

    var data = Data()
    data.append(version)
    data.append(messageType)

    // Convert length to 4 bytes and append (big-endian format)
    let bigEndianLength = length.bigEndian
    data.append(contentsOf: withUnsafeBytes(of: bigEndianLength) { Array($0) })

    // Append 2 bytes of padding for 8-byte alignment
    data.append(Data(repeating: 0, count: 2))

    // Add Message Data.
    data.append(messageData)
    return data
}

let quicOptions = NWProtocolQUIC.Options()

// TODO: Setup your QUIC options here with an ad-hoc identity for testing and 'demo' as the ALPN

let parameters = NWParameters(quic: quicOptions)

let queue = DispatchQueue(label: "NWTestQueue")
let listener = try NWListener(using: parameters, on: .init(integerLiteral: 4567))
let listenerReady = DispatchSemaphore(value: 0)
listener.stateUpdateHandler = { newState in
    switch (newState) {
    case .ready:
        print("listener entered \(newState) - \(listener)")
    case _:
        print("listener entered \(newState)")
    }
}


listener.newConnectionHandler = { newConnection in

    print("New connection: \(newConnection)")

    newConnection.stateUpdateHandler = { state in
        switch state {
        case .ready:
            let version: UInt8 = 1
            let messageType: UInt8 = 1
            let message = "hello, from the listening side."
            let messageData = createMessage(version: version, messageType: messageType, message: message)
            newConnection.send(content: messageData, completion: .contentProcessed( { error in
                if let err = error {
                    print("Sent with error: \(err.debugDescription)")
                } else {
                    print("Sent successfully")
                }

                listenerReady.signal()
            }))
        case _:
            print("stream entered \(state)")
        }

    }
newConnection.start(queue: .main)

}
listener.start(queue: queue)
listenerReady.wait()
print("Finished!")

On the client side then you should see something similar to :

running....
group state: waiting(POSIXErrorCode(rawValue: 50): Network is down)
group state: ready
Connected using QUIC!
Main Connection State: preparing
Main Connection State: ready
Local Stream Send: 36 bytes
Data was sent successfully from Main Connection.
Local Stream Send: 36 bytes
Local Stream Send: 36 bytes
Local Stream Send: 36 bytes
Local Stream Send: 36 bytes
Local Stream Send: 36 bytes

As for the error:

group state: waiting(POSIXErrorCode(rawValue: 50): Network is down)

This is the default error that is emitted from Swift API for NWConnection when no other nwerror is present. So as long as your connection progresses you can ignore this piece. As for:

Additionally, there seems to be some weird issues with a hidden stream being created but there is no way to gain access to the handle

You might be referring to the bidirectional stream handle because there is essentially on stream for sending and one for receiving when you use

options.direction = .bidirectional

But as far as your code goes you can just save a cache of the endpoints and extract a connection / stream when needed using NWEndpoint and `extract and you should be able to take advantage of these bidirectional stream by just using the send and receive functions on the connection / stream.

@meaton thanks for having a look and the example! I think there was a misunderstanding but I realize my code was missing a piece! I am not struggling with the connection aspect or the error you mentioned. I am conflating to two issue, apologies!

  1. Creating a new stream with new options for the stream.

  2. I noticed that there appears to be a hidden stream created when we create the connection with the NWConnectionGroup which is the more complex issue.

If my initial connection is established with options specified like so:

let options =  NWProtocolQUIC.Options(alpn: ["demo"])

// Set the initial stream to unidirectional.
options.direction = .unidirectional
...
let parameters = NWParameters(quic: options)
let descriptor = NWMultiplexGroup(to: endpoint)
let group = NWConnectionGroup(with: descriptor, using: parameters)

and after establishing a connection and the group set is .ready I am trying something like.

group.stateUpdateHandler = { newState in
    switch newState {
    case .ready:
        print("Connected using QUIC!")
        let options =  NWProtocolQUIC.Options(alpn: ["demo"])
        options.direction = .bidirectional
        
        mainConn = group.extract(connectionTo: endpoint, using: options)! // force unwrap
        print("new stream made: \(mainConn)")
        ...
    case .failed(let error):
        print("main connection failed.")
        break
    }
}
...

I am seeing this this error in the console log and specifically the nw_endpoint_flow_failed_with_error and then the new stream connection is in a failed state.

Connected using QUIC!

new stream made: Optional([C3 127.0.0.1:4567 quic, attribution: developer, attach protocol listener])

nw_endpoint_flow_setup_cloned_protocols [C3 127.0.0.1:4567 in_progress socket-flow (satisfied (Path is satisfied), viable, interface: lo0)] could not find protocol to join in existing protocol stack

nw_endpoint_flow_failed_with_error [C3 127.0.0.1:4567 in_progress socket-flow (satisfied (Path is satisfied), viable, interface: lo0)] failed to clone from flow, moving directly to failed state

Main Connection State: failed(POSIXErrorCode(rawValue: 50): Network is down)

main connection failed.

quic_recovery_pto PTO fired after validation

If I remove the endpoint and options from the extract call:

mainConn = group.extract(connectionTo: endpoint, using: options)! // force unwrap

// into
mainConn = group.extract()! // force unwrap

If works but my streams is not bidirectional.

Am I able to create new streams with different directions in this API? Or do I have to choice between bidirectional or unidirectional for the entire group?

The second issue of the hidden stream is more complex and requires some understanding of the server. Effectively, the server does something like this (sudo code, original code in the repo provided):

let conn = endpoint.accept_connection()
print(
    "[server] connection accepted: addr=\(conn.remote_address())",
)
print("[server] waiting for bi stream");
let (send_end, recv_end) = conn.accept_bi()

print("[server] awaiting recv")
recv_end.recv(....)

Now if I establish a connection with options.direction = .bidirectional on the client side:

let options =  NWProtocolQUIC.Options(alpn: ["demo"])

// Set the initial stream direction.
options.direction = .bidirectional
...
let parameters = NWParameters(quic: options)
let descriptor = NWMultiplexGroup(to: endpoint)
let group = NWConnectionGroup(with: descriptor, using: parameters)

and then in the group state handler, once the connection is ready I make a new stream (same directionality):

group.stateUpdateHandler = { newState in
    switch newState {
    case .ready:
        print("Connected using QUIC!")
        let options =  NWProtocolQUIC.Options(alpn: ["demo"])
        options.direction = .bidirectional
        
        mainConn = group.extract()! // force unwrap
        print("new stream made: \(mainConn)")
        ...
    case .failed(let error):
        print("main connection failed.")
        break
    }
}
...

When sending bytes through that new connection from the swift client, on the server side I will get logs like so:

[server] connection accepted: addr=127.0.0.1:53890
[server] waiting for bi stream
[server] awaiting recv

The server will receive the connection, wait for the bidirectional stream to be opened and then await the bytes but the bytes will never come through the newly accepted stream. I can see the swift client sending bytes and in higher verbosity levels the server IS getting the bytes.

quinn_proto::connection: got Data packet (67 bytes) from 127.0.0.1:56022 using id 8c3bc319a7211707
quinn_proto::connection: got Data packet (67 bytes) from 127.0.0.1:56022 using id 8c3bc319a7211707

but the only way for me start to recving the bytes from the stream accepted on the server and opened on the swift client in the group state handler is to accept the bidirectional stream twice.

print("[server] waiting for bi stream");
let (_, _) = conn.accept_bi()
let (send_end, recv_end) = conn.accept_bi()

I have already investigated with the maintainers of the package I use that is not a weird bug on their end and in higher verbosity levels we can see TWO stream being opened. But on the swift side, as far as I can see we are only establish a single stream and that 'first' stream is not accessible from the NWConnectionGroup at all.

If I don't send bytes that the server never makes to even attempting to recv bytes but I do think this indicates that there is stream being initiated under the hood of the NWConnection and the RFC quit does outline this behavior:

[A stream ID that is used out of order results in all streams of that type with lower-numbered stream IDs also being opened.

](https://www.rfc-editor.org/rfc/rfc9000.html#name-stream-types-and-identifier)

Is there a way for me to increase logging or inspect further where this initial stream is being opened?

Am I able to create new streams with different directions in this API? Or do I have to choice between bidirectional or unidirectional for the entire group?

With different options, no. The options must be derived from the base options. What happens is that new streams are derived from the initial QUIC connection that were used to setup the tunnel, so typically those new streams would take on the initial setup options of the main tunnel (or groups) connection. And this makes sense because otherwise you would have streams that would have different connection parameters (and behavior) than the parent connection / tunnel. And if thats happens then a case could be made for an entirely new connection or tunnel.

But back to your error, if you pass in options that are different from the main tunnels options then the protocol stack will be searched and the options will not be found and so an error will be returned.

If you want to extract via an endpoint you can with something like:

mainConn = group.extract(connectionTo: endpoint)

And this should give you:

2024-01-23 06:48:19.950474-0800 0xe525e5   Debug       0x0                  52666  0    TestQUIC: (libquic.dylib) [com.apple.network:quic] quic_stream_alloc_new_id allocated new stream id 4 for bidirectional stream
  • May I query where you got that log output, I am using CFNETWORK_DIAGNOSTICS=3 on my product scheme, are you attaching via Console.app? It'll help me with my other problem which which I will take to another thread.

Add a Comment

I understand! Thanks!

To summarize:

No, the options for a connection group for a given endpoint cannot differ. If you want to mix stream directionality options cannot do so with a NWConnectionGroup as there is a constrain that for a given endpoint, new "stream" cannot redefine the options of the main group.

What happens is that new streams are derived from the initial QUIC connection that were used to setup the tunnel, so typically those new streams would take on the initial setup options of the main tunnel (or groups) connection. And this makes sense because otherwise you would have streams that would have different connection parameters (and behavior) than the parent connection / tunnel. And if thats happens then a case could be made for an entirely new connection or tunnel.

I see, that make sense from an NW API pov and the way it works under the hood, even if that is not a limitation of the QUIC protocol itself. It would be nice if we could change could established a new stream with different directionality as that is not a property of the connection socket itself but of the streams which is legal for QUIC.

By establishing a new connection that will give me the ability to change stream directionality BUT from a server perspective that will be a completely new accepted connection and that's something servers need to keep in mind that may organize operations around a single connection. It's doable about its an unfortunate limitation (feature?) of the well NWConnectionGroups with NWMultiplexGroup parameters work. I would seem to me that the the directionality of streams should not be a property of the connection group as a whole and something that can be overridden for the new streams but I am no QUIC expert :)

@meaton That seems to contradict the WWDC21 video "Accelerate networking with HTTP/3 and QUIC" where at 16:22 it shows this example...

// Adjust stream properties let options = NWrotocolQUIC.Options () options direction = unidirectional let connection = NWConnection(from: group, using: options)

Notice it's attempting to manually (?) extract a stream from the group (notice the "from: group" in the connection initializer).

I found a workaround that lets you create a mix of unidirectional/bidirectional streams through a single NWConnectionGroup, but it's a little unintuitive. Basically, I was only able to do it by mutating the same options value I used when first creating the NWConnectionGroup, meaning the code would look something like this:

// (Same setup code seen above)
let options =  NWProtocolQUIC.Options(alpn: ["demo"])
options.direction = .unidirectional

let parameters = NWParameters(quic: options)
let descriptor = NWMultiplexGroup(to: endpoint)
let group = NWConnectionGroup(with: descriptor, using: parameters)

// ...

// Modify the same options value used to create the stream
options.direction = .bidirectional
// Receive a bidirectional stream
let bidirectionalStream = group.extract(using: options)!

My current working theory is that the options argument passed to extract(using:) is compared to the original options argument partly by using the sec_protocol_options_are_equal function. If that function returns false, that's when the "could not find protocol to join in existing protocol stack" error gets returned.

I found that if you call sec_protocol_options_set_verify_block on two different sec_protocol_options_t values, then sec_protocol_options_are_equal always returns false (at least from my testing in Swift). I didn't test it, but I think the above workaround isn't necessary if you don't call sec_protocol_options_set_verify_block (I need custom TLS verification for my use-case, so the above workaround seems to be my only choice).

  • Kyle this is awesome! Good sleuthing!

Add a Comment