I found a workaround that lets you create a mix of unidirectional/bidirectional streams through a single NWConnectionGroup, but it's a little unintuitive. Basically, I was only able to do it by mutating the same options value I used when first creating the NWConnectionGroup, meaning the code would look something like this:
// (Same setup code seen above)
let options = NWProtocolQUIC.Options(alpn: ["demo"])
options.direction = .unidirectional
let parameters = NWParameters(quic: options)
let descriptor = NWMultiplexGroup(to: endpoint)
let group = NWConnectionGroup(with: descriptor, using: parameters)
// ...
// Modify the same options value used to create the stream
options.direction = .bidirectional
// Receive a bidirectional stream
let bidirectionalStream = group.extract(using: options)!
My current working theory is that the options argument passed to extract(using:) is compared to the original options argument partly by using the sec_protocol_options_are_equal function. If that function returns false, that's when the "could not find protocol to join in existing protocol stack" error gets returned.
I found that if you call sec_protocol_options_set_verify_block on two different sec_protocol_options_t values, then sec_protocol_options_are_equal always returns false (at least from my testing in Swift). I didn't test it, but I think the above workaround isn't necessary if you don't call sec_protocol_options_set_verify_block (I need custom TLS verification for my use-case, so the above workaround seems to be my only choice).