How to write a NWProtocolFramer for Network.framework that splits streams into frames using a delimiter?

I tried to create a framer that splits a stream of ASCII bytes into frames separated by the pipe ascii character: "|".

So I made an NWProtocolFramerImplementation as seen in the attached document.

The problem is that from the moment I get a chunk that does not end with "|", the framer gets stuck on that chunk. So the other chunks that come after this incomplete chunk never fully arrive in the framer.parseInput(...) call. Because it always parses chunks of minimumIncompleteLength and hence never arrives to the point where the next "|" is.

Here is a simple reproduction of this problem:
  1. Create a TCP server

  2. Setup the server so that it sends chunks of messages when a client connects.

  3. Connect to the server (created in 1.) using the framer from above.

  4. Start receiving messages.

Code Block
import Network
let client = DispatchQueue(label: "Server")
let server = DispatchQueue(label: "Client")
let networkParameters = NWParameters.tcp
networkParameters.defaultProtocolStack.applicationProtocols.insert(NWProtocolFramer.Options(definition: PipeFramer.definition), at: 0)
let server = try! NWListener(using: .tcp)
server.newConnectionHandler = { connection in
print("server: new connection from", connection.endpoint)
print("server (client \(connection.endpoint)): state", connection.state)
connection.viabilityUpdateHandler = { viable in
print("server (client \(connection.endpoint)): state", connection.state)
if viable {
print("server: sending")
connection.send(content: "A|Be||Sea".data(using: .utf8)!, isComplete: false, completion: .idempotent)
serverQueue.asyncAfter(deadline: .now() + 5) {
print("server: sending second part")
connection.send(content: " is longer than expected|0|".data(using: .utf8)!, isComplete: true, completion: .idempotent)
}
serverQueue.asyncAfter(deadline: .now() + 8) {
print("server: sending last part")
connection.send(content: "Done|".data(using: .utf8)!, isComplete: true, completion: .idempotent)
}
}
}
connection.start(queue: serverQueue)
}
server.stateUpdateHandler = { state in
print("server:", state)
if state == .ready, let port = server.port {
print("server: listening on", port)
}
}
server.start(queue: serverQueue)
let client = NWConnection(to: .hostPort(host: "localhost", port: server.port!), using: networkParameters)
func receiveNext() {
client.receiveMessage { (data, context, complete, error) in
let content: String
if let data = data {
content = String(data: data, encoding: .utf8) ?? data.description
} else {
content = data?.debugDescription ?? "<no data>"
}
print("client: received \"\(content)\"", context.debugDescription, complete, error?.localizedDescription ?? "No error")
receiveNext()
}
}
client.stateUpdateHandler = { state in
print("client:", state)
if state == .ready {
print("client: receiving")
receiveNext()
}
}
client.start(queue: clientQueue)

Results in:
Code Block
server: waiting(POSIXErrorCode: Network is down)
server: ready
server: listening on 54894
client: preparing
client: ready
client: receiving
server: new connection from ::1.53179
server (client ::1.53179): state setup
server (client ::1.53179): state ready
server: sending
client: parsing buffer: "A|BeSea"
client: minLength set to 1
client: parsing buffer: "BeSea"
client: minLength set to 1
client: parsing buffer: "|Sea"
client: minLength set to 1
client: parsing buffer: "Sea"
client: minLength set to 4
client: parsing buffer: ""
client: minLength set to 1
client: received "A" Optional(Network.NWConnection.ContentContext) true No error
client: received "Be" Optional(Network.NWConnection.ContentContext) true No error
client: received "<no data>" Optional(Network.NWConnection.ContentContext) true No error
client: parsing buffer: "Sea"
client: minLength set to 4
server: sending second part
client: parsing buffer: "Sea "
client: minLength set to 5
client: parsing buffer: "Sea i"
client: minLength set to 6
server: sending last part
client: parsing buffer: "Sea is"
client: minLength set to 7
client: parsing buffer: "Sea is "
client: minLength set to 8


Notice that the fourth and fifth message are never received by the client. How should I write the Framer so that it receives messages after an incoming incomplete chunk?
Yep, this can be difficult because you will need to save previously read data so that it can fit into the next frame. Take a look at a TCP / UDP framing implementation I wrote here on this post that was based off the framer in the TicTacToe example. This would be a great place to start with this.


Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com
Thanks for the quick response Matt. I looked at your other post but that is demonstrating a TLV (type, length, value) framer. I was talking about separating frames using a separation character (or delimiter).

you will need to save previously read data so that it can fit into the next frame

Hmm... Why should we be saving the old chunk's data while it is still available in NWProtocolFramer.Instance's parseInput(...) function. The old data (which you refer to as previously read data) is not lost (and I don't think it is marked read until you call deliverInput...(...) or return its length in parseInput(...)). The problem I am experiencing, is that the framer does not get the new data (but is stuck on the old data).

I made small GitHub repo where you can debug the problem: https://github.com/Dev1an/Simple-Framer

Could you explain how your other posts (TicTacToe & TLV) are relevant to this delimiting problem (where the message lengths are not known in advance)?

Could you explain how your other posts (TicTacToe & TLV) are relevant to this delimiting problem (where the message lengths are not known in advance)?

These are examples of how to handle frame parsing. As you mentioned you situation is unique because you do not know that length of the data you need to read and this can create some tricky situations. For more detailed help on this please open a TSI and I would be happy to help dig in.

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Did you ever find a solution to this? I am stuck with the same issue, where an incomplete ASCII sentence stalls the receiving client, and keep trying to pass the same data over and over again, until the connection is finally canceled.

Any help or hint would be greatly appreciate!!

How to write a NWProtocolFramer for Network.framework that splits streams into frames using a delimiter?
 
 
Q