We are building a macOS client where we make a web socket connection to our server using NWConnection. The code to to create NWParameters is:
let options = NWProtocolTLS.Options()
let securityProtocolOptions = options.securityProtocolOptions
//....configure security options for server and client cert validation...
let parameters = NWParameters(tls: options)
let wsOptions = NWProtocolWebSocket.Options()
wsOptions.autoReplyPing = true
wsOptions.setAdditionalHeaders(additionalHeaders.map { ($0.key, $0.value) } )
parameters.defaultProtocolStack.applicationProtocols.insert(options, at: 0)
With these parameters and an NWEndpoint object, we create a connection which connects and transfers data well from both sides. However, whenever server gracefully closes the connection, the client remains oblivious to this and does not close the connection. It only detects a timeout when trying to send some data over the connection after the server disconnect. We looked into Wireshark and we do not see any FIN,ACK or RST packets being received from server.
However, in our windows client, when same exact server closes the connection, we are seeing FIN, ACK and connection immediately closes on client side as well.
We also tried to test same behaviour in golang with a small snippet created by our team member running on Mac itself. This snippet also receives FIN,ACK from server and closes the connection immediately. Only NWConnection in our Mac client does not receive close connection.
So, the question arises, why is NWConnection not receiving FIN,ACK and not closing the connection when a windows as well as a golang client on Mac is able to. Is there any extra configuration required for NWConnection or NWParameter? Is the NWParameter creation code correct? We already checked and we are continuously calling receiveMessage on the NWConnection object. So, missing read is not the issue here.
Also, I do not see any connection timeout option in NWProtocolWebSocket but it exists in NWProtocolTCP. So, is there a way to set connection timeout for web socket connection using NWConnection?
Finally I was able to solve this problem. Here are few things we learned and fixed:
-
At first we were not receiving the FIN,ACK message from server when connection was closed from server side. When we looked at server packet capture, we could see that server was sending FIN,ACK but was not receiving ACK from client. So, server was retransmitting FIN,ACK. Similarly, client was not aware of server closing the connection and when trying to send something to server was also not being ACKed by server and was being retransmitted. This was happening after 5 or more minutes of idle. We narrowed it down to a NAT issue where NAT device was losing the mapping of client IP after certain period of idle connection. So, after losing the mapping, both server and client were not able to reach each other and were in re-transmission. We fixed this by sending TCP keep-alive every one minute during idle connection.
-
After fixing that we started getting FIN,ACK from server but were not able to receive it in NWConnection callback. SSL_shutdown from server closes the write direction and sends a FIN,ACK to client. On client side, this is indicates by
isFinalproperty ofNWConnection.ContentContextreceived inreceiveMessagecompletion handler. Our mistake was that we were expecting either data or error in the completion handler to be non-nil. Otherwise, we were discarding the call-back. But when FIN,ACK is sent by server, error and data, both are nil andisFinalin context is set. This gives the hint that server has closed the write channel and client side should also send any remaining message and close the connection. Once we implemented that, we were able to fix complete issue.
Thanks @eskimo for your help and suggestions. Let me know if you see any issue with above fixes or any improvements that can be made.