Network Connection Lost during POST request

Hello,

we are using AppAuth as a framework to communicate with our OAuth Server. While logging in or making a token refresh the completion handler sometimes returns with an -1005 network connection lost error. The following line in the AppAuth framework is responsible for this error: https://github.com/openid/AppAuth-iOS/blob/master/Source/AppAuthCore/OIDAuthorizationService.m#L449.

In a network proxy debugger we see that the OAuth server responds with HTTP Status code 200 but the response body is cut off (the json payload containing the JWT is only partly available, something like { "id-token": "137548 (Note the missing ending quote and curly brace. When this happens on a token refresh our OAuth server invalidates the old token because the server thinks the response was working correctly (HTTP status code 200). The next token refresh then fails because we request a token refresh with an already invalidated token and ultimately log the user out. We cannot reproduce this behavior consistently but found it to happen relatively often even when the network connection seems to be great (WLAN, LTE etc.).

We managed to capture a tcpdump while this happened which I attached in my comment below. Is there anything in there you see which might explain this behavior? This is a really serious issue for us.

I am not really an expert on this low level network stuff but looking at the tcpdump it seems to me that the client (10.131.189.183) sends the TCP RST packet to the server (35.201.94.164) closing the connection (Packet no. 6247).

Update: I also added the export from the network debugger as log.txt. There you can see the request + response in the error case.

Seems like I can't attach files? Is there any preferred way to share the pcap file? I uploaded it to my iCloud for now, here is the link: https://www.icloud.com/iclouddrive/0feIAezg8lTCHF18aqddPRSBA#trace

I also managed to reproduce this with the CFNETWORK_DIAGNOSTICS set to 3 I can share this also if this is of any help.

This is running over HTTPS, not HTTP, right? If so, a packet trace is unlikely to be helpful because everything will be encrypted within TLS.

In a network proxy debugger we see that the OAuth server responds with HTTP Status code 200 but the response body is cut off (the json payload containing the JWT is only partly available, something like { "id-token": "137548

Does the response have a Content-Length header? If so, does that reflect the correct side of the response? Or this truncated size?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

The Content-Length specifies ~3kB but only ~1kB was received at least in the case we reproduced while the network debugging proxy was attached. We also found other times the response body was empty.

It does sound like the network connection was, as the error suggests, just lost. My experience is that the vast majority of times this happens it’s because of a problem on the network side. However, proving that is quite tricky.

The general strategy is:

  1. Install the CFNetwork and Network Diagnostics profiles from our Bug Reporting > Profiles and Logs page.

  2. Start an RVI packet trace per Recording a Packet Trace.

  3. Reproduce the problem.

  4. Trigger a sysdiagnose, which captures the system log.

  5. Stop your packet trace.

  6. Look through the system log to uncover the source of the NSURLErrorNetworkConnectionLost error. You can usually do that by searching for the error number, -1005.

  7. Work back through the log to find the event that triggered that on the underlying TCP + TLS connection.

  8. Correlate that with the packet trace to see which end closed the TCP connection.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Network Connection Lost during POST request
 
 
Q