One HTTPS request is missing among others by Charles or Burp

The most common cause of requests not hitting the ‘wire’ is that they were serviced from the cache. If you control the session that issues these requests, you can use NSURLSessionTaskMetrics to confirm that one way or the other.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

No the request is not cached. It is sent to server and get get a response back. because it contains timestamp and requesting new resource, not a same URL that can be cached.

OK, then the next step is to look at an RVI packet trace to see if the iOS device made a direct connection to the server (as opposed to going through the proxy.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

Hi @eskimo, first they are HTTPS requests, and I can see the HTTPS request is sent from the debugger and I can add break points at the response handler, so IMO it's sent.

My new first question is, if my iPhone setup a proxy, can I use RVI to capture all iOS traffic on interface rvi0?

My current setup is iPhone setup a proxy, and my macOS is the proxy (Charles or Burp), and my macOS is connected to company VPN to test the requests, I used to having trouble that the rvi0 seems not able to capture iPhone's traffic.

The new second question is, what possible ways are there that the HTTPS client can ignore the proxy to send a request to server and get response? Because I can see all other requests in Charles, and only this specific request cannot be seen most of the time. Occasionally I can capture once or twice, but it's really random to me.

if my iPhone setup a proxy, can I use RVI to capture all iOS traffic on interface rvi0?

Yes. RVI works at a completely different level that the proxy support in NSURLSession.

what possible ways are there that the HTTPS client can ignore the proxy to send a request to server and get response?

There are lots of potential reasons for this, including:

• Most proxy configurations will avoid sending local requests via the proxy.

• Proxy configurations can use PAC files, which means the system runs JavaScript to decide whether to run the request through the proxy or not.

The other thing to consider is that, while the ability to use an HTTP proxy for debugging is cool and all, the proxy support exists to navigate proxies not to help with debugging. If the system detects that it can get to your server without going through the proxy, it’s well within its rights to do so.

IMPORTANT I’m not saying that’s what’s going on here — right now, I don’t have enough info to say anything for sure — rather I’m making a general point.

---

There’s four levels you can debug network requests:

• At the API level, using NSURLSessionTaskMetrics

• Within your process, using Debugging HTTPS Problems with CFNetwork Diagnostic Logging

• On the ‘wire’, using RVI

• At the proxy itself

If you run tests at each level you should be able to see where your missing requests diverge from the majority of requests that show up at your proxy.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

Hi @eskimo,

I tried CFNetwork Diagnostic Logging today, and I can see the diagnostic log, and indeed, one of the request is missing all the time. I can search and find other brother requests, but the problematic request is not showing up in the diagnostic log.

What could go wrong then? The missing request is a NSURLMutableRequest. What are the options that it can ignore the proxy? I mean I wanted to let the request through the proxy, in order to let other team to check and debug the request.

At this point I’m out of ideas. My recommendation is that you open a DTS tech support incident so that I, or my colleague Matt, can take a proper look at this.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

Sure I will consider. Thanks for you big support!