Tune NSURLSessionUploadTask retry for NSBackgroundURLSession (watchOS) to limit number of requests.

The fact that it’s retrying would indicate that the client thinks the request has failed. Do you know why it might think that?

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Well, I know for sure that they're not actually failing, because they go through.

defining success as getting a 2xx response code, and failure as not getting that,

A couple guesses:

— maybe there's something wrong with the headers of the response that makes the client presume failure?

— Maybe the API isn't threadsafe? I'm doing this work from a background thread

— This is a large payload of the Recorded accelerometer data in JSON format from the watch. (not ideal, but its a requirement right now). Its sent direct from the watch.

i've got my session configured like so:

s.discretionary = NO;

s.HTTPMaximumConnectionsPerHost = 1;

s.timeoutIntervalForRequest = 60 * 4;

perhaps nsurlsessiond on watchOS relays the request to the connected iphone and starts the countdown immediately, and doesnt get a response back from the iphone in `timeoutIntervalForRequest` seconds?

Maybe the API isn't threadsafe? I'm doing this work from a background thread

That’s unlikely to be an issue. NSURLSession is, in general, thread safe.

maybe there's something wrong with the headers of the response that makes the client presume failure?

It’s possible, but it doesn’t seem very likely. In general NSURLSession only concerns itself with headers directly related to the transfer (things like

Content-Length

One thing you could do here is test the same code in a test iPhone app. If there was something wrong with how the server handles the transfer, I’d expect that to affect your test iPhone app in the same way. I recommend you run this test two ways:

• With a standard NSURLSession

• With a background NSURLSession, which handles requests in a manner very similar to those being proxied from the match

If you see problems in one test but not the other, that’d be a very useful data point.

This is a large payload …

How large?

It’s certainly possible that timeouts could be the issue. The one obvious gotcha in this space relates to the time it takes for a server to ingest an upload. The request timeout (as set by

timeoutIntervalForRequest

timeoutInterval

When you look at this traffic on the wire, how big a gap do you see between the client sending the last of the body and the server replying with the beginning of the response?

Talking about traffic on the wire, can you post an overview of how a typical ‘failed’ request (that is, one that ends up being retried) appears on the wire. Something like:

1. At time X, the client opens the TCP connection

2. At time X + n1, the client starts sending the HTTP request header

3. At time X + n2, the client starts sending the HTTP request body

4. At time X + n3, the client has finished sending the HTTP body

5. At time X + n4, the server starts sending the HTTP response header

6. At time X + n5, the server starts sending the HTTP response body

7. At time X + n6, the server has finished sending the HTTP response body

Also, do you see different behaviour if the watch does the upload itself. You can test this by turning off your phone, so the watch can’t proxy the request through the phone.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Hi I'm having the same problem as OP and this post is the only one where I can kind of get an idea of what's happening.

Like OP, I'm using a backround NSURLSession to upload files and it keeps retrying on large file uploads. I've observed that this happens when I wait for more than 60 seconds for the server to send a response back after uploading my files.

From what I understand in your post, I should change either timoutInterval (NSURLRequest) or timeoutIntervalForRequest (NSURLSessionConfiguration) to greater than 60 seconds or until the server finishes sending a response back to the client. But I've set my sesssion configuration timoutIntervalForRequest and timeoutInterval for NSURLRequest to greater than 60 seconds and the session is still trying the upload task multiple times. It seems like the background NSURLSession is ignoring the timeout setting I've set in both session configuration and the NSURLRequest level. To be clear the server is sending the correct response for every upload request, it's just that it's taking more than 60 seconds and the session is resending another upload request again.

I'm not using watchOS and I've also observed the same behaviour in iOS 8, 9 and 10.

It seems like the background NSURLSession is ignoring the timeout setting I've set in both session configuration and the NSURLRequest level.

In my limited tests today I’m seeing similar behaviour to what you’ve described, that is, NSURLSession background sessions not honouring

timeoutIntervalForRequest

timeoutInterval

You should definitely file a bug about this. It would be super helpful if you included a small test project that illustrates the issue and any other evidence you have to hand (like a packet trace).

Please post your bug number so I can add my own comments to it.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks for the confirmation! Bug number is

30553839

I wasn't able to create a sample project but I exported a packet trace from Charles Proxy.

I believe we are hitting a similar issue. I've included a summary of our issue below, and more details in radar #31740364, but am hoping you can answer:

- What was the resolution for iosmarkjim's radar #30553839?

- Any code-level changes we can make to change this behavior?

- If not, are there any mitigations?

We have a watchOS app that is using background URL Sessions and `scheduleBackgroundRefresh` to periodically schedule upload tasks to send sensor data to our servers for processing.

Everything works great in development and testing locally, however it's very clear from our server logs that some users' watch are getting into a state where they repeatedly send the same request to our server over and over, even though based on the server logs the server is responding with a 200 http status in a timely manner (less than 2 seconds).

The upload task's payload is multi-part form data, and I'm using some helper methods from Alamofire to generate the on-disk fileURL to upload, but am not using Alamofire for any actual traffic (in this code path, but it is used in other parts of the app).

The code for generating the upload task is:

try mfd.writeEncodedData(to: fileURL)

var request = try URLRequest(

url: url,

method: .post,

headers: headers

)

let uploadTask = self.backgroundSession.uploadTask(

with: request,

fromFile: fileURL

)

uploadTask.resume()

Where `url` is the server endpoint and `fileURL` is the local multipart-form-data file generated using Alamofire's MultipartFormData class (instance mfd above).

The background session is configured like so:

let backgroundConfigObject = URLSessionConfiguration.background(

withIdentifier: "WatchBackgroundRefreshManager"

)

var backgroundSession: URLSession!

var backgroundURLTask: WKURLSessionRefreshBackgroundTask? = nil

override init() {

super.init()

backgroundConfigObject.sessionSendsLaunchEvents = true

backgroundConfigObject.timeoutIntervalForRequest = 5.0 * 60.0

backgroundConfigObject.timeoutIntervalForResource = 30.0 * 60.0

backgroundSession = URLSession(

configuration: backgroundConfigObject,

delegate: self,

delegateQueue: nil

)

}

NOTE: I have run this same code on the watchOS simulator included in Xcode Version 8.2 (8C38) and the experience seems to reproduce the issue 100%, in that even though the server responds w/ a 200 my delegate methods are never called and the simulator continues to hit the server every 15 seconds until the Resource timeout is hit.

Is there a way to get a system log from the watchOS simulator that might contain debug info on the url session behavior that I can attach to the radar?

I haven't yet reproduced the issue on an actual device, even if I introduce arbitrary lag on the development server, but can confirm it's happening on real devices in the wild based on server logs (examples are in the radar).

What was the resolution for iosmarkjim's radar #30553839?

It’s still under investigation.

I’m not entirely sure whether the problem you’re seeing is directly related to 30553839. You wrote:

it's very clear from our server logs that some users' watch are getting into a state where they repeatedly send the same request to our server over and over, even though based on the server logs the server is responding with a 200 http status in a timely manner (less than 2 seconds).

Back on 14 Feb iosmarkjim indicated that they provoked the problem by deliberately delaying the server’s response until the default request timeout. However you’re saying that you’re seeing the problem even though the server is responding promptly. That looks like a very different thing to me.

I haven't yet reproduced the issue on an actual device …

That’s problematic. The simulator is good for many things but problems like this need to be investigated on a real device.

Keep in mind that watchOS does networking in one of two ways:

• If the paired iPhone is nearby, watchOS passes the request to the iPhone to do on its behalf

• If not, watchOS joins the Wi-Fi network and does the request itself

When trying to reproduce this problem have you exercised both of these cases? If not, you should, because it might be that the problem only reproduces in one of them.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Interesting, I've not tried changing the pairing status to reproduce the issue, I'll give that a shot.

So, when the paried iPhone is available, does that mean that the URLSession is run on the phone, or just that the Watch uses the phone as a network proxy? In short, what I think I'm asking is "which nsurlsessiond 'owns' the background upload task, the one on the Watch or on the iPhone?"

The reason I ask is, what happens in this scenario?

- The Watch starts out paired (Swipe up on Watch shows Connected)

- The background URL session and upload Task are created

- The Watch gets disconnected from the Phone (Swipe up shows either Cloud or Disconnected)

If the iPhone "owns" the task, what happens when the iPhone completes the task but the Watch isn't available? Is it possible the iPhone's nsurlsessiond gets into a state where rather than detecting that the Watch isn't available to finalize the request it gets into a retry loop?

If the Watch "owns" the task, what happens if the connection between the Watch and iPhone is "poor" (e.g. far away so high packet loss or other issue)? Will a different network path be chosen each time the background upload task is triggered by nsurlsessiond, or will the first path chosen always be used?

If think to see the issue I'm seeing, the upload task "owner" needs to be able to reach our server, but not be able to deliver the result to the Watch.

I'll try various toggles of the iPhone/Watch pairing to see if I can get into this state. Is there a way to simulate a poor connection between Watch and iPhone without physically moving the devices around?

Is there any way from inspecting the HTTP requests to tell if they are originating from the Phone or Watch? The User Agent is always for the Watch extension, but perhaps there's another indicator (another HTTP field, source port pattern, etc.)?

So, when the paried iPhone is available, does that mean that the URLSession is run on the phone, or just that the Watch uses the phone as a network proxy?

The answer to this depends on how you define the terms. Clearly your watch extension has an

URLSession

nsurlsessiond

nsurlsessiond

nsurlsessiond

Is there a way to simulate a poor connection between Watch and iPhone without physically moving the devices around?

Not that I’m aware of. It is, however, easy to simulate no connection; just put the phone in Airplane Mode (make sure to turn off the setting that mirrors this to the watch).

Is there any way from inspecting the HTTP requests to tell if they are originating from the Phone or Watch?

There’s an obvious answer to this (turn off WWAN on the phone, so everything goes over Wi-Fi, then run a packet trace on the Wi-Fi), but I think you’re really asking whether you can tell this by looking at server logs. It seems like the source IP address would be your best option.

As to your other questions, I don’t have definitive answers for you there (that is, any responses would be mere guesses). You might consider opening a DTS tech support incident here, which would allow me to allocate time to research this properly.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks Quinn,

I'll file a DTS here shortly.

Ryan

Hey, I just reproduced the issue once!

Here's how I did it:

- Added code to my Watch app to trigger the background upload 5 seconds after doing a force touch on the app

- Forced touched the app, then immediately backgrounded it

- Quickly set my phone to Airplane mode

- The Watch sent two requests to my local dev server, with the same multi-part form boundary (which is randomly generated per request, so is only repeated if the same URLRequest was sent twice).

Interestingly, the two requests were only sent within a couple seconds of one another:

{ "CONTENT_TYPE": "multipart/form-data; boundary=alamofire.boundary.d7f16bae86119fbb" }

[27/Apr/2017 22:54:32] "POST /resting/upload HTTP/1.1" 200 24210

{ "CONTENT_TYPE": "multipart/form-data; boundary=alamofire.boundary.d7f16bae86119fbb" }

[27/Apr/2017 22:54:34] "POST /resting/upload HTTP/1.1" 200 24210

[As mentioned in the radar, I'm using alamofire to generate the multi-part form document, but not using it for sending the request, that's using the code I posed above]

It's only two repeats versus the sometimes hundreds we've seen in server logs from real users, but it's a start. If I start with the phone in airplane mode it doesn't seem to happen, so far I've only reproduced it by switching the phone to airplane mode just before the background request should fire.

Ryan

We have the exact same issue as this. Has there been any resolution to 30553839? The problem was reported over a year ago now.

To be clear, iOS 11 retries our NSURLSessionUploadTasks about every 70 seconds continuously for as long as timeoutIntervalForResource is configured (7 days by default). iOS retries because our server does not respond until after the 70 seconds are elapsed but then that response is ignored by iOS presumably because a new attempt has already been kicked off. I can find no hook nor configuration that governs this 70 second timeout used internally by iOS.

I understand why iOS is continually retrying given the nature of how "fire and forget" background uploads/downloads are supposed to work, and why the default for timeoutIntervalForResource is set a high as 7 days. But it's a serious problem when uploads are completing on the server but the responses are being ignored. We end up in an loop that repeatedly uploads the same file every 70 seconds for an entire week.

Is there any news on this?

We have a cache job, which takes some time to return based on the size of the cache, and anything over a minute causes my iOS background NSURLSession to fire didSendBodyData delegate method AGAIN AND AGAIN to which shortly after the server receives a new request to cache.

Kind of stuffed. Also can't find how to track the bug mentioned on here.

But I am simply using an iPhone and simple restful api. No watchOS or anything.