Downloading many files in the background using NSURLSession

Good day,

I have a use case that I am currently facing problems with.

I am currently implementing a downloader service that uses some c++ code to download in the foreground, and when the user backgrounds the app, NSURLSession kicks in.

Due to chunking, we have a bunch of endpoints that contain ~1mb worth of data for each chunk, and these chunks make up a full file when processed. Small files would be downloaded as is.

As such, there might potentially be ten thousands worth of endpoints that NSURLSession might need to download from in the background.

From what I've read around here, there are four things to take note of.

  1. Resume Rate Limiter
  2. Number of tasks queued into nsurlsessiond
  3. Number of concurrently running tasks
  4. NSUrlSessionDownloadTask creation

My current implementation is to create a bunch of tasks in applicationDidEnterBackground(_:) and then starting them all at once. However, NSUrlSessionDownloadTask creation is very expensive. As a reference, it takes 3.7 seconds to create 400 tasks on my iPhone 7+.

Upwards of 500 task creation and it runs into the territory of the OS terminating my app as I did not return out of applicationDidEnterBackground(_:) in five seconds.

Creation of new tasks when the first batch ends is also not an option because of resume rate limiter. According to here, the delay is really heavy and is something I would like to avoid.

Are there any viable solutions that would solve my problem of downloading many files in the background? Thanks in advance!

Answered by DTS Engineer in 703614022

We can place a cap on the number of files that are able to be downloaded in the background, but what would be a good number?

One? (-:

Seriously though, your best option is a single, large, resumable download. Everything else just takes you away from that optimum.

As to how many you can get away with, that’s hard to say. Thousands is definitely bad and tens is definitely fine, so the cut off point is somewhere in the low hundreds.

If we were to do this, are we supposed to create a NSURLSession with a background configuration right off the bat, and use that for both foreground downloads and background downloads?

Why are you’re using libcurl? My guess is that it’s adding a bunch of complexity without yielding much benefit.

As to how you manage your instances, my general advice is that you try to divide your network requests into groups based on their expected size and latency:

  • For small, interactive requests, run them in a standard session. These typically fail if your app gets suspended, but that’s not a problem because you can retry them when your app resumes.

  • For large requests, run them in a background session. These will continue if your app gets suspend.

Or is there a way that we can still use curl in the foreground, and then startup a NSURLSessionDownload to be used in the background and resume the rest of the downloads there?

I don’t really understand your question, so let’s start with some factoids:

  • libcurl has no special affordance in iOS, so it’ll only work if your app remains running.

  • By default the system suspends your app shortly after moving it to the background.

  • When that happens, your libcurl code will stop execution and an in-flight transfers will fail.

  • There’s no specific mechanism to take a partial download done by libcurl and create a NSURLSessionDownloadTask to continue the download in a background session.

  • However, if your server supports byte ranges you could probably make that work.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Have you read this already?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

with HTTPMaximumConnectionsPerHost = 5.

That’s unlikely to have any effect in a background session. The session imposes its own limits on the connection count but, even if that weren’t the case, we’re now in a world of HTTP/2 (and HTTP/3!) where this value is generally not relevant.

Is there some sort of cap on the number of queued-up tasks that nsurlsessiond can take?

There is no hard limit but my experience is that dumping thousands of tasks on a background session does not end well.

It’s weird that your server implements this chunk support for downloads. For uploads it makes sense — there’s no standard way to resume an upload (yet!) — but for downloads there’s a widely-deployed mechanism for supporting resumable downloads and so going to the trouble of implementing your own chunking support seems… well… weird, and in this case counter-productive.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Accepted Answer

We can place a cap on the number of files that are able to be downloaded in the background, but what would be a good number?

One? (-:

Seriously though, your best option is a single, large, resumable download. Everything else just takes you away from that optimum.

As to how many you can get away with, that’s hard to say. Thousands is definitely bad and tens is definitely fine, so the cut off point is somewhere in the low hundreds.

If we were to do this, are we supposed to create a NSURLSession with a background configuration right off the bat, and use that for both foreground downloads and background downloads?

Why are you’re using libcurl? My guess is that it’s adding a bunch of complexity without yielding much benefit.

As to how you manage your instances, my general advice is that you try to divide your network requests into groups based on their expected size and latency:

  • For small, interactive requests, run them in a standard session. These typically fail if your app gets suspended, but that’s not a problem because you can retry them when your app resumes.

  • For large requests, run them in a background session. These will continue if your app gets suspend.

Or is there a way that we can still use curl in the foreground, and then startup a NSURLSessionDownload to be used in the background and resume the rest of the downloads there?

I don’t really understand your question, so let’s start with some factoids:

  • libcurl has no special affordance in iOS, so it’ll only work if your app remains running.

  • By default the system suspends your app shortly after moving it to the background.

  • When that happens, your libcurl code will stop execution and an in-flight transfers will fail.

  • There’s no specific mechanism to take a partial download done by libcurl and create a NSURLSessionDownloadTask to continue the download in a background session.

  • However, if your server supports byte ranges you could probably make that work.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Also, sorry that the comments are just a slew of text

No worries. In future, I recommend that you just create a new reply. The comments box is best reserved for short comments [1]

is there a limit to the number of concurrent http requests for NSURLSession? We want to maximize the bandwidth usage as much as possible when the app is in the foreground.

And you think that running multiple requests concurrently will do that? That’s a common misconception. The goal of modern HTTP protocols (HTTP/2 and HTTP/3) is to multiplex all the requests to a specific host over a single connection [2]. So, you can dump as many HTTP requests into your NSURLSession as you like, and you’ll still only get one connection.

That’s assuming your server supports a modern version of HTTP. If you’re still using HTTP/1.1, you have much bigger problems.

Also, just to be clear, for downloads to be resumable, all the server needs are to fulfil these conditions stated in this page right?

AFAIK, yes.

We use libcurl because the code is pretty much a cross-platform SDK of sorts

HTTP networking is one place where you can benefit from moving your cross-platform abstraction up a few layers. That is, rather than putting the abstraction layer at libcurl, create your own abstraction based around “get me this HTTP resource”. You can then build platform-specific implementations that benefit from platform-specific features, falling back to a libcurl implementation on the more bare-bones platforms.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] A fact that the DevForums UI doesn’t make obvious, which is something I’m still hoping to get fixed (r. 80839588.

[2] In HTTP/2 this is TCP; in HTTP/3 it’s QUIC.

And you think that running multiple requests concurrently will do that? That’s a common misconception. The goal of modern HTTP protocols (HTTP/2 and HTTP/3) is to multiplex all the requests to a specific host over a single connection [2]. So, you can dump as many HTTP requests into your NSURLSession as you like, and you’ll still only get one connection.

So what does httpMaximumConnectionsPerHost in NSURLSession actually do for HTTP/(2 or 3)? According to the documentation,

The default value is 6.

This is used for HTTP/1.1 as anything > 6 might be perceived as a DOS attack by a server.

In HTTP/2, SETTINGS_MAX_CONCURRENT_STREAMS in the "SETTINGS" frame is used to determine the maximum number of concurrent connections that can be multiplexed. Hence, it leads me to believe that this setting actually does nothing, which some people have pointed out here.

So what does httpMaximumConnectionsPerHost in NSURLSession actually do for HTTP/(2 or 3)?

Probably nothing useful, but I can’t say that for sure without doing more research and I don’t have time for that in the context of DevForums. If you open a DTS tech support incident, either Matt or I will be able to allocate time to find a definitive answer.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Downloading many files in the background using NSURLSession
 
 
Q