-
Re: Background uploads with NSURLSession
eskimo Jul 16, 2015 1:27 AM (in response to pictarine)Do you recommend me to use NSURLSessionUploadTask or another class?
NSURLSessionUploadTask
What do I have to do to continue all these uploads in background?
What do I have to do to continue all these uploads when app is not running?
Dump all the uploads into an NSURLSession background session.
For all of the first 3 points, do you recommend me to wrap NSURLSession objects in NSOperation ones in order to use an NSOperationQueue? Or iOS would use its own internal queue to properly handle NSURLSession objects?
That's hard to say. In general I'm a big fan of NSOperation but it's a tricky fit for NSURLSession background sessions because of the way that the background session interacts with your app's lifecycle. Specifically, a common sequence is this:
your app starts a bunch of background transfers
the user moves it into the background
the system suspends your app
the transfers continue in the background
the system terminates your app
the transfers continue in the background
the transfers complete
the system relaunches your app
If you're using an NSOperation to track each transfer, you have to reconstitute those operations when you're relaunched at step 8. Of course, you have to reconstitute something at step 8, it's just that reconstituting an NSOperation may be trickier than reconstituting some less complex object.
Share and Enjoy
—
Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardwarelet myEmail = "eskimo" + "1" + "@apple.com"
-
Re: Background uploads with NSURLSession
pictarine Jul 16, 2015 5:03 AM (in response to eskimo)Thanks Quinn!
I understand your point concerning NSOperationQueue. I guess iOS has its own queuing system for network requests...
But, are you saying that if I add IN ONLY ONE TIME 200 NSURLSessionUploadTask instances in the same NSURLSession configured with a background session configuration, iOS will propertly handle all of them?
If yes, is there any limit in term of number of tasks?
Thank you
Best
-
Re: Background uploads with NSURLSession
eskimo Jul 17, 2015 12:51 AM (in response to pictarine)But, are you saying that if I add IN ONLY ONE TIME 200 NSURLSessionUploadTask instances in the same NSURLSession configured with a background session configuration, iOS will propertly handle all of them?
Yes. The NSURLSession background download system will happily deal with a few hundred requests. It serialises them internally, so only a few of those requests hit the 'wire' at any time.
If yes, is there any limit in term of number of tasks?
There is no hard limit but I generally recommend that you avoid pushing things too far. IMO hundreds of requests are fine, thousands of requests are pushing things, tens of thousands of requests would be silly.
I generally recommend that, if you have to deal with thousands of individual items, you zip them up and transfer them as one resumable transfer. There are, however, some gotcha there:
o Doing this sort of thing requires sophisticated server-side support.
o NSURLSession background sessions automatically handle resumable downloads (if the server supports it). That's not true for uploads. If you want to implement a resumable upload, you'll have to get involved each time the connection 'tears'. For that reason, it makes sense to chunk your uploads into reasonably sized chunks, so the upload get make a bunch of progress, even in the presence of connection failures, before your app has to resume again.
Share and Enjoy
—
Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardwarelet myEmail = "eskimo" + "1" + "@apple.com"
-
Re: Background uploads with NSURLSession
literatemonk Jan 17, 2017 11:38 PM (in response to eskimo)Hey Quinn -- I've been reading a lot of your posts for the past hour trying to understand some gotcha's I'm facing with my app. Basically I'm trying to upload ~30 large files in the background, when the user's phone is plugged in and connected to wifi. I wrote a file manager that is dispatching these one-by-one, but from what I'm reading it looks like the better approach would be to send them all to NSURLSession at once.
Given this, I want to run a completion handler when all the files are finished uploading. I read your post about the resume rate limiter. Since the resume time backs off exponentially, I'm worried that by the time the 30th file finishes uploading, I won't be able to resume my app to handle the upload's completion handler. Since I have no clue which file is going to finish first, I can't pass nil as the completion handler for 29 and an actual completion handler for the 30th. Can you provide any insight?
-
Re: Background uploads with NSURLSession
eskimo Jan 18, 2017 1:24 AM (in response to literatemonk)In general an NSURLSession background session won’t resume (or relaunch) an app in the background until all of the requests in that session have completed. So, if you dump all 30 requests into a session and then get put in the background by the user, you’ll resume exactly once, when all 30 requests are complete.
When that happens you’ll see a sequence of events like this:
App resumes (or relaunches)
In the relaunch case you must remember to re-create your background session with the same identifier that you used originally. A common mistake is not re-creating the background session in the relaunch case. The best approach in most cases is to create that session in code called from
-application:willFinishLaunchingWithOptions:
.System calls
-application:handleEventsForBackgroundURLSession:completionHandler:
, passing it a completion handlerNSURLSession calls
-URLSession:task:didCompleteWithError:
for each task that’s completeNSURLSession calls
-URLSessionDidFinishEventsForBackgroundURLSession:
, at which point it’s appropriate to call the completion handler from step 2
IMPORTANT You can’t use NSURLSession’s convenience APIs, things like
-uploadTaskWithRequest:fromFile:completionHandler:
, in a background session. You must use the delegate-based API, for example,-uploadTaskWithRequest:fromFile:
. This makes sense when you think about it; if your app is terminated and then relaunched, there’s no way for the system to reconstitute the state held in a completion handler block.Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardwarelet myEmail = "eskimo" + "1" + "@apple.com"
-
Re: Background uploads with NSURLSession
literatemonk Jan 18, 2017 7:45 AM (in response to eskimo)Thanks for the help. I'm currently using the S3 transfer utility library (https://cocoapods.org/pods/AWSS3) so I'll have to figure out what that's doing under the hood. I may need to write my own implementation based on signed URLs. Will report back when I find out.
-
Re: Background uploads with NSURLSession
Sreenivas34 Jan 26, 2017 2:01 PM (in response to eskimo)Hi,
I'm working on a similar iOS project, where I need to upload a very large video file (2 GB) to S3.
I tried setting up a single upload request to handle the entire thing, but as you pointed out, when there is a connection failure or when the user terminates the app, the upload transfer fails and I'd have to start over again.
If I were to split the 2 GB file into smaller 25 mb portions, I could add them all to my NSURLSession object to upload.
However, if the app is terminated and relaunched, and I retrieve the NSURLSession object, can I resume all of the pending upload requests? Does it have to upload all of the 25 mb files again or can it tell which ones have already been uploaded and only do the ones that haven't?
Thanks,
-Sreeni
-
Re: Background uploads with NSURLSession
eskimo Jan 27, 2017 1:17 AM (in response to Sreenivas34)However, if the app is terminated and relaunched, and I retrieve the NSURLSession object, can I resume all of the pending upload requests?
By “terminated”, do you mean:
“Removed from the multitasking UI”, so that all of your uploads get cancelled?
Terminated by the OS due to memory pressure, so that all of your uploads continue in the background?
Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardwarelet myEmail = "eskimo" + "1" + "@apple.com"
-
Re: Background uploads with NSURLSession
Sreenivas34 Jan 27, 2017 10:46 AM (in response to eskimo)Hi,
By terminated, I mean “Removed from the multitasking UI”. Given that the app resumes (or relaunches) only when all tasks in the session complete, is there no way of keeping track of which files have been become uploaded and which are still pending? If all of the uploads become cancelled when a user force quits, then I'm guessing I can't retrieve the tasks that haven't completed (and were cancelled) and only resume those.
Is the best way to approach uploading a very large file to do a multipart upload with appropriate server side support so the upload can be resumed?
Thanks,
-Sreeni
-
Re: Background uploads with NSURLSession
eskimo Jan 29, 2017 3:33 PM (in response to Sreenivas34)Given that the app resumes (or relaunches) only when all tasks in the session complete …
To be clear, if the user removes your app from the multitasking UI then you won’t be resumed or relaunched in the background. The user will have to manually relaunch your app.
The behaviour for tasks that haven’t completed when the user removes the app from the multitasking UI varies by OS:
Originally the entire session would just disappear; it would be like you’d never issued those tasks
On current systems the tasks fail with
NSURLErrorCancelled
I believe the cutover was with iOS 8, that is, iOS 7 have the first behaviour and iOS 8 and later have the second, although I’ve never sat down to check that.
I’ve also never looked at the behaviour for tasks that have completed. Based on my understanding of how things fit together I suspect they’ll also get
NSURLErrorCancelled
, but you’d have to test this.Regardless, the best way to recover from this situation is to talk to the server to see which of the uploads completed successfully. This is generally pretty easy to do via the
HEAD
HTTP method..Is the best way to approach uploading a very large file to do a multipart upload with appropriate server side support so the upload can be resumed?
That’s true for downloads but not for uploads. There’s actually a couple of problems doing a single, resumable upload:
With downloads, NSURLSession can automatically resume a failed download using HTTP standard technology (QA1761 gives an outline of how it works). For uploads there’s no way for NSURLSession to automatically restart the transfer, so it has to relaunch or resume your app to do the job. So, if you upload as a single file and the upload fails a bunch of times, you eventually provoke the ire of the resume rate limiter.
There’s no way to tell NSURLSession to start a resume from some offset in a file. So if you have a large upload that fails, you have to make a copy of your upload file to remove the bytes at the front that have successfully been uploaded.
The alternative is to segment the file into relatively large chunks and upload those. This helps with both of these problems, at the cost of some code complexity.
Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardwarelet myEmail = "eskimo" + "1" + "@apple.com"
-
Re: Background uploads with NSURLSession
Sreenivas34 Feb 1, 2017 9:36 AM (in response to eskimo)Thanks for the help! I think I know how to tackle the issue now.
-
Re: Background uploads with NSURLSession
Sreenivas34 Mar 15, 2017 8:40 PM (in response to eskimo)Hi,
I have two more questions, if you can offer any advice:
1. If I begin the transfers in the background (call the [uploadtask resume] while the app is in the background), and the discretionary property is automatically set to YES, what exactly does this mean? Does the phone have to be on wifi AND plugged in for it transfer or just be on wifi? What if I bring the app back to the foreground, is the discretionary property for these transfers set back to NO (given that session.discretionary = NO when I setup the background session)?
2. I setup and initiate around a 100 upload tasks to a NSURLSession object. If I lose network connectivity and regain it a while later (say 1 hour), does the session automatically resume all pending upload tasks? I'm only allowing uploads over wifi. I've been seeing the following strange behavior:
1. I initate 100 upload tasks in a single NSURLSession
2. 10 uploads were successfully completed
3. I lose wifi connection
4. I gain wifi connection an hour later
5. The session would complete another 15 upload tasks
6. The session stops uploading anymore data (the URLSession didSendBodyData is no longer being called, so I'm guessing all of the tasks got cancelled somehow).
Any help would be greatly appreciated,
Thanks,
-Sreeni
-
Re: Background uploads with NSURLSession
eskimo Mar 16, 2017 5:17 AM (in response to Sreenivas34)If I begin the transfers in the background (call the [uploadtask resume] while the app is in the background), and the discretionary property is automatically set to YES, what exactly does this mean?
To start, understand that the “discretionary property” isn’t an actual property, at least not one that you can see. Rather, it’s a value computer by NSURLSession based on a number of criteria, including:
The type of session (standard vs background)
The
discretionary
property of the configuration used to create that sessionWhether the app was in the foreground when the task was created
Whether the app is in the foreground right now
Moreover, the specific effects of this ‘property’ are not guaranteed; they have changed in the past and I fully expect them to change in the future.
Does the phone have to be on wifi AND plugged in for it transfer or just be on wifi?
In general, yes. However, it’s really up to the OS as to when it schedules discretionary tasks, and it’s possible that, for example, if the device ‘learns’ that it’s never on Wi-Fi, it may chose to run discretionary tasks over WWAN.
What if I bring the app back to the foreground, is the discretionary property for these transfers set back to NO (given that session.discretionary = NO when I setup the background session)?
Again, no property values change here but, yes, modern versions of iOS include the foreground state of the originating app when determining whether to run a discretionary request.
I setup and initiate around a 100 upload tasks to a NSURLSession object. If I lose network connectivity and regain it a while later (say 1 hour), does the session automatically resume all pending upload tasks?
Again, the exact behaviour is not specified and depends on a bunch of things. But, yes, I would expect these tasks to eventually complete.
(the URLSession didSendBodyData is no longer being called, so I'm guessing all of the tasks got cancelled somehow)
You shouldn’t need to guess here. If a task gets cancelled you should be told about it (via the standard task completion mechanism). You can also confirm whether a task is still around using
-getAllTasksWithCompletionHandler:
.I expect what you’ll find is that the remaining tasks are still around, they’re just not running right now for some reason (lack of Wi-Fi, lack of power, resource budgets, and so on).
Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardwarelet myEmail = "eskimo" + "1" + "@apple.com"
-
-
-
-
-
-
-
Re: Background uploads with NSURLSession
rmpangahds Jun 14, 2019 8:58 AM (in response to pictarine)Hi,
This thread has helped me a lot but i am still seeing some strange behavior when I try to do background upload tasks The problem I am seeing occurs only for large file uploads (300-500 MBs). What I am observing is that iOS sometimes does NOT call didCompleteWithError even after my server sends a response. In the client logs I can see that the didSendBodyData method gets called several times. Which is expected, however it does not seem to stop even after sending all the bytes. I have also noticed that sometimes the taskId changes or the totalBytesSent gets reset. Which does not make any sense since I have not queued up another upload task. I only called resume once.
Also note: This behavior happens when the app is in the foreground. I have not attempted testing background uploads yet.
Everythin works as expected the file is small.(~50 Mbs). The didRecieveData method gets called along with the didCompleteWithError method.
Below are some code snippets of the background task:
//FileOperationsDelegate.mm - (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task didSendBodyData:(int64_t)bytesSent totalBytesSent:(int64_t)totalBytesSent totalBytesExpectedToSend:(int64_t)totalBytesExpectedToSend { NSNumber* taskId = [NSNumber numberWithUnsignedInteger:task.taskIdentifier]; AwLog(@"DidSendBodyData for taskID: %@ BytesSent: %lld TotalBytesSent: %lld ExpectedToSend: %lld\n", taskId, bytesSent, totalBytesSent, totalBytesExpectedToSend); if (updateFileProgressCallback != NULL) { updateFileProgressCallback(bytesSent, totalBytesSent); } }
Here is by backgroundSession.
//FileOperationsDelegate.mm - (NSURLSession *)backgroundSession { static NSURLSession *session = nil; static dispatch_once_t onceToken; dispatch_once(&onceToken, ^{ NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration backgroundSessionConfigurationWithIdentifier: KEY_BACKGROUND_SESSSION_ID]; configuration.discretionary = YES; configuration.sessionSendsLaunchEvents = YES; configuration.URLCache = nil; bool wifiOnly = [ClientConfiguration.sharedInstance isWifiOnlyEnabled]; if (wifiOnly) { configuration.allowsCellularAccess = false; } NSOperationQueue* delegateMainQueue = [NSOperationQueue mainQueue]; delegateMainQueue.maxConcurrentOperationCount = 1; session = [NSURLSession sessionWithConfiguration:configuration delegate:self delegateQueue: delegateMainQueue]; }); return session; }
Creating the FileUpload task
//FileUploader.mm NSURLSession* backgroundSession = [FileOperationsSessionDelegate.sharedInstance backgroundSession]; NSMutableString *file = [[NSMutableString alloc] initWithString:absLocalPath]; [file insertString:@"file://" atIndex:0]; NSURL* fileToBeUploaded = [NSURL URLWithString:file]; NSURLSessionUploadTask* uploadTask = [backgroundSession uploadTaskWithRequest:urlRequest fromFile:fileToBeUploaded]; uploadTask.countOfBytesClientExpectsToSend = self.uploadSizeBytes; uploadTask.countOfBytesClientExpectsToReceive = 450; ConnectionCallback uploadCompleteCallback = ^(NSURLResponse * response, NSData * data, NSError * error) { // A callback to complete the file upload. It will update the UI and notify the user // on how the upload went. } BytesUploadedCallback updateFileProgressCallback = ^(long bytesWritten, long totalBytesWritten) // A callback that will update the progress bar. }; [FileOperationsSessionDelegate.sharedInstance setUploadCallback:uploadCompleteCallback]; [FileOperationsSessionDelegate.sharedInstance setConnectionCallback:updateFileProgressCallback];; [uploadTask resume];
Any help with this will be greatly apperiated.
Edit: I have attempted using a non background session and I observe the same behavior.
Thanks : )
Specs:
iOS version 12.3
Device: iPad Pro 11
Xcode: 10.1