13 Replies
      Latest reply: Mar 16, 2017 5:17 AM by eskimo RSS
      pictarine Level 1 Level 1 (0 points)

        Hi,

         

        I need to upload up to 200 images in my app:

         

        1. Do you recommend me to use NSURLSessionUploadTask or another class?
        2. What do I have to do to continue all these uploads in background?
        3. What do I have to do to continue all these uploads when app is not running?
        4. For all of the first 3 points, do you recommend me to wrap NSURLSession objects in NSOperation ones in order to use an NSOperationQueue? Or iOS would use its own internal queue to properly handle NSURLSession objects?

         

        Thank you!

         

        Best Regards

        • Re: Background uploads with NSURLSession
          eskimo Apple Staff Apple Staff (6,665 points)

          Do you recommend me to use NSURLSessionUploadTask or another class?

          NSURLSessionUploadTask

          What do I have to do to continue all these uploads in background?

          What do I have to do to continue all these uploads when app is not running?

          Dump all the uploads into an NSURLSession background session.

          For all of the first 3 points, do you recommend me to wrap NSURLSession objects in NSOperation ones in order to use an NSOperationQueue? Or iOS would use its own internal queue to properly handle NSURLSession objects?

          That's hard to say.  In general I'm a big fan of NSOperation but it's a tricky fit for NSURLSession background sessions because of the way that the background session interacts with your app's lifecycle.  Specifically, a common sequence is this:

          1. your app starts a bunch of background transfers

          2. the user moves it into the background

          3. the system suspends your app

          4. the transfers continue in the background

          5. the system terminates your app

          6. the transfers continue in the background

          7. the transfers complete

          8. the system relaunches your app

          If you're using an NSOperation to track each transfer, you have to reconstitute those operations when you're relaunched at step 8.  Of course, you have to reconstitute something at step 8, it's just that reconstituting an NSOperation may be trickier than reconstituting some less complex object.

          Share and Enjoy

          Quinn "The Eskimo!"
          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
          let myEmail = "eskimo" + "1" + "@apple.com"

            • Re: Background uploads with NSURLSession
              pictarine Level 1 Level 1 (0 points)

              Thanks Quinn!

               

              I understand your point concerning NSOperationQueue. I guess iOS has its own queuing system for network requests...

               

              But, are you saying that if I add IN ONLY ONE TIME 200 NSURLSessionUploadTask instances in the same NSURLSession configured with a background session configuration, iOS will propertly handle all of them?

               

              If yes, is there any limit in term of number of tasks?

               

              Thank you

               

              Best

                • Re: Background uploads with NSURLSession
                  eskimo Apple Staff Apple Staff (6,665 points)

                  But, are you saying that if I add IN ONLY ONE TIME 200 NSURLSessionUploadTask instances in the same NSURLSession configured with a background session configuration, iOS will propertly handle all of them?

                  Yes.  The NSURLSession background download system will happily deal with a few hundred requests.  It serialises them internally, so only a few of those requests hit the 'wire' at any time.

                  If yes, is there any limit in term of number of tasks?

                  There is no hard limit but I generally recommend that you avoid pushing things too far.  IMO hundreds of requests are fine, thousands of requests are pushing things, tens of thousands of requests would be silly.

                  I generally recommend that, if you have to deal with thousands of individual items, you zip them up and transfer them as one resumable transfer.  There are, however, some gotcha there:

                  o Doing this sort of thing requires sophisticated server-side support.

                  o NSURLSession background sessions automatically handle resumable downloads (if the server supports it).  That's not true for uploads.  If you want to implement a resumable upload, you'll have to get involved each time the connection 'tears'.  For that reason, it makes sense to chunk your uploads into reasonably sized chunks, so the upload get make a bunch of progress, even in the presence of connection failures, before your app has to resume again.

                  Share and Enjoy

                  Quinn "The Eskimo!"
                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                  let myEmail = "eskimo" + "1" + "@apple.com"

                    • Re: Background uploads with NSURLSession
                      literatemonk Level 1 Level 1 (0 points)

                      Hey Quinn -- I've been reading a lot of your posts for the past hour trying to understand some gotcha's I'm facing with my app.  Basically I'm trying to upload ~30 large files in the background, when the user's phone is plugged in and connected to wifi.  I wrote a file manager that is dispatching these one-by-one, but from what I'm reading it looks like the better approach would be to send them all to NSURLSession at once.

                       

                      Given this, I want to run a completion handler when all the files are finished uploading. I read your post about the resume rate limiter.  Since the resume time backs off exponentially, I'm worried that by the time the 30th file finishes uploading, I won't be able to resume my app to handle the upload's completion handler.  Since I have no clue which file is going to finish first, I can't pass nil as the completion handler for 29 and an actual completion handler for the 30th.  Can you provide any insight?

                        • Re: Background uploads with NSURLSession
                          eskimo Apple Staff Apple Staff (6,665 points)

                          In general an NSURLSession background session won’t resume (or relaunch) an app in the background until all of the requests in that session have completed.  So, if you dump all 30 requests into a session and then get put in the background by the user, you’ll resume exactly once, when all 30 requests are complete.

                          When that happens you’ll see a sequence of events like this:

                          1. App resumes (or relaunches)

                            In the relaunch case you must remember to re-create your background session with the same identifier that you used originally.  A common mistake is not re-creating the background session in the relaunch case.  The best approach in most cases is to create that session in code called from -application:willFinishLaunchingWithOptions:.

                          2. System calls -application:handleEventsForBackgroundURLSession:completionHandler:, passing it a completion handler

                          3. NSURLSession calls -URLSession:task:didCompleteWithError: for each task that’s complete

                          4. NSURLSession calls -URLSessionDidFinishEventsForBackgroundURLSession:, at which point it’s appropriate to call the completion handler from step 2

                          IMPORTANT You can’t use NSURLSession’s convenience APIs, things like -uploadTaskWithRequest:fromFile:completionHandler:, in a background session.  You must use the delegate-based API, for example, -uploadTaskWithRequest:fromFile:.  This makes sense when you think about it; if your app is terminated and then relaunched, there’s no way for the system to reconstitute the state held in a completion handler block.

                          Share and Enjoy

                          Quinn “The Eskimo!”
                          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                          let myEmail = "eskimo" + "1" + "@apple.com"

                            • Re: Background uploads with NSURLSession
                              literatemonk Level 1 Level 1 (0 points)

                              Thanks for the help. I'm currently using the S3 transfer utility library (https://cocoapods.org/pods/AWSS3) so I'll have to figure out what that's doing under the hood. I may need to write my own implementation based on signed URLs.  Will report back when I find out.

                              • Re: Background uploads with NSURLSession
                                Sreenivas34 Level 1 Level 1 (0 points)

                                Hi,  

                                 

                                I'm working on a similar iOS project, where I need to upload a very large video file (2 GB) to S3. 

                                 

                                I tried setting up a single upload request to handle the entire thing, but as you pointed out, when there is a connection failure or when the user terminates the app, the upload transfer fails and I'd have to start over again.  

                                 

                                If I were to split the 2 GB file into smaller 25 mb portions, I could add them all to my NSURLSession object to upload.

                                 

                                However, if the app is terminated and relaunched, and I retrieve the NSURLSession object, can I resume all of the pending upload requests? Does it have to upload all of the 25 mb files again or can it tell which ones have already been uploaded and only do the ones that haven't?  

                                 

                                Thanks,  

                                -Sreeni

                                  • Re: Background uploads with NSURLSession
                                    eskimo Apple Staff Apple Staff (6,665 points)

                                    However, if the app is terminated and relaunched, and I retrieve the NSURLSession object, can I resume all of the pending upload requests?

                                    By “terminated”, do you mean:

                                    • “Removed from the multitasking UI”, so that all of your uploads get cancelled?

                                    • Terminated by the OS due to memory pressure, so that all of your uploads continue in the background?

                                    Share and Enjoy

                                    Quinn “The Eskimo!”
                                    Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                    let myEmail = "eskimo" + "1" + "@apple.com"

                                      • Re: Background uploads with NSURLSession
                                        Sreenivas34 Level 1 Level 1 (0 points)

                                        Hi,

                                         

                                        By terminated, I mean “Removed from the multitasking UI”. Given that the app resumes (or relaunches) only when all tasks in the session complete, is there no way of keeping track of which files have been become uploaded and which are still pending? If all of the uploads become cancelled when a user force quits, then I'm guessing I can't retrieve the tasks that haven't completed (and were cancelled) and only resume those.

                                         

                                        Is the best way to approach uploading a very large file to do a multipart upload with appropriate server side support so the upload can be resumed? 

                                         

                                        Thanks,  

                                        -Sreeni

                                          • Re: Background uploads with NSURLSession
                                            eskimo Apple Staff Apple Staff (6,665 points)

                                            Given that the app resumes (or relaunches) only when all tasks in the session complete …

                                            To be clear, if the user removes your app from the multitasking UI then you won’t be resumed or relaunched in the background.  The user will have to manually relaunch your app.

                                            The behaviour for tasks that haven’t completed when the user removes the app from the multitasking UI varies by OS:

                                            • Originally the entire session would just disappear; it would be like you’d never issued those tasks

                                            • On current systems the tasks fail with NSURLErrorCancelled

                                            I believe the cutover was with iOS 8, that is, iOS 7 have the first behaviour and iOS 8 and later have the second, although I’ve never sat down to check that.

                                            I’ve also never looked at the behaviour for tasks that have completed.  Based on my understanding of how things fit together I suspect they’ll also get NSURLErrorCancelled, but you’d have to test this.

                                            Regardless, the best way to recover from this situation is to talk to the server to see which of the uploads completed successfully.  This is generally pretty easy to do via the HEAD HTTP method..


                                            Is the best way to approach uploading a very large file to do a multipart upload with appropriate server side support so the upload can be resumed?

                                            That’s true for downloads but not for uploads.  There’s actually a couple of problems doing a single, resumable upload:

                                            • With downloads, NSURLSession can automatically resume a failed download using HTTP standard technology (QA1761 gives an outline of how it works).  For uploads there’s no way for NSURLSession to automatically restart the transfer, so it has to relaunch or resume your app to do the job.  So, if you upload as a single file and the upload fails a bunch of times, you eventually provoke the ire of the resume rate limiter.

                                            • There’s no way to tell NSURLSession to start a resume from some offset in a file.  So if you have a large upload that fails, you have to make a copy of your upload file to remove the bytes at the front that have successfully been uploaded.

                                            The alternative is to segment the file into relatively large chunks and upload those.  This helps with both of these problems, at the cost of some code complexity.

                                            Share and Enjoy

                                            Quinn “The Eskimo!”
                                            Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                            let myEmail = "eskimo" + "1" + "@apple.com"

                                              • Re: Background uploads with NSURLSession
                                                Sreenivas34 Level 1 Level 1 (0 points)

                                                Thanks for the help! I think I know how to tackle the issue now.

                                                • Re: Background uploads with NSURLSession
                                                  Sreenivas34 Level 1 Level 1 (0 points)

                                                  Hi, 

                                                   

                                                  I have two more questions, if you can offer any advice:

                                                   

                                                  1. If I begin the transfers in the background (call the [uploadtask resume] while the app is in the background), and the discretionary property is automatically set to YES, what exactly does this mean? Does the phone have to be on wifi AND plugged in for it transfer or just be on wifi? What if I bring the app back to the foreground, is the discretionary property for these transfers set back to NO (given that session.discretionary = NO when I setup the background session)? 

                                                   

                                                  2. I setup and initiate around a 100 upload tasks to a NSURLSession object. If I lose network connectivity and regain it a while later (say 1 hour), does the session automatically resume all pending upload tasks? I'm only allowing uploads over wifi. I've been seeing the following strange behavior: 

                                                            1. I initate 100 upload tasks in a single NSURLSession

                                                            2. 10 uploads were successfully completed

                                                            3. I lose wifi connection

                                                            4. I gain wifi connection an hour later

                                                            5. The session would complete another 15 upload tasks

                                                            6. The session stops uploading anymore data (the URLSession didSendBodyData is no longer being called, so I'm guessing all of the tasks got cancelled somehow). 

                                                   

                                                  Any help would be greatly appreciated, 

                                                  Thanks, 

                                                  -Sreeni

                                                    • Re: Background uploads with NSURLSession
                                                      eskimo Apple Staff Apple Staff (6,665 points)

                                                      If I begin the transfers in the background (call the [uploadtask resume] while the app is in the background), and the discretionary property is automatically set to YES, what exactly does this mean?

                                                      To start, understand that the “discretionary property” isn’t an actual property, at least not one that you can see.  Rather, it’s a value computer by NSURLSession based on a number of criteria, including:

                                                      • The type of session (standard vs background)

                                                      • The discretionary property of the configuration used to create that session

                                                      • Whether the app was in the foreground when the task was created

                                                      • Whether the app is in the foreground right now

                                                      Moreover, the specific effects of this ‘property’ are not guaranteed; they have changed in the past and I fully expect them to change in the future.

                                                      Does the phone have to be on wifi AND plugged in for it transfer or just be on wifi?

                                                      In general, yes.  However, it’s really up to the OS as to when it schedules discretionary tasks, and it’s possible that, for example, if the device ‘learns’ that it’s never on Wi-Fi, it may chose to run discretionary tasks over WWAN.

                                                      What if I bring the app back to the foreground, is the discretionary property for these transfers set back to NO (given that session.discretionary = NO when I setup the background session)?

                                                      Again, no property values change here but, yes, modern versions of iOS include the foreground state of the originating app when determining whether to run a discretionary request.


                                                      I setup and initiate around a 100 upload tasks to a NSURLSession object.  If I lose network connectivity and regain it a while later (say 1 hour), does the session automatically resume all pending upload tasks?

                                                      Again, the exact behaviour is not specified and depends on a bunch of things.  But, yes, I would expect these tasks to eventually complete.

                                                      (the URLSession didSendBodyData is no longer being called, so I'm guessing all of the tasks got cancelled somehow)

                                                      You shouldn’t need to guess here.  If a task gets cancelled you should be told about it (via the standard task completion mechanism).  You can also confirm whether a task is still around using -getAllTasksWithCompletionHandler:.

                                                      I expect what you’ll find is that the remaining tasks are still around, they’re just not running right now for some reason (lack of Wi-Fi, lack of power, resource budgets, and so on).

                                                      Share and Enjoy

                                                      Quinn “The Eskimo!”
                                                      Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                                      let myEmail = "eskimo" + "1" + "@apple.com"