15 Replies
      Latest reply on Apr 18, 2019 3:32 PM by helloethan
      tomwyckhuys Level 1 Level 1 (0 points)

        Hi!

         

        I am implementing the Alexa Voice Service in a native iOS app (swift 2.3) and i'm struggling with the http/2 implementation.

         

        The AVS documentation states the following:

        "The request sent to AVS is a multipart message: the first part a JSON-formatted object, the second part binary audio captured by the microphone. We encourage streaming (chunking) captured audio to the Alexa Voice Service to reduce latency; the stream should contain 10ms of captured audio per chunk (320 bytes)."

         

        To be a bit more clear: I want to upload (stream) the audio to AVS while I'm still recording the user's voice.

         

        I already succeeded in uploading a full sound sample and sending that to AVS as a whole. However, the streaming implemtation is what I need and I have no idea how to accomplish this.

         

        Can somebody give me some guidance ?

         

        Thank you!!!

        • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
          eskimo Apple Staff Apple Staff (11,505 points)

          I presume you’re asking about the networking side of this, not the audio recording side.  I can help with the former but not the latter.

          You can use URLSession to do a streamed upload by creating the upload task with uploadTask(withStreamedRequest:).  The session will then call the urlSession(_:task:needNewBodyStream:) delegate method to get an input stream for the upload body.  It will then open and read (asynchronously) data from that stream, uploading it as it becomes available.

          IMPORTANT It’s possible for the session to call your delegate method more than once.  For example, if the request fails with an authentication error, the session may automatically send the request again, and that will need a new body stream.

          You can set up this input stream in two ways:

          • As the input side of a stream pair (via    Stream.getBoundStreams(withBufferSize:inputStream:outputStream:)), whereupon writing data to the output side of the stream will make it appear on the input side, and hence cause it to get uploaded

          • As an InputStream subclass

          IMPORTANT Remember that the session reads the data from the stream asynchronously.  If you choose to implement the second option, you must support asynchronous reads properly.

          Share and Enjoy

          Quinn “The Eskimo!”
          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
          let myEmail = "eskimo" + "1" + "@apple.com"

            • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
              tomwyckhuys Level 1 Level 1 (0 points)

              Hi!

              First of all, thank you so much for showing interest in the question! I posted this topic on several other forums but people just ignored me there..

              Ì looked into the uploadTask(withStreamedRequest:) function you talked about in your reply. I grasp the concept of the delegate methods but one thing I could not understand was how the multipart message is going to work. I tried to make it work today but kept getting 400 responses from AVS.

              Here is what I tried to do today:

              To keep things simple, I pre-recorded a sound sample containing "hi" and used that to get some response out of AVS. This is the function I used to build the request body

              func uploadSoundSample() -> Void {
                      let configuration = URLSessionConfiguration.default
                      let session = URLSession(configuration: configuration, delegate: self, delegateQueue: nil)
                      let request = NSMutableURLRequest(url: URL(string: "https:/
                      let boundry = UUID().uuidString
                      let bodyData = NSMutableData()
                   
                      bodyData.append("--\(boundry)\r\n".data(using: String.Encoding.utf8)!)
                      bodyData.append("Content-Disposition: form-data; name=\"metadata\"\r\n".data(using: String.Encoding.utf8)!)
                      bodyData.append("Content-Type: application/json; charset=UTF-8\r\n\r\n".data(using: String.Encoding.utf8)!)
                      bodyData.append(getJSONData().data(using: String.Encoding.utf8)!)
                      bodyData.append("\r\n".data(using: String.Encoding.utf8)!)
                      bodyData.append("--\(boundry)\r\n".data(using: String.Encoding.utf8)!)
                      bodyData.append("Content-Disposition: form-data; name=\"audio\"\r\n".data(using: String.Encoding.utf8)!)
                      bodyData.append("Content-Type: application/octet-stream\r\n\r\n".data(using: String.Encoding.utf8)!)
                   
                      request.setValue("Bearer \(Settings.TOKEN)", forHTTPHeaderField: "authorization")
                      let contentType = "multipart/form-data; boundary=\(boundry)"
                      request.setValue(contentType, forHTTPHeaderField: "content-type")
                      request.httpMethod = "POST"
                      request.httpBody = bodyData as Data
                      request.httpBodyStream = InputStream(url: URL(fileURLWithPath: Bundle.main.path(forResource: "hi", ofType: "wav")!))
                      let task = session.uploadTask(withStreamedRequest: request as URLRequest)
                      task.resume()
                  }
              
                  fileprivate func getJSONData() -> String {
                      return "{\"context\":[{\"header\":{\"namespace\":\"Alerts\",\"name\":\"AlertsState\"},\"payload\":{\"allAlerts\":[],\"activeAlerts\":[]}},{\"header\":{\"namespace\":\"AudioPlayer\",\"name\":\"PlaybackState\"},\"payload\":{\"token\":\"\",\"offsetInMilliseconds\":0,\"playerActivity\":\"IDLE\"}},{\"header\":{\"namespace\":\"Speaker\",\"name\":\"VolumeState\"},\"payload\":{\"volume\":100,\"muted\":false}},{\"header\":{\"namespace\":\"SpeechSynthesizer\",\"name\":\"SpeechState\"},\"payload\":{\"token\":\"\",\"offsetInMilliseconds\":0,\"playerActivity\":\"FINISHED\"}}],\"event\":{\"header\":{\"namespace\":\"SpeechRecognizer\",\"name\":\"Recognize\",\"messageId\":\"messageId-123\",\"dialogRequestId\":\"dialogRequestId-321\"},\"payload\":{\"profile\":\"NEAR_FIELD\",\"format\":\"AUDIO_L16_RATE_16000_CHANNELS_1\"}}}"
                  }
              

              And these are the delegate methods I've implemented:

              func urlSession(_ session: URLSession, task: URLSessionTask, needNewBodyStream completionHandler: @escaping (InputStream?) -> Void) {
                      let inputStream = InputStream(url: URL(fileURLWithPath: Bundle.main.path(forResource: "hi", ofType: "wav")!))
                      completionHandler(inputStream)
                  }
                
                  func urlSession(_ session: URLSession, task: URLSessionTask, didSendBodyData bytesSent: Int64, totalBytesSent: Int64, totalBytesExpectedToSend: Int64) {
                      print("body data sent")
                  }
                
                  func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
                      print("task done sending data: \(error)")
                  }
                  func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive data: Data) {
                      print("got data")
                  }
                
                  func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive response: URLResponse, completionHandler: @escaping (URLSession.ResponseDisposition) -> Void) {
                      print("got response: \(response)")
                  }
              

              I guess I'm still doing a couple of things wrong but I have limited understanding of Swift networking in general

              I'm looking forward to your answer!

              Greets from Belgium!

                • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
                  eskimo Apple Staff Apple Staff (11,505 points)

                  You’re definitely getting mixed up here.  When you use urlSession(_:task:needNewBodyStream:) that stream has to yield the entire HTTP body.  You’re trying to use two streams, one that contains the multipart/form-data preamble, which you set up via the httpBodyStream property, and another that contains the bulk data for that form, which you set up via urlSession(_:task:needNewBodyStream:).  That won’t work.  You need one stream that yields the entire body.

                  For a simple test you could:

                  1. Remove the httpBody and httpBodyStream setup code from uploadSoundSample()

                  2. Implement urlSession(_:task:needNewBodyStream:)

                  3. In that implementation:

                    1. Generate the multipart/form-data preamble using the bodyData code you currently have in uploadSoundSample()

                    2. Load hi.wav into another data value

                    3. Concatenate the two data values

                    4. Build a stream from the result

                  Once you have this basic test working you can then go on to investigate using an indefinite stream.


                  btw Given that you’re using Swift 3 you should skip NSMutableData and use a Swift Data value.  You can make this mutable by declaring it using var.

                  Share and Enjoy

                  Quinn “The Eskimo!”
                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                  let myEmail = "eskimo" + "1" + "@apple.com"

                    • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
                      tomwyckhuys Level 1 Level 1 (0 points)

                      Hi!

                      I modified the code to meet the requirements of your previous reply. I now get 200 responses from AVS!

                       

                      However, AVS sends back data containing the binary audio to play in my app but I never receive the data in the urlSession:didReceive data function. Is this normal behaviour ?

                       

                      Just for reference: here is the code I have now:

                       

                      fileprivate let boundary = UUID().uuidString
                         
                          func uploadSoundSample() -> Void {
                              let configuration = URLSessionConfiguration.default
                              let session = URLSession(configuration: configuration, delegate: self, delegateQueue: nil)
                              let request = NSMutableURLRequest(url: URL(string: "https:/
                              let contentType = "multipart/form-data; boundary=\(boundary)"
                             
                              request.setValue("Bearer \(Settings.TOKEN)", forHTTPHeaderField: "authorization")
                              request.setValue(contentType, forHTTPHeaderField: "content-type")
                              request.httpMethod = "POST"
                             
                              let task = session.uploadTask(withStreamedRequest: request as URLRequest)
                              task.resume()
                          }
                      
                      func urlSession(_ session: URLSession, task: URLSessionTask, needNewBodyStream completionHandler: @escaping (InputStream?) -> Void) {
                              var bodyData = Data()
                              do {
                                  let audioData = try Data(contentsOf: URL(fileURLWithPath: Bundle.main.path(forResource: "hi", ofType: "wav")!))
                                
                                  bodyData.append("--\(boundary)\r\n".data(using: String.Encoding.utf8)!)
                                  bodyData.append("Content-Disposition: form-data; name=\"metadata\"\r\n".data(using: String.Encoding.utf8)!)
                                  bodyData.append("Content-Type: application/json; charset=UTF-8\r\n\r\n".data(using: String.Encoding.utf8)!)
                                  bodyData.append(getJSONData().data(using: String.Encoding.utf8)!)
                                  bodyData.append("\r\n".data(using: String.Encoding.utf8)!)
                                  bodyData.append("--\(boundary)\r\n".data(using: String.Encoding.utf8)!)
                                  bodyData.append("Content-Disposition: form-data; name=\"audio\"\r\n".data(using: String.Encoding.utf8)!)
                                  bodyData.append("Content-Type: application/octet-stream\r\n\r\n".data(using: String.Encoding.utf8)!)
                                
                                  bodyData.append(audioData)
                                  bodyData.append("\r\n".data(using: String.Encoding.utf8)!)
                                  bodyData.append("--\(boundary)\r\n".data(using: String.Encoding.utf8)!)
                              } catch let error {
                                  print("something went wrong: \(error)")
                              }
                              let inputStream = InputStream(data: bodyData)
                              completionHandler(inputStream)
                          }
                      

                       

                      Thanks!

                        • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
                          eskimo Apple Staff Apple Staff (11,505 points)

                          However, AVS sends back data containing the binary audio to play in my app but I never receive the data in the urlSession:didReceive data function.

                          How did you determine that the server actually send you any data in that case?

                          Is this normal behaviour ?

                          I would expect you to get the ‘did receive data’ call with the body data of the response.

                          Share and Enjoy

                          Quinn “The Eskimo!”
                          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                          let myEmail = "eskimo" + "1" + "@apple.com"

                            • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
                              tomwyckhuys Level 1 Level 1 (0 points)

                              Well, before posting this thread, I already played around a bit with the NSUrlSession class. Back then, the AVS communication was implemented like this:

                              let uploadTask = self.session.uploadTaskWithRequest(request, fromData: bodyData){
                              (data:NSData?, response:NSURLResponse?, error:NSError?) -> Void in
                                   //handle response
                              
                              }
                              uploadTask.resume()
                              
                              

                               

                              When this function threw a response, the audio of AVS was contained in the data parameter of the function. That's why I expect to receive data in the current implementation with the delegate methods but that ain't happening.

                               

                              I'm looking forward to your answer!

                                • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
                                  eskimo Apple Staff Apple Staff (11,505 points)

                                  So, to confirm, you've implemented the urlSession(_:dataTask:didReceive:) delegate callback and it’s not getting called, right?

                                  Weird.  I tried this out here in my office and, while I saw some weird stuff, I definitely get the response data back from the server.

                                  Consider the test code pasted in at the end of this post.  When I run startIdentity(), it prints this:

                                  2017-05-02 10:01:35.288 xxsi[2506:145728] success, status: 200
                                  2017-05-02 10:01:35.288 xxsi[2506:145728] response: {
                                      args =    {
                                      };
                                      data = "";
                                      files =    {
                                      };
                                      form =    {
                                          greetings = "Hello Cruel World!";
                                          salutations = "Goodbye Cruel World!";
                                      };
                                      headers =    {
                                          Accept = "*/*";
                                          "Accept-Encoding" = "gzip, deflate";
                                          "Accept-Language" = "en-us";
                                          Connection = close;
                                          "Content-Length" = 383;
                                          "Content-Type" = "multipart/form-data; boundary=Boundary-0DFADA5A-6C2F-4FD9-9E92-CCBB2A100278";
                                          Host = "httpbin.org";
                                          "User-Agent" = "xxsi/1.0 CFNetwork/811.4.18 Darwin/16.5.0";
                                      };
                                      json = "<null>";
                                      origin = "88.97.8.212";
                                      url = "https://httpbin.org/post";
                                  }
                                  

                                  As expected:

                                  • The server (httpbin.org) has sent me some JSON that describes the POST request I sent it; I correctly received and printed that

                                  • The server has correctly parsed my form data (not the form element within the JSON)

                                  Now look at when happens when I run startChunked():

                                  2017-05-02 10:01:40.811 xxsi[2506:145728] data chunk: <7b0a2020 …>
                                  2017-05-02 10:01:40.811 xxsi[2506:145728] success, status: 200
                                  2017-05-02 10:01:40.812 xxsi[2506:145728] response: {
                                      args =    {
                                      };
                                      data = "";
                                      files =    {
                                      };
                                      form =    {
                                      };
                                      headers =    {
                                          Accept = "*/*";
                                          "Accept-Encoding" = "gzip, deflate";
                                          "Accept-Language" = "en-us";
                                          Connection = close;
                                          "Content-Type" = "multipart/form-data; boundary=Boundary-0DFADA5A-6C2F-4FD9-9E92-CCBB2A100278";
                                          Host = "httpbin.org";
                                          "Transfer-Encoding" = Chunked;
                                          "User-Agent" = "xxsi/1.0 CFNetwork/811.4.18 Darwin/16.5.0";
                                      };
                                      json = "<null>";
                                      origin = "88.97.8.212";
                                      url = "https://httpbin.org/post";
                                  }
                                  

                                  There’s two things to note here:

                                  • As before, the server has sent me some JSON and I’ve been able to parse and print it

                                  • The server did not recognise my form data.

                                  AFAICT I’ve done nothing wrong here.  Looking at a packet trace of what I sent on the wire, the only difference between the two requests is the transfer encoding.  The identity request uses no transfer encoding and the chunked request uses a transfer encoding of Chunked.  This is both expected (NSURLSession doesn’t know the length of the body stream and thus has to use chunked encoding) and correct (HTTP 1.1 requires that the server support chunked encoding).  However, it’s clear that httpbin.org is not handling this properly.


                                  Coming back to your situation, I’m not sure what this means for you.  To do a streamed upload you’re going to have to use chunked encoding.  I presume that the server you’re talking to will accept that, but it’s hard to know for sure because I don’t control the server.

                                  IMO your best way forward here is to:

                                  • Learn how to use packet tracing technologies so you can see exactly what’s happening on the wire

                                  • Find a working client (perhaps the server’s SDK includes some sample code) and compare its on-the-wire behaviour to that of your client’s

                                  I discuss this idea in depth in my Debugging HTTP Server-Side Errors post.

                                  Share and Enjoy

                                  Quinn “The Eskimo!”
                                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                  let myEmail = "eskimo" + "1" + "@apple.com"

                                  import UIKit
                                  
                                  class MainViewController : UITableViewController, URLSessionDataDelegate {
                                  
                                      var session: URLSession!
                                      var accumulated = Data()
                                  
                                      override func viewDidLoad() {
                                          super.viewDidLoad()
                                          let config = URLSessionConfiguration.default
                                          config.requestCachePolicy = .reloadIgnoringLocalCacheData
                                          self.session = URLSession(configuration: config, delegate: self, delegateQueue: OperationQueue.main)
                                      }
                                  
                                      override func tableView(_ tableView: UITableView, didSelectRowAt indexPath: IndexPath) {
                                          switch (indexPath.section, indexPath.row) {
                                              case (0, 0): self.startIdentity()
                                              case (0, 1): self.startChunked()
                                              default: fatalError()
                                          }
                                          self.tableView.deselectRow(at: indexPath, animated: true)
                                      }
                                  
                                      func startIdentity() {
                                          self.session.uploadTask(with: MainViewController.testRequest, from: MainViewController.testBody) { (data, response, error) in
                                              MainViewController.print(error: error, response: response, body: data)
                                          }.resume()
                                      }
                                  
                                      func startChunked() {
                                          self.accumulated.removeAll()
                                          self.session.uploadTask(withStreamedRequest: MainViewController.testRequest).resume()
                                      }
                                  
                                      func urlSession(_ session: URLSession, task: URLSessionTask, needNewBodyStream completionHandler: @escaping (InputStream?) -> Void) {
                                          let bodyStream = InputStream(data: MainViewController.testBody)
                                          completionHandler(bodyStream)
                                      }
                                  
                                      func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive data: Data) {
                                          NSLog("data chunk: %@", data as NSData)
                                          self.accumulated.append(data)
                                      }
                                  
                                      func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
                                          MainViewController.print(error: error, response: task.response, body: self.accumulated)
                                      }
                                  
                                      static let testRequest: URLRequest = {
                                          var result = URLRequest(url: URL(string: "http://httpbin.org/post")!)
                                          result.httpMethod = "POST"
                                          result.setValue("multipart/form-data; boundary=Boundary-0DFADA5A-6C2F-4FD9-9E92-CCBB2A100278", forHTTPHeaderField: "Content-Type")
                                          return result
                                      }()
                                  
                                      static let testBody: Data = {
                                          var result = Data()
                                  
                                          func add(_ string: String) {
                                              result.append(string.data(using: String.Encoding.utf8)!)
                                          }
                                  
                                          // empty preamble
                                          add("\r\n") 
                                          add("--Boundary-0DFADA5A-6C2F-4FD9-9E92-CCBB2A100278\r\n") 
                                          add("Content-Disposition: form-data; name=\"greetings\"\r\n") 
                                          add("Content-Type: text/plain; charset=UTF-8\r\n") 
                                          add("\r\n") 
                                          add("Hello Cruel World!") 
                                          add("\r\n") 
                                          add("--Boundary-0DFADA5A-6C2F-4FD9-9E92-CCBB2A100278\r\n") 
                                          add("Content-Disposition: form-data; name=\"salutations\"\r\n") 
                                          add("Content-Type: text/plain; charset=UTF-8\r\n") 
                                          add("\r\n") 
                                          add("Goodbye Cruel World!") 
                                          add("\r\n") 
                                          add("--Boundary-0DFADA5A-6C2F-4FD9-9E92-CCBB2A100278--\r\n") 
                                          add("\r\n") 
                                          //empty epilogue
                                  
                                          return result
                                      }()
                                  
                                      static func print(error: Error?, response: URLResponse?, body: Data?) {
                                          if let error = error as NSError? {
                                              NSLog("failed, error: %@ / %d", error.domain, error.code)
                                          } else {
                                              let response = response! as! HTTPURLResponse
                                              let body = body!
                                  
                                              NSLog("success, status: %d", response.statusCode)
                                  
                                              if let root = try? JSONSerialization.jsonObject(with: body, options: []) {
                                                  NSLog("response: %@", "\(root)")
                                              } else {
                                                  NSLog("response not parsed")
                                              }
                                          }
                                      }
                                  }
                                  
                                    • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
                                      tomwyckhuys Level 1 Level 1 (0 points)

                                      Hi eskimo!

                                       

                                      I searched for the cause of the didReceivedata function not being called and found out that I just forgot to call the completion handler in the didreceiveresponse function.. To be clear: I receive the data now, so that's working!

                                       

                                      Now there is one part missing to make all of this work.. We are now sending prerecorded sound samples but how do we get to stream the sound of the voice recorder directly to the request. You mentioned it could be possible with something called a stream pair ?

                                       

                                      How could I implement this in my current request code/configuration ?

                                       

                                      PS. I really appreciate the amount of effort and time you spend on this thread!

                                        • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
                                          eskimo Apple Staff Apple Staff (11,505 points)

                                          I receive the data now, so that's working!

                                          Yay!

                                          … how do we get to stream the sound of the voice recorder directly to the request. You mentioned it could be possible with something called a stream pair ?

                                          There are two approaches you can use:

                                          • Subclass InputStream

                                          • Stream pair


                                          The stream pair approach is the simplest to described:

                                          1. Call Stream.getBoundStreams(withBufferSize:inputStream:outputStream:) to create a bound pair of streams

                                          2. Pass the input stream to the upload task

                                          3. Open the output stream

                                          4. Write data to it

                                          That data will show up on the input stream, whereupon it’ll be read by the upload task and sent to the server.


                                          Subclassing InputStream sounds pretty straightforward but it’s actually quite tricky because you have to deal with run loops.  The basic idea is:

                                          1. Create a subclass of InputStream that returns your audio data

                                          2. Instantiate that and pass it to the upload task

                                          The upload task will then open that stream, read data from it, and send it to the server.  The complication here is that the upload task will read the stream asynchronously, by scheduling it on a run loop, and correctly implementing that is a bit tricky (in fact, it wasn’t even supported prior to iOS 7).

                                          Share and Enjoy

                                          Quinn “The Eskimo!”
                                          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                          let myEmail = "eskimo" + "1" + "@apple.com"

                                            • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
                                              vivekmp_ios Level 1 Level 1 (0 points)

                                              Currently i am recroding the audio data into  a temp file and then fetching data from that file in form of NSData (while setting up request to upload).

                                              Same as above

                                               

                                              func urlSession(_ session: URLSession, task: URLSessionTask, needNewBodyStream completionHandler: @escaping (InputStream?) -> Void) {
                                                      var bodyData = Data()
                                                      do {
                                                          let audioData = try Data(contentsOf: URL(fileURLWithPath: Bundle.main.path(forResource: "hi", ofType: "wav")!))
                                              
                                                          bodyData.append("--\(boundary)\r\n".data(using: String.Encoding.utf8)!)
                                                          bodyData.append("Content-Disposition: form-data; name=\"metadata\"\r\n".data(using: String.Encoding.utf8)!)
                                                          bodyData.append("Content-Type: application/json; charset=UTF-8\r\n\r\n".data(using: String.Encoding.utf8)!)
                                                          bodyData.append(getJSONData().data(using: String.Encoding.utf8)!)
                                                          bodyData.append("\r\n".data(using: String.Encoding.utf8)!)
                                                          bodyData.append("--\(boundary)\r\n".data(using: String.Encoding.utf8)!)
                                                          bodyData.append("Content-Disposition: form-data; name=\"audio\"\r\n".data(using: String.Encoding.utf8)!)
                                                          bodyData.append("Content-Type: application/octet-stream\r\n\r\n".data(using: String.Encoding.utf8)!)
                                              
                                                          bodyData.append(audioData)
                                                          bodyData.append("\r\n".data(using: String.Encoding.utf8)!)
                                                          bodyData.append("--\(boundary)\r\n".data(using: String.Encoding.utf8)!)
                                                      } catch let error {
                                                          print("something went wrong: \(error)")
                                                      }
                                                      let inputStream = InputStream(data: bodyData)
                                                      completionHandler(inputStream)
                                                  }
                                              

                                               

                                              How can I write audio data directly to my request body also uploading it at the same time??

                                              I am using AVAudioRecorder for recording audio into a temp file.

                                               

                                                  let settings : [String : AnyObject] = [
                                                          AVFormatIDKey: Int(kAudioFormatLinearPCM),
                                                          AVSampleRateKey: samplerate,
                                                          AVNumberOfChannelsKey: noOfChannel,
                                                          AVEncoderBitRatePerChannelKey : bitRate,
                                                          AVEncoderAudioQualityKey: AVAudioQuality.High.rawValue
                                                      ]
                                                    
                                                      do {
                                                          let url = NSURL(fileURLWithPath: tempFilename)
                                                          self.mRecorder?.deleteRecording()
                                                          self.mRecorder = try AVAudioRecorder(URL: url, settings: settings)
                                                          self.mRecorder?.prepareToRecord()
                                                          self.mRecorder?.record()
                                              
                                                      } catch {
                                                          throw RecorderSetUpError.Error("Something went wrong when audiorecorder is setup")
                                                      }
                                              

                                               

                                              Thanks and Regards.

                                                • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
                                                  eskimo Apple Staff Apple Staff (11,505 points)

                                                  I can’t help you with the audio side of this.  I’m going to assume that you have some way to get chunks of audio in a format that your server will accept.  If not, you’ll need to ask questions about that over in Media > AVFoundation (Audio).

                                                  On the networking front the basic strategy for uploading a stream of data is as follows:

                                                  1. Create an upload that uses a streamed body by calling uploadTask(withStreamedRequest:).

                                                  2. Once you resume the request, URLSession will call urlSession(_:task:needNewBodyStream:) to get a body stream for the request.

                                                  3. You have two choices for creating this body stream:

                                                    • You can subclass InputStream

                                                    • You can create a bound stream pair.

                                                    I’ll discuss the details of each below.

                                                  4. Your implementation of the urlSession(_:task:needNewBodyStream:) method must call the completion handler with that input stream.


                                                  Subclassing InputStream is simple in the synchronous case but your subclass must support async reads.  This should be possible on recent systems but it’s quite tricky.  I generally recommend the bound stream pair approach.


                                                  Creating a bound stream pair is a bit tricky from Swift because the API to do so, getBoundStreams(withBufferSize:inputStream:outputStream:), uses a bunch of ‘out’ parameters.  I generally wrap that API as follows:

                                                  extension Stream {
                                                      func boundPair(bufferSize: Int) -> (inputStream: InputStream, outputStream: OutputStream) {
                                                          var inStream: InputStream? = nil
                                                          var outStream: OutputStream? = nil
                                                          Stream.getBoundStreams(withBufferSize: bufferSize, inputStream: &inStream, outputStream: &outStream)
                                                          return (inStream!, outStream!)
                                                      }
                                                  }

                                                  Once you have a bound stream pair your can pass the input stream to the completion handler (step 4 above) and then you ‘own’ the output stream.  You should run the stream asynchronously, writing data to it in chunks as that data gets recorded.  Any data you write to the stream will be sent as a chunk of the HTTP body (that is, using the chunked Transfer Encoding).

                                                  The output stream here is no different from any other output stream, and there’s tonnes of examples of how to run an output stream asynchronously.

                                                  Share and Enjoy

                                                  Quinn “The Eskimo!”
                                                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                                  let myEmail = "eskimo" + "1" + "@apple.com"

                                                    • Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
                                                      vivekmp_ios Level 1 Level 1 (0 points)

                                                      Thanks @eskimo  for details you have mentioned here.

                                                       

                                                      Yes i am already recording and uploading to the server and getting 200 in response from my server. But its like uploading a pre-recorded audio data to the request, using similar code like this one -

                                                       

                                                          func uploadSoundSample() -> Void {
                                                              let configuration = URLSessionConfiguration.default
                                                              let session = URLSession(configuration: configuration, delegate: self, delegateQueue: nil)
                                                              let request = NSMutableURLRequest(url: URL(string: "https:/
                                                              let contentType = "multipart/form-data; boundary=\(boundary)"
                                                      
                                                              request.setValue("Bearer \(Settings.TOKEN)", forHTTPHeaderField: "authorization")
                                                              request.setValue(contentType, forHTTPHeaderField: "content-type")
                                                              request.httpMethod = "POST"
                                                      
                                                              let task = session.uploadTask(withStreamedRequest: request as URLRequest)
                                                              task.resume()
                                                          }
                                                      
                                                      

                                                      and providing reuest body via NSInputStream in the delegate "needNewBodyStream". As you can see the audio data is coming from a file (already recorded with correct audio format).

                                                      func urlSession(_ session: URLSession, task: URLSessionTask, needNewBodyStream completionHandler: @escaping (InputStream?) -> Void) {
                                                              var bodyData = Data()
                                                              do {
                                                                  let audioData = try Data(contentsOf: URL(fileURLWithPath: Bundle.main.path(forResource: "hi", ofType: "wav")!))
                                                      
                                                                  bodyData.append("--\(boundary)\r\n".data(using: String.Encoding.utf8)!)
                                                                  bodyData.append("Content-Disposition: form-data; name=\"metadata\"\r\n".data(using: String.Encoding.utf8)!)
                                                                  bodyData.append("Content-Type: application/json; charset=UTF-8\r\n\r\n".data(using: String.Encoding.utf8)!)
                                                                  bodyData.append(getJSONData().data(using: String.Encoding.utf8)!)
                                                                  bodyData.append("\r\n".data(using: String.Encoding.utf8)!)
                                                                  bodyData.append("--\(boundary)\r\n".data(using: String.Encoding.utf8)!)
                                                                  bodyData.append("Content-Disposition: form-data; name=\"audio\"\r\n".data(using: String.Encoding.utf8)!)
                                                                  bodyData.append("Content-Type: application/octet-stream\r\n\r\n".data(using: String.Encoding.utf8)!) 
                                                      
                                                                  bodyData.append(audioData)
                                                                  bodyData.append("\r\n".data(using: String.Encoding.utf8)!)
                                                                  bodyData.append("--\(boundary)\r\n".data(using: String.Encoding.utf8)!)
                                                              } catch let error {
                                                                  print("something went wrong: \(error)")
                                                              }
                                                              let inputStream = InputStream(data: bodyData)
                                                              completionHandler(inputStream)
                                                          }
                                                      
                                                      

                                                      I’m going to assume that you have some way to get chunks of audio in a format that your server will accept :


                                                      So do you mean that now i should get the chunks in real time from the recorder and pass it to same inputStream in that delegate method??
                                                      or do you mean i should i write body(wiithout audio part) before -   task.resume()
                                                      and when the delegate gets called then i should pass the audio part via inputstream (not sure if we can pass body in different streams i.e httpBody/HttpsBodyStream  and rest in "needNewBodyStream" delegate via NSInputStream).

                                                       

                                                      Also i  had this one doubt, previously i was using "uploadTaskWithRequest" for same request and now this - "uploadTaskWithStreamedRequest" .
                                                      Although request and responses are working just fine, just wondering i was checking logs in my delegate method -

                                                       

                                                          func URLSession(session: NSURLSession, task: NSURLSessionTask, didSendBodyData bytesSent: Int64, totalBytesSent: Int64, totalBytesExpectedToSend: Int64) {
                                                           
                                                              print("bytesSent = \(bytesSent)")
                                                              print("totalBytesSent = \(totalBytesSent)")
                                                              print("totalBytesExpectedToSend = \(totalBytesExpectedToSend)")    
                                                          }
                                                      
                                                      
                                                      

                                                       

                                                      which is almost same (in both cases - uploadTaskWithRequest and StreamRequest), still same chunks of data are being sent. Is it expected behaviour, or should there be some difference between the dataBytes sent over the network?

                                                       

                                                      Thanks & Regards!!