NSURLSession: streaming latency spikes

We have streaming app, which essentially upload data (video chunks) on server in very tight schedule. We use http streaming to do so (via NSURLSession uploadTaskWithStreamedRequest), not websockets/direct connection.

Chunk sizes are around 100Kb-300Kb. And we have an uploading problem, which is hard to trace to the roots... any advise welcome.



Problem: There is occasional spikes on upload latency for SOME files. Like 1 file out of 15-20 uploaded twice as slow as others. This happens sporadically and without any visible patters. This happens even on very good connection.

And we have to get rid of that spikes when network conditions allow for stable upload - since we do video streaming even 1 of 15-20 "late" files give our users "pause" on realtime playback.



We was able to reproduce this situation in details in local network with ideal conditions.

1) Client (uploader): iPhone XR with IOS12.

2) Client uploading single file with 300Kb in size via local network via Wi-Fi. After succesfull upload client start to upload it again, so we can measure sequentional uploads statistics

3) Server: Node.js doing nothing - just receiving data and log timings.

4) Uploading code:

***
NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration ephemeralSessionConfiguration];
configuration.allowsCellularAccess = YES;
configuration.HTTPShouldUsePipelining = YES;
configuration.networkServiceType = NSURLNetworkServiceTypeVoice;
***
NSInputStream* inputStream = [[NSInputStream alloc] initWithFileAtPath: pathTo300KbFile ];
[httpManager setTaskNeedNewBodyStreamBlock:^NSInputStream * _Nonnull(NSURLSession * _Nonnull session, NSURLSessionTask * _Nonnull task) {
return inputStream;
}];
***
NSMutableURLRequest *fileRequest = [NSMutableURLRequest requestWithURL:URL
   cachePolicy:NSURLRequestReloadIgnoringLocalCacheData
   timeoutInterval:100];
[fileRequest setHTTPMethod:@"POST"];
***
NSURLSessionUploadTask* uploadTask = [httpManager uploadTaskWithStreamedRequest:fileRequest progress:... completionHandler:...];
[uploadTask resume];
***

This code just start upload again from the beginning, after uploadTask returns success (via callback)


5) "Normal" Results of the test. This happens most of the time, you can see that average upload time of 300Kb file on local network is quite low and stable - around 0.5 seconds

2019-04-18 15:07:44.793512+0300 DVGCore_Example[2016:1384429] VidLib: #191:300kb-s1-7. Success: +0.54

2019-04-18 15:07:46.019874+0300 DVGCore_Example[2016:1384370] VidLib: #192:300kb-s1-8. Success: +0.67

2019-04-18 15:07:46.828315+0300 DVGCore_Example[2016:1383910] VidLib: #193:300kb-s1-9. Success: +0.37

2019-04-18 15:07:48.383446+0300 DVGCore_Example[2016:1383909] VidLib: #194:300kb-s1-10. Success: +0.83

2019-04-18 15:07:48.948218+0300 DVGCore_Example[2016:1384429] VidLib: #195:300kb-s1-1. Success: +0.29

2019-04-18 15:07:50.116136+0300 DVGCore_Example[2016:1383910] VidLib: #196:300kb-s1-2. Success: +0.37

2019-04-18 15:07:51.168454+0300 DVGCore_Example[2016:1384428] VidLib: #197:300kb-s1-3. Success: +0.33

2019-04-18 15:07:52.244631+0300 DVGCore_Example[2016:1384370] VidLib: #198:300kb-s1-4. Success: +0.33


Example of the server logs regarding average (with receivement timings), normal upload: https://gist.github.com/IPv6/27dc7a8e4a53b5219e39cce73691c0be


6) But for some uploads there is occasional spikes in upload time. 2-3 times longer

2019-04-18 15:16:50.492425+0300 DVGCore_Example[2022:1385864] VidLib: #124:300kb-s1-8. Success: +1.87

2019-04-18 15:22:06.395094+0300 DVGCore_Example[2027:1387221] VidLib: #200:300kb-s1-6. Success: +2.11

2019-04-18 15:23:28.329843+0300 DVGCore_Example[2027:1387610] VidLib: #281:300kb-s1-9. Success: +2.84


Example of the server logs for this upload: https://gist.github.com/IPv6/8d92b3899682c4cf697dbfb64a1b752c


To the bottom: There is no reasons for this (no network problems, etc) and seems like IOS should be able to do streaming without delays for hours. But this does not happen.

We see this in ideal local environment, and in "real world" same spikes are even more frequent.

The question is - how to get rid of them? any ideas?

Accepted Reply

First up, have a read of my Investigating Network Latency Problems post. While it’s focus is on lower-level concerns, it’s still full of good advice.

In your case I see two options:

  • You can use CFNetwork diagnostic logging to look for causes at the CFNetwork level.

  • If the problem seems to be lower than that, you can start looking at the techniques from Investigating Network Latency Problems.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Replies

First up, have a read of my Investigating Network Latency Problems post. While it’s focus is on lower-level concerns, it’s still full of good advice.

In your case I see two options:

  • You can use CFNetwork diagnostic logging to look for causes at the CFNetwork level.

  • If the problem seems to be lower than that, you can start looking at the techniques from Investigating Network Latency Problems.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks, i followed instructions from your article and after looking into tcpdump we see clear difference between "usual" uploads and "lengthy" ones - lengthy uploads has a lot of "TCP Dup ACK"/Retransmissions for tcp packets for uploaded data.
We also tried to run test on several devices at once (in the same network). And we can see that delays are not correlated between devices (so when there is delay on one device - everything is smooth on another). So it looks like a problems between device and wifi router?

So it looks like a problems between device and wifi router?

It’s hard to know for sure without looking at the traffic at other positions along the route to the original server. At a minimum you need to do a second packet trace immediately upstream of the AP. That’ll tell you whether the packets got dropped between the client and the AP, or somewhere upstream.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"