UrlSession.shared.data returns "The network connection was lost."

A couple of updates:

• the download works from Chrome and .NET (created a sample app)
• old installation of the app, deployed to old iPhones, manifest now the same issue
• tried to run the app from MacOS - same error

I have found this thread https://developer.apple.com/forums/thread/84608 where the server was blocking downloads from unknows networks - but still it works with WiFi.

To me, it doesn't work in both 5G and wifi, and even from computers - but the .NET code works fine.

So I would still appeciate some help from the community.

To me, it doesn't work in both 5G and wifi, and even from computers - but the .NET code works fine.

If you take one of your download methods and move it to either a macOS command line tool or a just a single view app and run just the download with a known file that you can access from a local server, does it work then? If it does then you know you either have to look further into your app code or the resources on your server.

My code was working in the past, and it still works on other URLs

Still, for this particular URL, it doesn't work from iOS and MacOS. But on the same machine works for example from .NET.

Have you been able to verify if the GitHub sample works for you?

Hi, I still have this issue and I have done the following tests:

• on a Mac app, it has the same identical issue (so I am testing on the Mac app now)
• I have developed a MAUI app (.NET) and deployed to the iPhone. It works fine and can download the file
• If I use other urls, my code works fine
• I have tried disabling NSAppTransportSecurity allowing NSAllowsArbitraryLoads. I can download http content, but this content is still wrong
• I have tested on this url, that sends a redirect, and it also works fine: https://www.backade.com/22Z9H358/SZCPXRW/?uid=2297

So the issue is just between the Swift libraries and this website. Other websites (even in Swift) or other languages (.NET on both Windows and iOS) work fine Clearly both Chrome and Safari can download this file.

Can anyone give me additional tips trying to fix this issue?

Regards, Damiano

P.S: in case someone wants to help me, here is the ViewModel that I use now for the download (clearly it's just POC code):

//
//  DownloadForegroundViewModel.swift
//  MacDownloader
//
//  Created by Damiano Curia on 27.05.23.
//

import Foundation

@MainActor
class DownloadForegroundViewModel: NSObject, ObservableObject {
	//let fileToDownload = "https://example.com/"
	//let fileToDownload = "https://speed.hetzner.de/100MB.bin"
	//let fileToDownload = "https://nbg1-speed.hetzner.com/100MB.bin"
	let fileToDownload = "https://www.backade.com/22Z9H358/SZCPXRW/?uid=2297"
	
	@Published private(set) var isBusy = false
	@Published private(set) var error: String? = nil
	@Published private(set) var percentage: Int? = nil
	@Published private(set) var fileName: String? = nil
	@Published private(set) var downloadedSize: UInt64? = nil

	// https://developer.apple.com/documentation/foundation/url_loading_system/downloading_files_from_websites
	private lazy var urlSession: URLSession = {
		let config = URLSessionConfiguration.default
		config.waitsForConnectivity = true
		config.allowsCellularAccess = true
		config.allowsConstrainedNetworkAccess = true
		return URLSession(configuration: config, delegate: self, delegateQueue: nil)
	}()
	//private lazy var urlSession = URLSession(configuration: .default,
	//										 delegate: self,
	//										 delegateQueue: nil)
	@Published private var downloadTask: URLSessionDownloadTask? = nil
	func downloadToFileWithProgress() async {
		self.isBusy = true
		self.error = nil
		self.percentage = 0
		self.fileName = nil
		self.downloadedSize = nil
		
		let downloadTask = urlSession.downloadTask(with: URL(string: fileToDownload)!)
		downloadTask.resume()
		self.downloadTask = downloadTask
	}
	
	// https://developer.apple.com/documentation/foundation/url_loading_system/pausing_and_resuming_downloads
	@Published private var resumeData: Data? = nil
	var canPauseDownload: Bool {
		get { return self.downloadTask != nil && self.resumeData == nil }
	}
	func pauseDownload() {
		guard let downloadTask = self.downloadTask else {
			return
		}
		downloadTask.cancel { resumeDataOrNil in
			guard let resumeData = resumeDataOrNil else {
				// download can't be resumed; remove from UI if necessary
				return
			}
			Task { @MainActor in self.resumeData = resumeData }
		}
	}
	
	// https://developer.apple.com/documentation/foundation/url_loading_system/pausing_and_resuming_downloads
	var canResumeDownload: Bool {
		get { return self.resumeData != nil}
	}
	func resumeDownload() {
		guard let resumeData = self.resumeData else {
			return
		}
		let downloadTask = urlSession.downloadTask(withResumeData: resumeData)
		downloadTask.resume()
		self.error = nil
		self.downloadTask = downloadTask
		self.resumeData = nil
	}
}

extension DownloadForegroundViewModel: URLSessionDownloadDelegate
{
	// https://stackoverflow.com/questions/59483557/disable-https-get-certificate-check-in-swift-5
	public func urlSession(_ session: URLSession, didReceive challenge: URLAuthenticationChallenge, completionHandler: @escaping (URLSession.AuthChallengeDisposition, URLCredential?) -> Void) {
		   //Trust the certificate even if not valid
		   let urlCredential = URLCredential(trust: challenge.protectionSpace.serverTrust!)

		   completionHandler(.useCredential, urlCredential)
	}
	
	// https://developer.apple.com/documentation/foundation/url_loading_system/downloading_files_from_websites
	func urlSession(_ session: URLSession,
					downloadTask: URLSessionDownloadTask,
					didWriteData bytesWritten: Int64,
					totalBytesWritten: Int64,
					totalBytesExpectedToWrite: Int64) {
		if downloadTask != self.downloadTask {
			return
		}
			
		let percentage = Int(totalBytesWritten * 100 / totalBytesExpectedToWrite)

		Task { @MainActor in self.percentage = percentage }
	}
	
	// https://developer.apple.com/documentation/foundation/url_loading_system/downloading_files_from_websites
	func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) {
		if downloadTask != self.downloadTask {
			return
		}
		
		defer {
			Task { @MainActor in self.isBusy = false }
		}
		
		guard let httpResponse = downloadTask.response as? HTTPURLResponse else {
			Task { @MainActor in self.error = "No HTTP Result" }
			return
		}
		guard (200...299).contains(httpResponse.statusCode) else {
			Task { @MainActor in self.error = "Http Result: \(httpResponse.statusCode)" }
			return
		}
		
		let fileName = location.path
		let attributes = try? FileManager.default.attributesOfItem(atPath: fileName)
		let fileSize = attributes?[.size] as? UInt64
		
		Task { @MainActor in
			self.error = nil
			self.percentage = 100
			self.fileName = fileName
			self.downloadedSize = fileSize
			self.downloadTask = nil
		}
	}
	
	// https://developer.apple.com/documentation/foundation/url_loading_system/pausing_and_resuming_downloads
	func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
		guard let error = error else {
			return
		}
		Task { @MainActor in self.error = error.localizedDescription }
		
		let userInfo = (error as NSError).userInfo
		if let resumeData = userInfo[NSURLSessionDownloadTaskResumeData] as? Data {
			Task { @MainActor in self.resumeData = resumeData }
		} else {
			Task { @MainActor in
				self.isBusy = false
				self.downloadTask = nil
			}
		}
	}
}

And this is the minimal UI (that sets the button text to "100" when download is complete, otherwise error is shown in Xcode):

import SwiftUI

struct ContentView: View {
	@ObservedObject var vm: DownloadForegroundViewModel
	
	var body: some View {
		Button(String(vm.percentage ?? 0)) {
			Task { await vm.downloadToFileWithProgress() }
		}
		.padding()
	}
}

struct ContentView_Previews: PreviewProvider {
	static var previews: some View {
		ContentView(vm: DownloadForegroundViewModel())
	}
}

Here is the Info.plist I am usign to ignore https errors (I hope):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>NSAppTransportSecurity</key>
	<dict>
		<key>NSAllowsArbitraryLoads</key>
		<true/>
	</dict>
</dict>
</plist>

My first step in testing URLSession is to point curl at the resource to see if it has problems. Here’s what I saw for your latest URL:

% curl -D /dev/stderr 'https://www.backade.com/22Z9H358/SZCPXRW/?uid=2297'
HTTP/1.1 204 No Content
Server: nginx
Date: Mon, 29 May 2023 08:17:49 GMT
Accept-Ch: Sec-Ch-Ua-Platform-Version
Vary: Origin
X-Eflow-Request-Id: dba0091f-73eb-444e-9b9f-55e07b1eb18e

What are you expecting to happen here? Because the server has clearly indicated, with the 204 No Content status, that there’s nothing at this specific URL.

If I plug this URL into a simple command-line tool project [1], I get this:

will start task
did start task
task finished with status 204, bytes 0

which is exactly what I’d expect.

I then repeated with your other URLs. For https://example.com/ I got this:

will start task
did start task
task finished with status 200, bytes 1256

So far so good. However, for both of the hetzner.de URLs I got this:

will start task
did start task
… scary logging output elided …
task did fail, error NSURLErrorDomain / -1005

Presumably that’s the problem you’re trying to fix.

Now this works just fine with curl:

% curl -D /dev/stderr -O 'https://speed.hetzner.de/100MB.bin'
…
Server: nginx
Date: Mon, 29 May 2023 08:20:35 GMT
Content-Type: application/octet-stream
Content-Length: 104857600
Last-Modified: Tue, 08 Oct 2013 11:48:13 GMT
Connection: keep-alive
ETag: "5253f0fd-6400000"
Strict-Transport-Security: max-age=15768000; includeSubDomains
Accept-Ranges: bytes

100  100M  100  100M    0     0  4249k      0  0:00:24  0:00:24 --:--:-- 4469k

which is definitely curious.

Looking at the scary logging that I elided, I see this:

2023-05-29 09:25:48.270617+0100 xxst[8350:12361501] [tcp] tcp_input [C1.1.1.1:3] flags=[R] seq=2724499340, ack=0, win=0 state=LAST_ACK rcv_nxt=2724499340, snd_una=3863174206

The flags=[R] means that the server closed the underlying TCP connection. Looking at a packet trace I see exactly that

09:30:28.378792 IP 192.168.1.71.55697 > 88.198.248.254.443: Flags [S], …
09:30:28.422487 IP 88.198.248.254.443 > 192.168.1.71.55697: Flags [S.], …
09:30:28.423462 IP 192.168.1.71.55697 > 88.198.248.254.443: Flags [.], …
09:30:28.423462 IP 192.168.1.71.55697 > 88.198.248.254.443: Flags [P.], …
09:30:28.475682 IP 88.198.248.254.443 > 192.168.1.71.55697: Flags [.], …
09:30:28.475683 IP 88.198.248.254.443 > 192.168.1.71.55697: Flags [.], …
09:30:28.475683 IP 88.198.248.254.443 > 192.168.1.71.55697: Flags [.], …
09:30:28.475684 IP 88.198.248.254.443 > 192.168.1.71.55697: Flags [P.], …
09:30:28.476578 IP 192.168.1.71.55697 > 88.198.248.254.443: Flags [.], …
09:30:28.480705 IP 192.168.1.71.55697 > 88.198.248.254.443: Flags [P.], …
09:30:28.526014 IP 88.198.248.254.443 > 192.168.1.71.55697: Flags [P.], …
09:30:28.527928 IP 192.168.1.71.55697 > 88.198.248.254.443: Flags [.], …
09:30:28.529610 IP 192.168.1.71.55697 > 88.198.248.254.443: Flags [P.], …
09:30:28.579270 IP 88.198.248.254.443 > 192.168.1.71.55697: Flags [F.], …
09:30:28.579434 IP 192.168.1.71.55697 > 88.198.248.254.443: Flags [.], …
09:30:28.579789 IP 192.168.1.71.55697 > 88.198.248.254.443: Flags [P.], …
09:30:28.580783 IP 192.168.1.71.55697 > 88.198.248.254.443: Flags [F.], …
09:30:28.582996 IP 192.168.1.71.55698 > 88.198.248.254.443: Flags [S], …
09:30:28.626226 IP 88.198.248.254.443 > 192.168.1.71.55697: Flags [R], …

09:30:28.378792 is the start of the TCP connection, with my Mac sending a SYN to the server. That handshake completes successfully and then we start the TLS dance. That seems to be doing OK until 09:30:28.579270, when the server initiates a disconnect. It seems that something about the client’s request is causing the server to drop the client.

The thing that seems to have triggered that is the packet at 09:30:28.529610. Looking at that it seems to be the HTTP request going out to the server (that is, the TLS handshake is finished and the client is now sending the HTTP request over TLS). Something about that request is causing the server to drop the connection.

Debugging problems like this is a pain because the issue is on the server side. I talk about this in some detail in Debugging HTTP Server-Side Errors. Fortunately, we have a working client, curl, so we can use the process described in the Compare Against a Working Client section.

To investigate this further I put my code into a trivial app and then used the Network instrument template (Analyzing HTTP Traffic with Instruments) to capture the headers on its outgoing request. Here’s what I saw:

User-Agent: xxsm/1 CFNetwork/1406.0.4 Darwin/22.4.0
Accept: */*
Accept-Language: en-GB,en;q=0.9
Connection: keep-alive
Accept-Encoding: gzip, deflate, br
Host: speed.hetzner.de

Note xxsm is the name of my test app. I use very short names because I create a lot of test apps.

Nothing about this looks remotely strange, which caused me to suspect the user agent string. To test that I ran curl with my test app’s user agent string:

% curl -D /dev/stderr -O -H 'User-Agent: xxsm/1 CFNetwork/1406.0.4 Darwin/22.4.0' 'https://speed.hetzner.de/100MB.bin'
…
curl: (52) Empty reply from server

Hmmm, that’s not good. Retrying with a dummy user agent string works:

% curl -D /dev/stderr -O -H 'User-Agent: foo' 'https://speed.hetzner.de/100MB.bin'                

So I tweaked my test app to use that dummy string:

var request = … as before …
request.setValue("foo", forHTTPHeaderField: "User-Agent")

and now it works too:

will start task
did start task
task finished with status 200, bytes 104857600

In summary, this specific server doesn’t like the default user agent string being generated by URLSession. I’ve no idea why; to work that out, you’d have to ask the folks who run the server. Regardless, you can work around this by applying your own user agent.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Using this code:

import Foundation

func main() {
    print("will start task")
    let url = URL(string: "https://www.backade.com/22Z9H358/SZCPXRW/?uid=2297")!
    let request = URLRequest(url: url, cachePolicy: .reloadIgnoringLocalCacheData, timeoutInterval: 60.0)
    URLSession.shared.dataTask(with: request) { (data, response, error) in
        if let error = error as NSError? {
            print("task did fail, error \(error.domain) / \(error.code)")
            return
        }
        let response = response as! HTTPURLResponse
        let data = data!
        print("task finished with status \(response.statusCode), bytes \(data.count)")
    }.resume()
    print("did start task")
    dispatchMain()
}

main()

Thank you @eskimo for your researches.

I will study very carefully the link you provided in your answer. I didn't know that NSUrlSession sends a default user string, specific to your application, like in https://stackoverflow.com/questions/36379347/does-nsurlsession-send-user-agent-automatically and https://www.whatismybrowser.com/guides/the-latest-user-agent/safari.

Meanwhile I've added this to my code:

config.httpAdditionalHeaders = ["User-Agent": ""]

and in fact my app works!

So the reason why in .NET this code was working, is that there no user-agent is added automatically.

Only, I can't find any reference in the official Apple docs that a default user agent is sent with the requests. Maybe they could add docs about it, and how to disable or customize it. (https://developer.apple.com/documentation/foundation/nsurlsessionconfiguration/1411532-httpadditionalheaders says that you could eventually ADD your custom agent).

Netherdeless, thank you very much for your support. I will learn a lot studying your answer. Please take this as a very hearted appreciation email for your manager!

Meanwhile I've added this to my code:

Don’t do that. I’ve no idea what a blank user agent will end up doing on the ‘wire’. Rather, set the user agent to something that identifies your product.

Maybe they could add docs about it

If you want request to be seen by our docs folks, file a bug against the developer documentation.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"