Chaining multiple URLRequests and get next JSON page of results?

Question

Created Oct ’20

Replies 1

Boosts 0

Views 1.6k

Participants 2

Hi!
I'm looping over an array of strings and I need to make a URLRequest call on each element (id).
For each JSON there could be a href linking to other pages with additional data in it.
How can I chain multiple URLRequests?

Currently, in parse(data: data), I check to see if the href key is not null and then I call retrievePages(with url: URL) which is similar to retrieveRelease(). It gets messy. Duplicate code. I get somehow the results I want but I don't think it's the best approach at all.
How could I get the next page of results? Considering that I'm also inside a loop in retrieveRelease()

Code Block func retrieveRelease() {
dataTask?.cancel()
for id in idArray {    
	idUrl = URL(string: "url+id")!
	let request = NSMutableURLRequest(url: idUrl)
  
	let session = URLSession.shared
	dataTask = session.dataTask(with: request as URLRequest) { data, response, error in
		if let error = error as NSError? {
          return 
		} else if let httpResponse = response as? HTTPURLResponse, httpResponse.statusCode == 200 {
		if let data = data {
			self.parse(data: data)
			DispatchQueue.main.async {
       	//update views
				}
            return
     	}
		}
	}
	dataTask?.resume()
}
}
}

Answered by DTS Engineer in 639453022

The best way to do this depends on your specific circumstances but I generally use an array to hold the list of pending URLs. I populate that array with the initial set of URLs. When a parse completes it adds the resulting URLs (if any) to that array. It then looks to see if the array is empty and, if not, starts the next request.

I usually use an object to manage all of this but you can implement a simple version using closures. For example:

Code Block  func fetchURLs(_ initialURLs: [URL], completionHandler: @escaping () -> Void) {
    var pendingURLs = initialURLs
    func startNext() {
        guard let url = pendingURLs.first else {
            completionHandler()
            return
        }
        pendingURLs.removeFirst()
        loadAndParse(url: url) { parsedURLs in
            pendingURLs.append(contentsOf: parsedURLs)
        }
        startNext()
    }
    startNext()
}

This assumes a helper function, func loadAndParse(url: URL, completionHandler: @escaping ([URL]) -> Void), that loads and parses the URL asynchronously.

The biggest drawback with this simplistic approach is that everything is serialised: The requests never overlap on the ‘wire’. This will significantly slow things down if you have a lot of requests to run over a high-latency link.

Running requests in parallel requires more state management because it’s best to limit the parallelism (you don’t want to run thousands of requests in parallel). This is when I start using an object to manage this, with a property the counts the number of currently running requests.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

Boost

Answer 1

DTS Engineer OP

Apple

Oct ’20

Accepted Answer

The best way to do this depends on your specific circumstances but I generally use an array to hold the list of pending URLs. I populate that array with the initial set of URLs. When a parse completes it adds the resulting URLs (if any) to that array. It then looks to see if the array is empty and, if not, starts the next request.

I usually use an object to manage all of this but you can implement a simple version using closures. For example:

Code Block  func fetchURLs(_ initialURLs: [URL], completionHandler: @escaping () -> Void) {
    var pendingURLs = initialURLs
    func startNext() {
        guard let url = pendingURLs.first else {
            completionHandler()
            return
        }
        pendingURLs.removeFirst()
        loadAndParse(url: url) { parsedURLs in
            pendingURLs.append(contentsOf: parsedURLs)
        }
        startNext()
    }
    startNext()
}

This assumes a helper function, func loadAndParse(url: URL, completionHandler: @escaping ([URL]) -> Void), that loads and parses the URL asynchronously.

The biggest drawback with this simplistic approach is that everything is serialised: The requests never overlap on the ‘wire’. This will significantly slow things down if you have a lot of requests to run over a high-latency link.

Running requests in parallel requires more state management because it’s best to limit the parallelism (you don’t want to run thousands of requests in parallel). This is when I start using an object to manage this, with a property the counts the number of currently running requests.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

1