Chaining NSOperation : Pass result from an operation to the next one

I've been looking for a way to pass results for chained

NSOperation
, for example lets assume we have 3 operations chained: 1.
Operation1
to download
JSON
data from server 2.
Operation2
to parse & model JSON received 3.
Operation3
to download user images

so Op3 would be dependent on Op2, which is dependent on Op1, but i'm looking for way to pass results from Op1 -> Op2, then from Op2 -> Op3 as:

[operation1 startWithURL:url];
[operation2 parseJSONfromOp1IntoModel:JSONData];
[operation3 downloadUserImagesForUser: UserModelObject];



and nesting blocks doesn't seem to be a clean readable solution, any idea?

Accepted Reply

There’s a bunch of different ways to design this. The basic tools at your disposal include:

  • operation dependencies

  • operation completion blocks (the

    completionBlock
    property)
  • operation data source and delegate callbacks

  • adapter block properties

  • adapter block operations

The biggest challenge, IMO, is that of coupling. You want to avoid coupling a consumer operation to its producer operations for two reasons:

  • To facilitate operation reuse. One goal of using NSOperation is to build up a library of reusable parts that you can apply in a variety of circumstances. If your operations are tightly coupled, you won’t hit this goal.

    Let’s consider the example you outlined. Imagine that your ‘parse’ operation (the consumer operation) was tightly coupled to your ‘fetch’ operation (the producer operation). That would severely limit the usefulness of the parse operation. For example, if the parse operation started out in an old project that used an NSURLConnection-based fetch operation, it would be hard to move it to a new project that use an NSURLSession-based fetch operation. Or a CloudKit-based fetch operation. Or whatever.

  • To allow for testing. Again considering your example: if the parse operation is tightly coupled to the fetch operation, you can’t test it as an independent unit.

I also try to avoid coupling my operations to any operation infrastructure. That way I can move operations between projects without having to bring the infrastructure along for the ride.

One approach I looked at was to use NSOperation’s completion block facility to push results from the producer. Alas, it’s not as helpful as it could be. The issue is that the completion block runs when the operation is finished, so it runs asynchronously with respect to the starting of dependent operations. That means you can’t use the completion block to transfer data from one operation to its dependency.

After zenning on this for multiple years, I now lean towards using adapter block operations. Such operations can shuffle data from the producer operations to the consumer operation, adapting the data as appropriate for the context.

IMPORTANT The adapter block operation is dependent on the consumer operations. It won’t run until they’ve finished, so it can safely look at their results. Also, the producer operation is dependent on the adapter block operation, so it won’t start until the adapter block operation has finished its shuffling.

So, the code might look like this:

class FetchOperation: NSOperation {
    var url: NSURL?
    var data: NSData?
    var error: NSError?

    convenience init(url: NSURL) {
        self.init()
        self.url = url
    }

    // I've omitted the intricacies of implementing an async
    // network operation because that’s too complex a topic to
    // cover here.  The only thing to watch out for is that, for
    // adapter operations to work in a fully composible fashion,
    // the start of the async code must check self.error and fail
    // immediately if it’s set.
}

class ParseOperation: NSOperation {
    var data: NSData?
    var error: NSError?

    convenience init(data: NSData) {
        self.init()
        self.data = data
    }

    override func main() {
        if self.error == nil {
            if let data = self.data {
                self.parseData(data)
            } else {
                // We have no data to parse; this shouldn't happen.
                fatalError()
            }
        }
    }

    func parseData(data: NSData) {
        // ... parse the data ...
    }
}

let fetchOp = FetchOperation(url: url)
let parseOp = ParseOperation()
let adapterOp = NSBlockOperation(block: {
    parseOp.data  = fetchOp.data
    parseOp.error = fetchOp.error
})

adapterOp.addDependency(fetchOp)
parseOp.addDependency(adapterOp)

networkQueue.addOperation(fetchOp)
computeQueue.addOperation(adapterOp)
computeQueue.addOperation(parseOp)

Share and Enjoy

Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Replies

I would store your data in an external holding object with an identifier so that it can be accessed by operations when their dependent operations complete.


You could also add a final operation that depends on the other operations in the sequence which clears out the working data.

There’s a bunch of different ways to design this. The basic tools at your disposal include:

  • operation dependencies

  • operation completion blocks (the

    completionBlock
    property)
  • operation data source and delegate callbacks

  • adapter block properties

  • adapter block operations

The biggest challenge, IMO, is that of coupling. You want to avoid coupling a consumer operation to its producer operations for two reasons:

  • To facilitate operation reuse. One goal of using NSOperation is to build up a library of reusable parts that you can apply in a variety of circumstances. If your operations are tightly coupled, you won’t hit this goal.

    Let’s consider the example you outlined. Imagine that your ‘parse’ operation (the consumer operation) was tightly coupled to your ‘fetch’ operation (the producer operation). That would severely limit the usefulness of the parse operation. For example, if the parse operation started out in an old project that used an NSURLConnection-based fetch operation, it would be hard to move it to a new project that use an NSURLSession-based fetch operation. Or a CloudKit-based fetch operation. Or whatever.

  • To allow for testing. Again considering your example: if the parse operation is tightly coupled to the fetch operation, you can’t test it as an independent unit.

I also try to avoid coupling my operations to any operation infrastructure. That way I can move operations between projects without having to bring the infrastructure along for the ride.

One approach I looked at was to use NSOperation’s completion block facility to push results from the producer. Alas, it’s not as helpful as it could be. The issue is that the completion block runs when the operation is finished, so it runs asynchronously with respect to the starting of dependent operations. That means you can’t use the completion block to transfer data from one operation to its dependency.

After zenning on this for multiple years, I now lean towards using adapter block operations. Such operations can shuffle data from the producer operations to the consumer operation, adapting the data as appropriate for the context.

IMPORTANT The adapter block operation is dependent on the consumer operations. It won’t run until they’ve finished, so it can safely look at their results. Also, the producer operation is dependent on the adapter block operation, so it won’t start until the adapter block operation has finished its shuffling.

So, the code might look like this:

class FetchOperation: NSOperation {
    var url: NSURL?
    var data: NSData?
    var error: NSError?

    convenience init(url: NSURL) {
        self.init()
        self.url = url
    }

    // I've omitted the intricacies of implementing an async
    // network operation because that’s too complex a topic to
    // cover here.  The only thing to watch out for is that, for
    // adapter operations to work in a fully composible fashion,
    // the start of the async code must check self.error and fail
    // immediately if it’s set.
}

class ParseOperation: NSOperation {
    var data: NSData?
    var error: NSError?

    convenience init(data: NSData) {
        self.init()
        self.data = data
    }

    override func main() {
        if self.error == nil {
            if let data = self.data {
                self.parseData(data)
            } else {
                // We have no data to parse; this shouldn't happen.
                fatalError()
            }
        }
    }

    func parseData(data: NSData) {
        // ... parse the data ...
    }
}

let fetchOp = FetchOperation(url: url)
let parseOp = ParseOperation()
let adapterOp = NSBlockOperation(block: {
    parseOp.data  = fetchOp.data
    parseOp.error = fetchOp.error
})

adapterOp.addDependency(fetchOp)
parseOp.addDependency(adapterOp)

networkQueue.addOperation(fetchOp)
computeQueue.addOperation(adapterOp)
computeQueue.addOperation(parseOp)

Share and Enjoy

Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thank you @Eskimo for the detailed answer.


Actually i have 2 problems:

1. i have to use Async NSURLSession call to download JSON from the server, so should i just set the result to FetchOperation property and then read/copy it from the adapter ?


2.in WWDC15 Advanced operation session, it was all based on saving the result to Core data, therefore no need to pass data back & forth from operations to each other, meanwhile i need to pass them insted of saving to Core data, so do you think there's another way to do so ?


Thanks again.

1. i have to use Async NSURLSession call to download JSON from the server, so should i just set the result to FetchOperation property and then read/copy it from the adapter ?

Yes. That’s exactly what my code is showing.

2. in WWDC15 Advanced operation session, it was all based on saving the result to Core data, therefore no need to pass data back & forth from operations to each other, meanwhile i need to pass them insted of saving to Core data, so do you think there's another way to do so ?

There’s lots of ways to design this. I listed various options at the top of my first response, and that’s only some of them. The question is, what’s the best way to design this? I like my design because:

  • it eliminates coupling between operations

  • it doesn’t require any operation infrastructure

With regards the second point, if you look at Dave’s code you’ll see that it relies on a bunch of common infrastructure (for example, an NSOperationQueue subclass that calls

willEnqueue()
on the operation before it puts it into the queue, plus the
conditions
and
observers
stuff). That’s not necessarily a bad thing—in fact, I used to use a similar approach myself!—but these days I try to avoid that infrastructure because it makes it hard to move operations from one project to another.

Ultimately, and this is the standard answer for any design question, it’s really up to you.

Share and Enjoy

Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

.Hi Eskimo,


After read your post above, i try it with codes, but there still have a issue that I can't understand, please help me out.


codes like below:

- (void) doSomthing {
     NSInvocationOperation *op2 = [[NSInvocationOperation alloc] initWithTarget:self selector:@selector(op2Selector) object:nil];
     NSInvocationOperation *op3 = [[NSInvocationOperation alloc] initWithTarget:self selector:@selector(op3Selector) object:nil];
     [op2 addDependency:op3]; // Here can make sure op3Selector run and finish, then op2Selector run.
     [[NSOperationQueue mainQueue] addOperations:@[op2,op3] waitUntilFinished:NO];
}

- (void)op2Selector {
    NSURLRequest *request = [NSURLRequest requestWithURL:[NSURL URLWithString:@"https:"]];
    NSURLSessionTask *task = [[NSURLSession sharedSession] dataTaskWithRequest:request completionHandler:^(NSData * _Nullable data, NSURLResponse * _Nullable response, NSError * _Nullable error) {
        NSLog(@"%@", response);
    }];
    [task resume];
}
- (void)op3Selector {
    NSURLRequest *request = [NSURLRequest requestWithURL:[NSURL URLWithString:@"https:"]];
    NSURLSessionTask *task = [[NSURLSession sharedSession] dataTaskWithRequest:request
                                                             completionHandler:^(NSData * _Nullable data, NSURLResponse * _Nullable response, NSError * _Nullable error
        NSLog(@"response: %@, error: %@", response, error);
    }];
    [task resume];
}


But, what should I do to make sure after run completionHandler of NSURLSessionTask in op3Selector, then start NSURLSessionTask in op2Selector ? thanks.

AFAICT you’re very much in the weeds here.

op2
is an NSInvocationOperation, which is a standard operation. Thus
-op2Selector
acts like the
-main
method of a typical NSOperation subclass: when the
-main
method returns, the operation is marked as finished.

If you want to integrate async work (like an NSURLSession task) into the NSOperation world, you have to create an asynchronous (previously known as a concurrent) operation. This doesn’t finish until you explicitly tell it to finish. Once you do that, your operation will work properly with the NSOperation dependency mechanism.

My go-to sample code for async operations is the LinkedImageFetcher sample code. It’s a bit rusty these days, but I still think it shows the best approach.

For something more up-to-date, you can look at the sample code associated with WWDC 2015 Session 226 Advanced NSOperations.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Be careful that this pattern introduces a memory leak:


let fetchOp = FetchOperation(url: url)

let parseOp = ParseOperation()

let adapterOp = NSBlockOperation(block: {

parseOp.data = fetchOp.data // the block captures parseOp and fetchOp strongly here

parseOp.error = fetchOp.error

})


adapterOp.addDependency(fetchOp) //setting dependencies sets strong relationships

parseOp.addDependency(adapterOp)


You have to make sure you refer to the operations weakly within the NSBlockOperation...