Append buffers to an AVAssetWriterInputPixelBufferAdaptor *not* chronologically?

I'm appending CVPixelBufferRefs to an AVAssetWriterInputPixelBufferAdaptor which is connected to an AVAssetWriterInput to write to a file using AVAssetWriter.
So, I'm calling

Code Block 
[pixelAdaptor appendPixelBuffer:pxbuffer withPresentationTime:someCMTime];

and it works as it should.

Now, I'm doing this sequentially, so at CMTimeZero, I add the first frame, at time 0.5s I append another frame, at 1s another, and so on, and so on.

I'm wondering, do I have to do it sequentially, or can I do it randomly as well? For the sake of argument, the other way around? Like, append a buffer at 1s first, then at 0.5s and then at 0s?

The reason I'm asking is, I've tried, and it crashes, but I'm not sure if it crashes because I tried to append at random times (non-sequentially), or if there's another reason (like asynchronous appending)?

I'm only appending when the input tells me to (readyForMoreMediaData), on a serial dispatch queue.
The question is: Should non-sequential appending work, or is it by design that it doesn't?

Thank you,
Matthias