After some more investigation it seems that every time this happens the layer's videoRect is (0, 0, 0, 0).
Normally it is updated when the layer receives a KVO for "currentItem.presentationSize", but for the non working case this doesn't happen.
But logging currentItem.presentationSize form my code does show it has a size but for some reason the value change notification was not triggered...
Suggestions are welcome :-)
It looks like this is infact related to not using the main queue for setting the player to the layer and calling AVPlayer.replaceCurrentItemWithPlayerItem().
And it is documented the AVPlayer functions should be synced with the observer queue, which is the main queue.
Fixing this seems to have cured the issue.