I've encountered the same problem in tvOS 13, and found the following: If you change the size of the AVPlayerViewController's view, for some sizes it works, and for some sizes it doesn't.
The video stream manifest had weird pixel resolutions for the streams that didn't work.
For example one stream had a resolution of 720/406, where if you want a ratio of 16/9 it should be 720/405.