Using AVPlayerView with a running movie, the first scroll event (either direction), and only the first, resets the playhead to the start of a video, if it is the initial event the AVPlayerView receives - after that, the scroll wheel works as expected. This is a PITA when, e.g. after a third of the way through a movie, bumping the mouse wheel sends you back to the beginning. The bug also occurs notwithstanding the playhead is advanced in code with seekToTime(), or the scroll event is simulated with Keyboard Maestro - a wonderfully useful app for debugging and much more (no relation with the developer). Again, only for the first scroll event, and nothing else (see below) has happened.
In AppKit, it is possible to intercept scroll wheel events. But one truly useful feature of AVPlayerView is mapping scroll events to the left-/right-arrow keys which provide frame-wise forward/reverse movement. You have to give that up if you intercept scroll events, and there's no way AFAIK to otherwise pass to AVPlayerView an "advance- or reverse-frame" message, at least not exactly the same amount as the scroll wheel does, except on first use. Similarly, I can't figure out a way to send a "forward-one-scroll-unit" event on initial video start to suppress the bug.
The AVPlayerView j-k-l shortcuts might be useful for FCP users, but I could do without - alternatively, they should be remappable. If you use these keys before scrolling, or manually advance the playhead with UI, the bug evaporates. The only way to replicate it is to start playing video with the play button (or in code), then wait long enough that you can discern if the playhead resets, then triggering the scroll wheel (or double-swipe on trackpad) - either direction. Since the bug occurs only with the first scroll event, you have to close the AVPlayerView and repeat to replicate. A long debug cycle...
Since the bug occurs with both AppKit and SwiftUI, I'm guessing that SwiftUI's PlayerView is just a convenience wrapper of the AppKit version. SwiftUI doesn't offer the developer an equivalent level of event control, which makes the problem even worse.
This report applies to macOS; YYMV with iOS. I have seen similar reports earlier by searching for "Calling AVPlayer seekToTime: results in incorrect scrollWheel behavior".