VNTrajectoryObservation inconsistent observations

I have a pre-recorded video loaded into Xcode for testing and when I run the trajectory request on the video playing in the simulator the observations I get back are inconsistent and vary with every build and run. For example one session will produce no observations and the next time I run it will produce multiple observations. Are there any suggestions or tips to improving the consistency of the observations?