How to transform Vision body tracking coordinates

Question

I am using the new Vision VNDetectHumanBodyPoseRequest from Apple, but I am not sure why my mapped coordinates are incorrect with the front-facing camera.

My drawing code is correct as I can see the nodes, but they are in the completely wrong position. I tried setting different orientations in the VNImage request handler:

Code Block VNImageRequestHandler(cmSampleBuffer: sampleBuffer, orientation: .left)

The file is attached.

CameraViewController.swift

Thanks for the help!

1.4k

Posted by

LemonSpike

Reply

Add a Comment

Answer 1

I figured it out! The body needs to be further away from the camera: tracking performs very well for distances greater than 5m apart, even in horizontal movements or obscured joint positions.

Posted by

LemonSpike

Add a Comment

Answer 2

I fixed the skeleton, it's now visible, but there is considerable lag and jerkiness in the movement.

Code Block captureConnection?.videoOrientation = .portrait
...
let requestHandler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer, orientation: .down)
...
var recognizedPoints = recFacePoints.merging(recTorsoPoints) { (current, _) in current }
recognizedPoints = recognizedPoints.merging(recLeftArmPoints) { (current, _) in current }
recognizedPoints = recognizedPoints.merging(recRightArmPoints) { (current, _) in current }
recognizedPoints = recognizedPoints.merging(recLeftLegPoints) { (current, _) in current }
recognizedPoints = recognizedPoints.merging(recRightLegPoints) { (current, _) in current }

were the changes I made

Posted by

LemonSpike

Add a Comment

Answer 3

Are there any ways to reduce this jerkiness and improve performance? Thanks in advance.

Posted by

LemonSpike

Add a Comment

Answer 4

I figured it out! The body needs to be further away from the camera: tracking performs very well for distances greater than 5m apart, even in horizontal movements or obscured joint positions.

Posted by

LemonSpike

Add a Comment

How to transform Vision body tracking coordinates

Accepted Reply

Replies