Swift: Project Points from 3D to 2D in ARKit

Question

Created Jan ’23

Replies 2

Boosts 0

Participants 3

I am trying to project rawFeaturePoints from ARFrame into the 2D Image plane, and get the distances from the points to the camera, but when I plot them using Python, they don't make sense. Here's my attempt:

func get2DPoints(arView: ARSCNView, arCamera: ARCamera, rawFeaturePoints: [vector_float3], filename: String) {
    file_path = filename
    let cameraPosition = arCamera.transform.columns.3
    let screenSize = UIScreen.main.bounds
    
    for point in rawFeaturePoints {
        let scn_point = SCNVector3(point)
        // get the pixel coordinates
        let projected_point = arView.projectPoint(scn_point)
        print("Projected Point: \(projected_point)")
        if projected_point.x > 0 && projected_point.y > 0 {
            // Calculating distance between a raw feature point and the camera
            let distance_from_camera = sqrt(pow(cameraPosition.x - scn_point.x, 2) + pow(cameraPosition.y - scn_point.y, 2) + pow(cameraPosition.z - scn_point.z, 2))
            let feature_point = FeaturePoint(x: projected_point.x, y: projected_point.y, z: distance_from_camera)
            featurePoints.append(feature_point)
        }
        
    }
    print("Finished processing rawFeaturePoints")
    
    
}

I am getting the points from frame.rawFeaturePoints?.points and then projecting them. The points mostly make sense, but when I try to plot them in Python on top of the frame, they are not correct. My understanding is that projectPoint already projects the point to the 2D Image plane so there would be no need for further transforms. It seems to be some that the origin could be different in python and ARKit but both are the same (as far as I know, top left). Any ideas?

Boost

Answer 1

bubba_byte OP

Jan ’23

I apologize; I know nothing about ARkit - but a couple of things:

you'll need to look at the docs to see exactly what space 'projectPoint' projects world space into to. Is it projecting points to some kind of normalized device coordinates? (NDC's typically have a range of -1 to 1) Or is it projecting points to some kind of device coordinates (to which it would need to know the device parameters - size and shape). Or, is it projecting points to some kind of eye coordinate system? You'll need to look at the docs to know
what the heck are you doing with setting up each feature points? You're using the x and y coordinates of the projected point, but the z coordinate is a world space distance - presumably the distance from the camera to each UNprojected point. From strictly a 'visualization' point of view, that doesn't make a whole lot of sense. Are you using the z coordinate when trying to plot the feature points?
Plot them in Python 'on top of the frame'? That doesn't say a lot. When you plot something, you tell the tool 'how to plot' - i.e., where the origin is, the scale of the axis, etc, etc.

0

Answer 2

samsonngov OP

Mar ’23

Hi, Did you find the cause and solution ? I am also trying to extract the 3D mesh point coordinated to project it on the 2D picture as well

0