This returns (x, y) but it is the case that x > 390.0 and y > 844.0 sometimes. Furthermore, sometimes, x and y are negative values. I am confused about how the function returns values that are greater than the viewportsize we gave. I am also confused by the values themselves are negative.
Values greater than the viewport size, as well as values smaller than zero are to be expected. Assuming you have provided all of the correct parameters to the projection method, values outside of the viewport mean that the 3D point in question has projected to a point that is not in the current viewport. In other words, it is a point on the same projection plane as the viewport, but, because the 3D point is not actually visible in the viewport, it has projected to a point that is outside of the viewport. If the 3D point truly is visible in the viewport, and you are getting a projected point that is outside of the viewport, then this indicates that you have set something up incorrectly.
My goal is to take a point, p, where p = (x,y,z) and convert it to p' = (x',y') where (x',y') are in range 0 - 1 as that corresponds to how pixels are organized on the phone screen.
The ARCamera projectPoint method returns a point based on the viewportSize that you have provided (typically, you would want this viewportSize to match the size of the 2D destination that you are trying to render the results in. That might be SpriteKit's coordinate space for example, or it might be a UIView, CALayer, etc). You could of course normalize the results after projection if you wanted to.
Additionally, ARCamera projectPoint assumes an origin in the top-left corner, so you need to make sure you account for this (for example, SpriteKit by default has an origin in the lower-left corner, so you would need to flip the Y-coordinate of the projected point to properly display it in SpriteKit, assuming SpriteKit's default origin is being used).
