visionOS VR mode / eye tracking / dynamic foveated rendering

I think it's kind of essential to have eye tracking data available to apps in VR mode (with the user's permission).

The biggest problem I've observed is that Unity isn't able to implement dynamic foveated rendering without eye tracking data.

Without the eye tracking it's only possible to have fixed foveated rendering. That gives a performance boost to rendering, but it also makes it so it gets blurry for the user if they look to the side without turning their head.

I understand why it's a privacy issue to have apps tracking where the user is looking in the real world, but video passthrough is disabled in VR -- so it should be ok to enable eye tracking in VR (with the user's permission).

Unity already supports dynamic foveated rendering (with eye tracking) for other VR headsets, and Vision Pro has the best eye tracking -- so Vision Pro should definitely have the best dynamic foveated rendering in VR.

Up vote post of Hawaiianz Down vote post of Hawaiianz
169 views
Posted by
Hawaiianz
Reply
Add a Comment

Replies

I asked about this in the Unity visionOS discussion, and one of their engineers confirmed that it is in fact the obstacle.

They simply can't implement dynamic/gazed foveated rendering without the eye tracking data. It's not even on the roadmap for Unity because there isn't anything they can do until Apple makes the eye tracking data available to Unity's rendering pipeline.

The way to solve this would be for visionOS to ask the user if they want to make their eye tracking data available in VR (this is how it currently is with hand tracking data). Then Unity can use dynamic foveated rendering (which is ideal, and already works in AR), or fall back to fixed foveated rendering if the user wants to keep eye tracking private.

That will make it so people can see everything clearly in high resolution in VR -- and VR developers won't need to choose between heavy upscaling and fixed foveated rendering (which lowers resolution around the center of the view -- and requires people to turn their heads instead of their eyes to see clearly).

It would make the VR rendering really amazing.

Posted by
Hawaiianz
Up vote reply of Hawaiianz Down vote reply of Hawaiianz
Add a Comment