visionOS VR mode / eye tracking / dynamic foveated rendering

Question

Hawaiianz OP

Created Apr ’24

Replies 3

Boosts 1

Views 1.1k

Participants 2

I think it's kind of essential to have eye tracking data available to apps in VR mode (with the user's permission).

The biggest problem I've observed is that Unity isn't able to implement dynamic foveated rendering without eye tracking data.

Without the eye tracking it's only possible to have fixed foveated rendering. That gives a performance boost to rendering, but it also makes it so it gets blurry for the user if they look to the side without turning their head.

I understand why it's a privacy issue to have apps tracking where the user is looking in the real world, but video passthrough is disabled in VR -- so it should be ok to enable eye tracking in VR (with the user's permission).

Unity already supports dynamic foveated rendering (with eye tracking) for other VR headsets, and Vision Pro has the best eye tracking -- so Vision Pro should definitely have the best dynamic foveated rendering in VR.

Boost

Answer 1

Hawaiianz OP

May ’24

I asked about this in the Unity visionOS discussion, and one of their engineers confirmed that it is in fact the obstacle.

They simply can't implement dynamic/gazed foveated rendering without the eye tracking data. It's not even on the roadmap for Unity because there isn't anything they can do until Apple makes the eye tracking data available to Unity's rendering pipeline.

The way to solve this would be for visionOS to ask the user if they want to make their eye tracking data available in VR (this is how it currently is with hand tracking data). Then Unity can use dynamic foveated rendering (which is ideal, and already works in AR), or fall back to fixed foveated rendering if the user wants to keep eye tracking private.

That will make it so people can see everything clearly in high resolution in VR -- and VR developers won't need to choose between heavy upscaling and fixed foveated rendering (which lowers resolution around the center of the view -- and requires people to turn their heads instead of their eyes to see clearly).

It would make the VR rendering really amazing.

0

Answer 2

Hawaiianz OP

May ’24

I only mentioned Unity, but every other game engine and rendering system will be needing dynamic foveated rendering for VR also.

It’s a key optimization for VR, and it’s not really optional for such high resolution displays. Vision Pro already has everything needed to get it working completely, and making the real-time eye tracking available to VR rendering (with the user’s permission) is the only thing missing.

0

Answer 3

Vision Pro Engineer OP

Apple

Jun ’24

On visionOS, where you look stays private. This means that your app process does not have access to a person's gaze, for privacy reasons.

You can use RealityKit on visionOS to support dynamic foveation for your spatial content, including content in a full immersive space ("VR content"). Unity PolySpatial is built on top of RealityKit, so you can also use PolySpatial if you need dynamic foveation for your Unity content.

0