RealityKit texture sampling behaviour

Hey there, I am working on an app that displays environmental data using PNG color channels to represent data ranges, which gets overlayed on a map. The sampled values aren't what I'm expecting though... for example an RGB value of 0x7f0000 (R = 0.5, G = 0, B = 0) would be seen as 0.21, 0, 0 in the shader. This basically makes it unusable if I'm trying to show scientific data... I'm half wondering if I am completely misunderstanding how sampling works in RealityKit / RealityComposerPro. Anybody have any idea why it works like this?

Actual result (chart labels added in photoshop):

Expected:

Red > 0.1 Shader Graph

Answered by Vision Pro Engineer in 818523022

Hi @imak604

When you say "for example an RGB value of 0x7f0000 (R = 0.5, G = 0, B = 0) would be seen as 0.21, 0, 0 in the shader", do you mean the shader renders the red channel (in the RealityKit scene) as 0.21 instead of 0.5? If so, light is to blame. If you want the surface to be unaffected by light, replace PreviewSurface with UnlitSurface.

If I misunderstood your question, please point me in the right direction so I can help.

Accepted Answer

Hi @imak604

When you say "for example an RGB value of 0x7f0000 (R = 0.5, G = 0, B = 0) would be seen as 0.21, 0, 0 in the shader", do you mean the shader renders the red channel (in the RealityKit scene) as 0.21 instead of 0.5? If so, light is to blame. If you want the surface to be unaffected by light, replace PreviewSurface with UnlitSurface.

If I misunderstood your question, please point me in the right direction so I can help.

Hey there, thanks for chiming in. The issue is that the texture sampler in the shader graph seems to see a (0.5,0,0) pixel in the texture image as something like (0.21,0,0) for example. The equivalent metal operation would be something like:

constexpr sampler s(address::clamp_to_edge, filter::linear); float4 s = tex.sample(s, float2(0.4,0.2)); // expected float4(0.5, 0, 0), got float4(0.21, 0, 0);

Does lighting affect the value of the texture sampler?

Here's a more direct example of the issue... in the image below, the upper gradient is a material that uses a textured image, the bottom gradient is generated from within the shader. The two gradients should be identical but the image textured material is noticably darker in the middle ranges. I imagine there's a reason for this, anybody know why the texture sampler behaves like this?

gradient image texture:

gradient shader graph:

RealityKit texture sampling behaviour
 
 
Q