Texture Write Rounding

Question

caijohn OP

Created Sep ’23

Replies 3

Boosts 0

Participants 2

Hello,

I used outTexture.write(half4(hx,0,0,0),uint2(x, y)) to write pixel value to texture and then read back by blitEncoder copyFromTexture to a MTLBuffer, but the integer value read from MTLBUffer is not as expected, for half value which less than 128/256, I got expected value. but got small value with half value huge than 128/256, for examples,

127.0/256; ==> 127

128.0/256; ==> 128

129.0/256; ==> 129

130.0/256; ==> 130

131.0/256; ==> 131

Any thoughts?

Thanks Caijohn

Answered by endecotp in 763728022

Your problem isn't completely clear to me; are you sure you posted the correct numbers?

The range of an 8-bit integer is 0 to 255. When those values are scaled to a floating-point value such as a colour component in the range 0.0 to 1.0, then 255 needs to map to 1.0. So you need to divide by 255, not 256. Does that explain what you are seeing?

Boost

Answer 1

caijohn OP

Sep ’23

Sorry for typo, re-edited example

127.0/256; ==> 127 128.0/256; ==> 128 129.0/256; ==> 128 130.0/256; ==> 129 131.0/256; ==> 130

0

Answer 2

endecotp OP

Sep ’23

Accepted Answer

Your problem isn't completely clear to me; are you sure you posted the correct numbers?

The range of an 8-bit integer is 0 to 255. When those values are scaled to a floating-point value such as a colour component in the range 0.0 to 1.0, then 255 needs to map to 1.0. So you need to divide by 255, not 256. Does that explain what you are seeing?

0

Answer 3

caijohn OP

Sep ’23

Thanks a lot! Yes, you're right. I changed it to be divided by 255, now I got expected integer values.

0