Same problem with MacBook Pro 13" 2017, and I tried these actions:
Wipe & install macOS Catalina, the problem was gone
Upgrade to Big Sur, the problem came back again
I guess the problem was caused by a software issue, since the brightness function is ok in Catalina.