I don't have a solution, but I have the exact same issue on a brand new M4 Pro MacBook Pro 14 (the 1TB model with the full CPU/GPU). My old M2 Pro MBP didn't have this issue, but the M4 Pro does. I have reported the issue to Apple as well, so you are not alone.
Just curious what hardware you're running it on.