Got the same issue.. frankly the whole "auto switching between devices" is 50% disaster for me. If I've got something like a Slack call going and pop in my AirPods Pro the right comes on but the Mircophone stays set (in Slack) to my MBP microphone. Change that to AirPods and it livens up both AirPods. No call going on.. or music. Pop in my AirPods and play music, only get audio in the right. Open sound Preferences and the balance slider is over to the right, move it to centre and it's fine.
Happens every single time. Had a look a the audio MIDI setup, can see the balance issue replicated in there, but doesn't seem to change anything.
Pre-Big Sur I had to switch manually but it worked.