This fixed it for me:
"I could resolve this issue finally with removing additional audio plugins in the folder /Library/Audio/Plug-Ins."
I just deleted everything in there, restarted and hey presto. Airpods and Jaybirds finally working perfectly and holding connection with Big Sur.