Same thing happening to me, M1 Pro MacBook 16" with an external USB DAC. Similar log entries as above:
Audio IO Overload thread: 773922 inputs: '<private>' outputs: '<private>' cause: 'Unknown' prewarming: no recovering: no
Does seem to improve if I reduce memory pressure, but that's not a workable long term solution, I get yellow memory pressure from firing up all the tools I need.