TabasKo, I hear you. When a new expensive supposedly high-quality laptop craps out on a basic function like USB audio, it makes me feel I've been scammed.

Btw, unfortunately the above fix didn't help.

Unfortunately it does not seem to be limited to this particular process but to all Intel type processes running.

Yes, I think I was too optimistic in this discovery. It could be that the these particular Intel processes I'm running cause more memory/CPU pressure than native "Apple" processes and therefore cause audio issues more quickly since they seem to be correlated but for sure they are not the whole story, as you say. It certainly is maddening and I'd really love to hear a serious comment from Apple addressing this issue.

TabasKo, I don't seem to have Core Audio Driver running as an Intel process, but CarbonComponentScannerXPC seems to be constantly there.

There seem to be no solutions except not to run other apps while listening to music. The only real solution is for Apple to fix their shit.

Indeed I'm running fewer software processes on the Mac Mini, so that might be a contributing factor as well.

Reboot also helped here. "Have you tried turning it off and on again?"