Same problem here, for years, on various MacBooks with RME UCX interface over USB. Now using M1 Pro, latest OS, same problem. Can't believe Apple haven't fixed this - my bloody iPad has more reliable audio.
Tried emailing RME support about it but they just say "it's an Apple problem - users on Linux or Windows don't have it". Maddening!
Anyone from Apple actually read these forums? Anyone got anywhere with Apple support? Last time I tried they were all about resetting CMOS or something, but M1 Macs don't even have that!