I'd like to announce that I can consistently reproduce the issue on demand.
When you open Audio MIDI Setup app, select the speaker you're currently listening to, then change the format from 2ch 32bit float to 16bit integer. At this point the audio will sound terrible. Wait a second and select 2ch 32bit. Notice that probably half a second after the selection is made, and the audio changes you'll hear a popping sound. This is the exact same popping sound I've been hearing randomly.
Caveat: It will pop about 90% of the time. On rare occasions it doesn't.
I'm not sure how this can be further diagnosed without digging into the driver. I'll look into this more in the future but for now at least we know how to reproduce it consistently and on demand.