Same here. I got a Razer Core X eGPU case with a Radeon RX 64, and it worked stably on Catalina. On Big Sur it used to work so, so, but I had to disconnect the eGPU by just pulling the plug otherwise it would crash the system.
Ever since 11.4 and 11.5 I get random freezes all the time. And when it goes to sleep random system shut downs. Almost feels deliberate, the timing is just a bit too convenient, just when the M1 Macs came out the newer system updates basically wrecked my experience with eGPUs. What are the odds?