I solved this problem by doing two things:
Making my Wifi 2.4ghz and 5ghz have the same name (SSID). I had them with different names.
Fixed both 2.4 and 5 to specific channels rather than letting my router choose the radio channel automatically. I chose ones that were not too busy. You might want to try a couple of different channels (particularly for 5ghz) in case it doesn't fix the problem right away.
If it helps, my router is a FRITZ!Box 7581.