AirPods Keep connecting too much

I know the title of this question is less than desirable. However, I can slide down to go to the audio options of a video that I’m playing and choose my AirPods as a speaker output. The problem is after words. I slide back down and choose my TV speaker as my output and then it goes back to it. However, if I open up the AirPod Case again it will automatically start listening to the TV again. This is very annoying if after a run on the treadmill I am using them to listen to my TV show, but then my kids want to watch TV. All goes well and then I opened up my case so I can make a phone call and I start getting theAudio us some kid show in my ear and they lose their audio. Is there a way to keep it from automatically using my AirPods as a speaker?