Hey @leonwu21,
Unfortunately I do not believe Apple has provided a way to do this the way that you (or I) are thinking about this. Make sure to file Feedback so they understand your use case and build more tools for the community!
With that being said, your approach is in line with what I have been doing. I query for betaGroups, then I query the betaTesters and create a consolidated list and remove any duplicates so I only have a list of unique betaTesters.
I then hit the betaTesterInvitations API for all of the testers even though I do not know who has been sent an invitation and who has not. This is where you can programmatically determine who has accepted the invitation based off of the API response. Apple returns a 409 for users who have already accepted the invitation and are testing the application along with additional information in the response body for us to process as developers.
Make sure to file Feedback but hopefully this helps!
Have a great day!