Same problem here. I tried reinstalling Catalina without effect, killing the viewer window and reopening it without effect as well. The Mail Activity window shows mail trying to download thousands of emails at those times - I suspect that's part of the clue to the cause, since it's possible that Mail is losing sync with the IMAP server and rechecking every message.

Found this in the Apple Discussion Forums, but I don't think this issue is the same. Option 1 would suppose that Google's and Apple's mail servers are going down simultaneously (since this affects both at the same time in my testing). Option 2 is a possibility, I suppose, but unlikely as the problem is intermittent.From Apple Discussion Forums:Linc Davis User level:Level 10 (209,164 points) NotebooksSpeciality level out of ten: 1 Jan 25, 2016 12:26 AM in response to MarkStar0013There several possible causes for this issue:1. The mail server is refusing connections.2. Something in the network, such as your router, a proxy server, or your ISP, is blocking the connection.3. Third-party software that you installed is blocking the connection.4. The settings for the mail account are wrong.You may be able to narrow down these possibilities by means of the following tests or observations:A. If another device on the same network can send or receive mail through the same server at the same time with what you're sure are the same settings, that rules out everything except 3.B. If the problem is intermittent (sometimes it works, sometimes not) and you're not changing anything, that rules out 3 and 4.C. If you can always use the account when connected to a different network, but never on this network, that rules out everything except 2.D. If you can never get or send mail from any device on any network, with the correct server settings according to the mail service provider, that rules out everything except 1.Please post the results of as many of the above tests as you can run.Jan 27, 2016 8:40 PM in response to MarkStar0013Please sign out of iCloud in its preference pane and see whether there's any change. No data will be removed from the servers, and you can retrieve it by signing back in. That said, you should always have a current archiveof the data for safety's sake, even if you don't sign out.If you use iCloud Keychain, when you sign back in to iCloud follow one of the procedures described in this support article to set it up on an additional device.