A different scenario and maybe a clue to what's broken in the Mail app?
I have an iPhone16Pro that was configured from a backup of an iPhone15Pro running iOS 17.7 with the same problem. Removing the mail accounts and restarting the phone did not work. Erasing all data and starting from fresh did not work.
I'm trying to connect to a dovecot instance with a cert signed by my own root certificate.
I then created a profile with those certs and installed it on the phone that had been restored once again from the iOS17.7 backup and had the mail account removed and the phone rebooted. After I installed the profile, I can see and have enabled my root cert in the Certificate Trust Settings on the phone.
When I add the Mail account, it negotiates the SSLv3/TLSv1.3 successfully. However, when the app tries to get mail, the mail server still gets the error code indicating that the client doesn't trust the certficate.
Note: The certs continue to work with Thunderbird as the mail client on macOS Sequoia 15.0.1.