Thank you all above for posting so much detail. In the end it was exactly the fact that it was not a RSA key that fixed the issue for me. I rotated out a ecdsa-sha2 key for rsa and it worked. Of course in doing so I updated the server authorized keys as well, so Xcode just likes a very particular setup.
Bruji
Users receive reputation points by contributing quality content to the forums.
Posts include post and replies published on the forums.
Post authors can mark replies as Accepted to indicate successful solutions.
Apple Developer Forums admins can mark replies as Apple Recommended to indicate an approved solution
Post
Replies
Boosts
Views
Activity
Switching ssh keys from ecdsa-sha2 to rsa on that git user fixed the issue for me.