The problem was that the focus could only move vertically or horizontally. And because of my .frame() modifier, the buttons were positioned diagonally. The solution was to use .focusSection().
Here it's explained: https://developerinsider.co/swiftui-focussection-uifocusguide-alternative-for-swiftui/
Jonatus
Users receive reputation points by contributing quality content to the forums.
Posts include post and replies published on the forums.
Post authors can mark replies as Accepted to indicate successful solutions.
Apple Developer Forums admins can mark replies as Apple Recommended to indicate an approved solution
Post
Replies
Boosts
Views
Activity
Actually, .navigationViewStyle(.stack) is the answer. I don't know why, it's just better than the usual one.