Try this for XCode 12.4,
XCode Preferrences... Navigation Navigation Style try keeping it as Open in Place. It should save your time while building the app.
Also check,
XCode File Workplace Settings Select Build System as New Build System (Default)