This issue has bugged me for some time. Whenever I do a full build whether it's for debugging or profiling, Xcode always insists on recompiling all files the second time I build (regardless of whether there are changes or not), but then behaves normally after that and only recompiles changed files when I build.
FWIW, I use a precompiled header. My project also contains other Xcode projects that consists of static libraries and frameworks that are built as dependencies. Those other Xcode projects are not rebuilt after the first build, just the source files included in my project.
This has been an ongoing problem for years and continues with Xcode 13.1. I'm sure it's something within my Xcode project that I've been maintaining since at least Xcode 3 (I don't remember which version of Xcode I started with) because new projects don't exhibit this behavior.
Again, building doesn't unnecessarily recompile all files, only the second time I build.