Just to make sure this thread gets the attention it deserves. I have the same issue on my brand new Macbook Pro 14" M1 Pro with a rather small and simple project. None of the suggestions above made any noticeable improvement, the only thing that works is to hide my SwiftUI previews...
Project stats:
find . -name '*.swift' | xargs wc -l | awk '{print $1, "***.swift"}'
58 ***.swift
36 ***.swift
40 ***.swift
65 ***.swift
180 ***.swift
70 ***.swift
70 ***.swift
71 ***.swift
147 ***.swift
67 ***.swift
70 ***.swift
35 ***.swift
102 ***.swift
89 ***.swift
194 ***.swift
74 ***.swift
319 ***.swift
115 ***.swift
183 ***.swift
123 ***.swift
84 ***.swift
38 ***.swift
87 ***.swift
22 ***.swift
476 ***.swift
14 ***.swift
30 ***.swift
21 ***.swift
26 ***.swift
27 ***.swift
2933 total
My Macbook's stats:
At this point I'm considering throwing all my iDevices out of the window as well and start a new career as a barista.