There's something not quite right with all of this and it should be looked into. In order for me to update to Xcode 12.4 I had to clear 45GB of space. Yes... 45GB! I ended up deleting stuff I needed just to work around this.
A large chunk of my disk space is already filled with Xcode and simulator related files. It would be nice if Apple could add in some features to clean this up once in a while, rather than having to rely on 3rd party disk sweepers to identify the issues.