Any ways to speed up codesign?

I've been noticing that code sign phases seem to take an inordinate amount of time, like on a scale of 3-8 seconds. This adds up when you have multiple binaries involved and are doing incremental builds.

Note that this is for debug builds with the --timestamp=none flag set. I've tried turning on verbosity but nothing interesting gets output. Any thoughts on what may be causing this and if there are ways to speed this up?

I'm running Xcode 12 beta 5.