I have the exact same issue. The only lead that I found is this answer on the CircleCI forum which mentions a permission issue on Sonoma: https://discuss.circleci.com/t/severe-performance-problems-with-xcode-15/49205/135
Maybe the same thing happens on Xcode Cloud...