Look at the console output. My guess is that you will see message like this pop up about 1 every 3-4 seconds. That is what is causing the long load times. It doesn't happen on a real device, but only on the simulators for iOS 13 & 14.
2020-08-09 20:53:24.798020-0400 HS4[19985:1381381] libMobileGestalt MobileGestaltCache.c:38: No persisted cache on this platform.
2020-08-09 20:53:30.748666-0400 HS4[19985:1380170] HALCADClient::ConnectToServer: failed to find the AHS sub-server port, Error: 0x10004003
2020-08-09 20:53:35.749000-0400 HS4[19985:1380170] HALCADClient::ConnectToServer: failed to find the AHS sub-server port, Error: 0x10004003
2020-08-09 20:53:35.749352-0400 HS4[19985:1380170] HALDefaultDevice::Initialize: couldn't add the default input device listener, Error: 268451843 (\^P)
2020-08-09 20:53:40.750540-0400 HS4[19985:1380170] HALCADClient::ConnectToServer: failed to find the AHS sub-server port, Error: 0x10004003
2020-08-09 20:53:40.750906-0400 HS4[19985:1380170] HALDefaultDevice::Initialize: couldn't add the default output device listener, Error: 268451843 (\^P)
2020-08-09 20:53:45.752149-0400 HS4[19985:1380170] HALCADClient::ConnectToServer: failed to find the AHS sub-server port, Error: 0x10004003
2020-08-09 20:53:45.752563-0400 HS4[19985:1380170] HALDefaultDevice::Initialize: couldn't add the default system output device listener, Error: 268451843 (\^P)
2020-08-09 20:53:50.753751-0400 HS4[19985:1380170] HALCADClient::ConnectToServer: failed to find the AHS sub-server port, Error: 0x10004003
2020-08-09 20:53:50.753981-0400 HS4[19985:1380170] HALDefaultDevice::Initialize: couldn't add the default shared output device listener, Error: 268451843 (\^P)
2020-08-09 20:53:50.754989-0400 HS4[19985:1380170] [plugin] AddInstanceForFactory: No factory registered for id <CFUUID 0x600003634e60> F8BB1C28-BAE8-11D6-9C31-00039315CD46
2020-08-09 20:53:50.756684-0400 HS4[19985:1380170] Error loading /Library/Audio/Plug-Ins/HAL/BartenderAudioPlugIn.plugin/Contents/MacOS/BartenderAudioPlugIn: dlopen(/Library/Audio/Plug-Ins/HAL/BartenderAudioPlugIn.plugin/Contents/MacOS/BartenderAudioPlugIn, 262): no suitable image found. Did find:
/Library/Audio/Plug-Ins/HAL/BartenderAudioPlugIn.plugin/Contents/MacOS/BartenderAudioPlugIn: mach-o, but not built for platform iOS-sim
2020-08-09 20:53:50.756915-0400 HS4[19985:1380170] [plugin] Cannot find function pointer NewSHPPlugIn for factory <CFUUID 0x600003634500> 834FC054-C1CC-11D6-BD01-00039315CD46 in CFBundle/CFPlugIn 0x7fa565019c70 </Library/Audio/Plug-Ins/HAL/BartenderAudioPlugIn.plugin> (bundle, not loaded)
2020-08-09 20:53:50.757190-0400 HS4[19985:1380170] AudioObjectSetPropertyData: no object with given ID 0
2020-08-09 20:53:50.757379-0400 HS4[19985:1380170] 83: Failed to set processVolumeScalar on device. Error: 560947818
2020-08-09 20:53:55.924562-0400 HS4[19985:1382805] HALCADClient::ConnectToServer: failed to find the AHS sub-server port, Error: 0x10004003
2020-08-09 20:53:55.925082-0400 HS4[19985:1382805] HALCADClient::GetPropertyData: unknown property
2020-08-09 20:53:55.925347-0400 HS4[19985:1382805] [ddagg] AggregateDevice.mm:800 couldn't get default input device, ID = 0, err = 0!
2020-08-09 20:54:00.927012-0400 HS4[19985:1382805] HALCADClient::ConnectToServer: failed to find the AHS sub-server port, Error: 0x10004003
2020-08-09 20:54:00.927255-0400 HS4[19985:1382805] HALCADClient::GetPropertyData: unknown property
2020-08-09 20:54:00.927365-0400 HS4[19985:1382805] [ddagg] AggregateDevice.mm:800 couldn't get default output device, ID = 0, err = 0!
2020-08-09 20:54:00.927567-0400 HS4[19985:1382805] [aqme] 352: error -66680 finding/initializing Device.AQDefaultDevice
2020-08-09 20:54:00.927709-0400 HS4[19985:1382805] 111: * * * NULL AQIONode object
2020-08-09 20:54:00.927757-0400 HS4[19985:1380170] [aqme] 352: error -66680 finding/initializing Device.AQDefaultDevice
2020-08-09 20:54:00.927858-0400 HS4[19985:1382805] 848: Can't make UISound Renderer
2020-08-09 20:54:00.927847-0400 HS4[19985:1380170] [AQ] 1899: failed (-66680); will stop (12000/0 frames)
2020-08-09 20:54:00.928927-0400 HS4[19985:1380170] AudioDeviceStop: no device with given ID
2020-08-09 20:54:00.929094-0400 HS4[19985:1380170] [aqme] 352: error -66680 finding/initializing Device.AQDefaultDevice
2020-08-09 20:54:00.929225-0400 HS4[19985:1380170] [AQ] 1899: failed (-66680); will stop (12000/0 frames)
2020-08-09 20:54:00.997159-0400 HS4[19985:1382805] [] nwprotocolgetquicimageblockinvoke dlopen libquic failed