iOS Callkit system screen, no audio if I wait for connection before calling CXAnswerCallAction::fulfill

I'm trying to integrate Callkit into a Flutter app that uses webRTC for calls and I have an issue with taking calls on locked screen. CXAnswerCallAction requires to have the action.fulfill() method called after the connection is established. Here is a pice of code without waiting for establishment of the connection:

    guard let call = self.callManager?.callWithUUID(uuid: action.callUUID) else{
        action.fail()
        return
    }

    call.data.isAccepted = true
    self.answerCall = call
    self.callManager?.updateCall(call)
    sendEvent(SwiftCallKeepPlugin.ACTION_CALL_ACCEPT, call.data.toJSON())

    DispatchQueue.main.asyncAfter(deadline: .now() + .milliseconds(1200)) {
        self.configureAudioSession()
    }
    action.fulfill()
}

This causes the connection time counter to be immediately visible on the screen, but the user still has to wait for connection establishment and can't hear anything.

Here is the code that waits for the establishment of the connection before calling action.fulfill():

    if(self.awaitedConnection.uuid != uuid) {
        action.fail()
    } else if(self.awaitedConnection.isConnected) {
        DispatchQueue.main.asyncAfter(deadline: .now() + .milliseconds(1200)) {
            self.configureAudioSession()
        }
        action.fulfill()
    } else {
        DispatchQueue.main.asyncAfter(deadline: .now() + .milliseconds(1000)) {
            self.waitForConnection(uuid: uuid, action: action)
        }
    }
}


public func provider(_ provider: CXProvider, perform action: CXAnswerCallAction) {
    guard let call = self.callManager?.callWithUUID(uuid: action.callUUID) else{
        action.fail()
        return

    }

    call.data.isAccepted = true
    self.answerCall = call
    self.callManager?.updateCall(call)
    self.awaitedConnection.uuid = action.callUUID
    self.awaitedConnection.isConnected = false
    sendEvent(wiftCallKeepPlugin.ACTION_CALL_ACCEPT, call.data.toJSON())

    waitForConnection(uuid: action.callUUID, action: action)
}

Unfortunately, though it works great on iOS 15.7, on 17.3 it causes lack of audio, no sound and no recording. I also can't enable it later when the call is ongoing. For reference:

    let session = AVAudioSession.sharedInstance()
    do{
        try session.setCategory(AVAudioSession.Category.playAndRecord, options: AVAudioSession.CategoryOptions.allowBluetooth)
        try session.setMode(self.getAudioSessionMode(data?.audioSessionMode ?? "voiceChat"))
        try session.setActive(data?.audioSessionActive ?? true)
        try session.setPreferredSampleRate(data?.audioSessionPreferredSampleRate ?? 44100.0)
        try session.setPreferredIOBufferDuration(data?.audioSessionPreferredIOBufferDuration ?? 0.005)
    }catch{
        print(error)
    }
}

I can see in the docs of action.fulfill() that "You should only call this method from the implementation of a CXProviderDelegate method". I this the reason for the issue? But how can I do it if I need to wait for the connection asynchronously and the provider method is synchronous?