Network framework instability in peer to peer

Hi there,


We've been looking into the new Network framework as an alternative to peer to peer communication, so we've been playing around it and running several tests.


We got impressed by the outcome when compearing it to Multipeer framework, Network FW offers faster transmissions and more reliability. However, there are a few things we don't understand.


The scenario we built is a couple of example apps, one based on Multipeer FW and the other on Network FW. Both apps set up a connection Server - Client in order to send and receive data from any side.

We ran different bunch of tests with different setups for the interfaces of devices (enable, disable and connected to Wifi or not). Also, we set up different payload sizes and the "distance" variable.


The result is very good when devices are together each other (no distance between them) for the scenario where they are connected peer to peer (Wifi is not connected to the Wifi infraestructure network), connection is stable and Nework FW is very fast for big payload of data. Yet, when there is distance between them, like 5 meters, things get worse.

We observed that distance affects a lot the performance of the transmission and connection stablishment. Besides, bandwith is very unstable.


As an example, I'll give the details of one of our tests:

Device 1 and 2

- Bluetooth enable or disable

- Wifi enable but disconnected from the Infrastructure network.


Scenario

- Network FW

- TCP protocol (default set up)

- 5 meters distance between them

- 500 KB payload

- Frequency for sending: 150 seconds. Device 1 sends and Device 2 receives. In the middle of that frequency (75 sec) Device 2 sends and Device 1 receives. After 150 s Device 1 sends again and the process is repeated recursively. This is done in this way to avoid full-duplex communication in parallel, since it hits the performance even more.


The outcome of that test is that there is a very big range in how many seconds the transmission takes. We got values from 0,285 spt (seconds per transmission) to 138,45 spt. Other results are: 5,83 - 47,45 - 8,09 - 19,61, 11, 24.....


What could the explanation be for that instability? Why minimum and maximum values are so distant and different? when devices are together the transmission is stable and more or less takes always the same.

We also saw that the number of chunk of data that the receptor receives per a sending is affected, but we can't find any logic between that number of segments and the time the transmission takes.



On the other hand, we'd like to ask another question.

Given the scenario of a peer to peer connection throught NWConnection, if "send" method is used a few times before the receptor finishes the reception of the 1st send called (1st sending has not been completed), what whould it happend? is the "receive" clousure going to be called receiving the data of other sendings before the first one is completed?


Thanks,

Manuel

Accepted Reply

First, some background. Given that way that Apple’s peer-to-peer Wi-Fi is currently implemented there’s a tradeoff between range and connection setup reliability. The longer the range, the harder it is for the system to set up the connection. To help with this Apple uses very low transmission power for peer-to-peer Wi-Fi. This reduces the effective range but makes connection setup much more reliable. So the range you’re testing at, 5 metres, is actually quite long range for peer-to-peer Wi-Fi.

As to what’s happening in the specific scenario you’ve outlined, it’s likely that this results from your TCP retransmissions. TCP has to deliver data in order, which means that if a packet containing segment N gets dropped then TCP can’t delivere segments N+1, N+2 and so until the retransmission of segment N is complete. This is a well-known source of problems with TCP on unreliable networks.

Be aware that Multipeer Connectivity (MPC) is not passed on TCP. When you send data with a send mode of

MCSessionSendDataReliable
, MPC uses UDP and does its own retransmissions. This has its pros and cons, with one pro being that it eliminates these artificial dependencies between unrelated segments.

To understand the performance problem you’re seeing you need to look at the actual data on the ‘wire’. The first step with that is to take an RVI packet trace (per QA1176) and see exactly what’s happening with your data. It’s possible that packets are simply being dropped, or its possible that link-layer retransmission is resulting in significant packet delays.

Finally, you wrote:

Given the scenario of a peer to peer connection throught NWConnection, if "send" method is used a few times before the receptor finishes the reception of the 1st send called (1st sending has not been completed), what whould it happend? is the "receive" clousure going to be called receiving the data of other sendings before the first one is completed?

TCP is a stream protocol, so it does not preserve record boundaries. So:

  • When you send data via an

    NWConnection
    , the system calls the send completion handler when the data has been accepted for transmission by the networking stack. This tells you very little about whether the data has been transmitted.

    Note The primary use of the completion handler is so that you can implement send-side flow control, that is, if you have too much outstanding data you need to stop generating new data because doing so just results in more data being buffered by the connection.

  • On the receive side the completion handler is called when data has arrived that satisfies the read request. If you have called

    receive(minimumIncompleteLength:maximumLength:completion:)
    then it’s clear what criteria apply. If you have called
    receiveMessage(completion:)
    then the read completes whenever any data is available (
    receiveMessage(completion:)
    is usually used with UDP, which does preserve record boundaries).

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Replies

First, some background. Given that way that Apple’s peer-to-peer Wi-Fi is currently implemented there’s a tradeoff between range and connection setup reliability. The longer the range, the harder it is for the system to set up the connection. To help with this Apple uses very low transmission power for peer-to-peer Wi-Fi. This reduces the effective range but makes connection setup much more reliable. So the range you’re testing at, 5 metres, is actually quite long range for peer-to-peer Wi-Fi.

As to what’s happening in the specific scenario you’ve outlined, it’s likely that this results from your TCP retransmissions. TCP has to deliver data in order, which means that if a packet containing segment N gets dropped then TCP can’t delivere segments N+1, N+2 and so until the retransmission of segment N is complete. This is a well-known source of problems with TCP on unreliable networks.

Be aware that Multipeer Connectivity (MPC) is not passed on TCP. When you send data with a send mode of

MCSessionSendDataReliable
, MPC uses UDP and does its own retransmissions. This has its pros and cons, with one pro being that it eliminates these artificial dependencies between unrelated segments.

To understand the performance problem you’re seeing you need to look at the actual data on the ‘wire’. The first step with that is to take an RVI packet trace (per QA1176) and see exactly what’s happening with your data. It’s possible that packets are simply being dropped, or its possible that link-layer retransmission is resulting in significant packet delays.

Finally, you wrote:

Given the scenario of a peer to peer connection throught NWConnection, if "send" method is used a few times before the receptor finishes the reception of the 1st send called (1st sending has not been completed), what whould it happend? is the "receive" clousure going to be called receiving the data of other sendings before the first one is completed?

TCP is a stream protocol, so it does not preserve record boundaries. So:

  • When you send data via an

    NWConnection
    , the system calls the send completion handler when the data has been accepted for transmission by the networking stack. This tells you very little about whether the data has been transmitted.

    Note The primary use of the completion handler is so that you can implement send-side flow control, that is, if you have too much outstanding data you need to stop generating new data because doing so just results in more data being buffered by the connection.

  • On the receive side the completion handler is called when data has arrived that satisfies the read request. If you have called

    receive(minimumIncompleteLength:maximumLength:completion:)
    then it’s clear what criteria apply. If you have called
    receiveMessage(completion:)
    then the read completes whenever any data is available (
    receiveMessage(completion:)
    is usually used with UDP, which does preserve record boundaries).

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Hi eskimo,


Thank you very much for you response, it is really useful.


We did some tests with Wireshark and we realised that multipeer uses UDP, but we weren't able to reproduce a scenario to capture the traffic between the devices peer to peer at 5 meters. Now, knowing about RVI we will try to figure out why that happens.


Regarding how Network framework works, we were assuming that the completion closure executed in the sender was trigger once the receptor received the whole message, which is wrong. Knowing that, we will put in place some mechanism to let the sender know that the receptor has received the whole message, so that we can control the sending flow.


Thanks for your help again.


We will come back with the outcome of the capture tests.


Manuel

we were assuming that the completion closure executed in the sender was trigger once the receptor received the whole message, which is wrong.

Indeed.

Knowing that, we will put in place some mechanism to let the sender know that the receptor has received the whole message, so that we can control the sending flow.

That may or may not be necessary. If you’re using TCP then TCP already has a limit to how much it will buffer once it stops receiving ACKs from the remote peer.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Finally we're glad to say that we found out what the issue was. The problem was in our implementation of the receptor closure. That fixed, the transmission is more stable and the second average has been reduced a lot for tests where there is distance between devices. We managed to figure it out thanks to Wireshark and RVI.


Regarding controlling the sending flow by ourselves, we will take into account what you said to assess whether we really need it or we can rely on TCP for that purpose.


Thanks eskimo!


Manuel

Is it possible to communicate between two different apps using P2P?

Is it possible to communicate between two different apps using P2P?

I’d appreciate you starting a new thread for this (because it’s far removed from the original question). Tag it with Network so that I see it.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"