Network framework and background tasks

I'm working on an MQTT client for Apple platforms (macOS, iOS, and possibly tvOS and watchOS).

The story here varies by platform:

• For macOS, it’s possible to run indefinitely in the background.

• watchOS has significantly more restrictions than iOS. See TN3135 Low-level networking on watchOS for the details.

• tvOS generally follows iOS, except that one of the common workarounds, push notifications, is not available there.

The rest of my response focuses on iOS (including iPadOS).

it seems that my best bet is to use a long-running background process with while setting up the connection. Does that sound like the right approach?

No. Quoting iOS Background Execution Limits:

To request extended background execution time, typically delivered overnight when the user is asleep, use BGProcessingTaskRequest.

So, unless you just happen to want to run these connections at some randomish time during the middle of the night (-: this is unlikely to be useful to you.

I currently have a working BSD socket. I'm not sure if it is necessary to switch to the Network Framework to have the background task working, but I'm open to switching if it's necessary.

Network framework doesn’t change this situation. BSD Sockets and Network framework are both low-level networking APIs, and they behave similarly when in comes to networking in the background.

As to what you should do, it very much depends on the specifics of your MQTT client. In many cases the best path forward is to offload this work to a server, and have it generate push notifications to notify the iPhone user of relevant conditions.

the link to "WWDC 2020 Session 10063 Background Execution Demystified" … is broken. Is there a way to access the content there?

That video is no longer available from Apple )-: I haven’t fixed the link because a) sometimes I’m able to ‘resurrect’ important videos like this one, and b) the details in the link make it easier to find related non-Apple resources.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hi Quinn,

Thank you so much for the response. I have some follow-up questions about network suspension and resumption (or socket reclaim?).

From Networking and Multitasking

All iOS networking APIs are ultimately implemented in terms of the BSD Sockets API

From this statement, I got the impression that the Apple Network Framework uses BSD Sockets for its underlying implementation. However, in your reply from iOS/tvOS seem to reuse closed sockets, you mentioned

this is all Network framework and thus there are no sockets in play

I'm a little confused about what does the "no sockets" means here.

Here’s my understanding of the reclaim mechanism for Listening Sockets—please correct me if I'm wrong:

• If the resource is reclaimed, to resume the app properly, the app needs to reconfigure the socket, perform the handshakes, and re-established the connection.
• If the resource is not reclaimed yet, the socket is put in idle while the app is in the background. When the socket is resumed, the app can continue processing the requests. (Of course, as the socket was idled, the requests might fail or time out, and the app would need to handle these cases.)

One peculiar thing I’ve noticed is that the "reclaim" behavior seems inconsistent. While testing with the Network framework using the following steps from Networking and Multitasking

1. putting your app in the background
2. ensuring that the app is suspended
3. locking the screen

Sometimes when the socket is reclaimed, the Network framework sends out a TCP FIN packet and closes the connection, and other times it does not. Does this mean the behavior is undefined for reclaim, or am I doing something wrong?

Additionally, I observed the same results using my own BSD socket implementation and the Apple Network framework (the test app is based on the SWIFT-NIO library). It seems they behave similarly. As I planed to move to the Network Framework for TLS1.3 support, are there any features or behaviors that the Network Framework provides that I should be aware of?

Thanks again for your help!

From this statement, I got the impression that the Apple Network Framework uses BSD Sockets for its underlying implementation.

This has changed in recent years [1], where Apple has been rolling out a user-space networking stack. Network framework will use either BSD Sockets or the user-space networking stack depending on the exact circumstances [2].

As to the behaviour you’re seeing, the key thing to note in TN2277 is the phrase “becomes eligible for suspension”. As an app developer, you’re not notified when your app gets suspended. There is one obvious high-level factor here — your app won’t be suspended while it’s in the foreground — but after that things get complex. If your app runs in the background, you have to track whether your eligible for suspension or not. If your app can run in the background for many different reasons, that tracking gets rather complicated )-:

Once you have this sorted out, the next step is to decide what to do with your network constructs. For listeners (listening socket and NWListener) the answer is clear: Close them when you become eligible for suspension and re-open them when that’s no longer the case. This is the only sensible option because, if you get suspended but your connection isn’t defuncted [3], clients will connect but not receive any response.

For connections you get to make the tradeoff described in the Data Socket section of TN2277. If you leave the connection open, you have to accept that the actual on-the-wire behaviour is unspecified. If you just get suspended then the connection goes ‘deaf’. If you get suspended and the connection is defuncted, the remote peer sees the connection close.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] TN2277 is in the Documentation Archive for a reason: While most of the concepts are still accurate, a bunch of details have changed.

[2] We’ve recently been discussing this on this thread.

[3] This is one of those places where TN2277 could do with an update. I’ve started using “defuncted” in preference to “socket resource reclaimed” because a) it better maps to what you’ll see in the Darwin open source, and b) it works for both BSD Sockets and Network framework. It’s also less typing (-:

Hi Quinn, sorry for revisiting this old question. Thank you so much for the explanation. I've been testing on Network Framework while app in background, and I have some further questions.

1. From TN2277 in section Data Socket:

If you do leave your data socket open when going into the background, you must correctly handle errors on that socket.

What is the error handling required by the system different from a "normal socket error handling"? The example given in the TN2772 was handling EBADF on socket reclaimed. Is there more information about what are the other errors we should take care of?

1. About suspension. The doc was quite vague about the

(app) becomes eligible for suspension when it stops doing the thing that prevented it from being suspend

I did more testing on this by creating a long running task which keep writing into the socket. It seems the application would be kept alive as long as the task is running. (It ran for about 3 hours before I stopped it. ) From my investigation, it seems that the Network Framework handler like nw_connection_set_state_changed_handler would be proceed as normal when the application in background. Does it means that the socket IO events would prevent the app from being suspended?

1. Also, probably because the socket was kept alive, I've never got error like EBADF in the tests above. Would you have any recommendation for a proper way to verify if the socket is affected by the suspension...?

How are you testing this stuff?

If you run the app from the debugger then it prevents the app from being suspended, so you’ll see unexpected behaviour. To test this properly, you need to run the app from the Home screen and then use logging to determine what actually happened. I generally recommend using the system log for this; see Your Friend the System Log for hints and tips on that.

Oh, and the EBADF code isn’t even accurate for BSD Sockets these days. At some point we changed to use a more sensible error code. I don’t remember the details but, once you get your test working reliably, it’ll be easy to see.

And likewise for the error you get when you exercise this case with Network framework.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Here’s an update on my findings in case anyone else is interested in this topic.

I attempted to keep the connection open while the app is in the background. As Quinn mentioned, when the app is backgrounded, the socket essentially "goes deaf." When I bring the app back to the foreground, the connection will encounters the ECONNABORTED error code from IO handlers.

My main issue is determining how to handle the connection after resuming the app. If the app was suspended but the connection is still viable, I was able to continue using the connection for further IO events after resume the app. However, if the connection is defunct, I have to recreate a new connection.

Unfortunately, I haven’t found a reliable way to distinguish between these two scenarios. For now, I’ve opted to always close and re-create the connection upon receiving an ECONNABORTED error, and I’m leaving it at that.

If anyone has insights on detecting when a socket resource has been reclaimed, I’d love to hear any suggestions for improvement!

Thanks for sharing your results.

Regarding your remaining question, I’d like to clarify your big picture goal. When your app comes back to the foreground, I see three cases:

• The connection is functional (A).

• The connection has failed via ‘natural causes’, like an asteroid strike on your data centre (B).

• The connection has been defuncted (C).

You can already distinguish case A, because your I/O handlers are still running, so you’re asking about distinguishing between case B and C. Right?

But why do you need to distinguish those? Either your app needs a connection to the server or it doesn’t. If it does need that connection, it should try to open it, and that’s true regardless of whether the connection failed due to B or C.

Or am I missing something?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thanks for following up!

My request stems from the requirements of the MQTT protocol. According to the protocol, the client is expected to keep retrying the connection to the server until either the user explicitly stops it or a retry limit is reached.

In case B, where there’s a temporary issue like a "data center outage," the client should continue sending "Connect" requests until the server get recovered from "asteroid strike" (or until it hits the retry limit).

In scenario C, however, we know the connection is permanently defunct. In this case, there’s no value in endlessly retrying, as the "Connect" requests will not succeed. We'd prefer to prompt the user to establish a new connection rather than engaging in pointless retry attempts.

In this case [C], there’s no value in endlessly retrying, as the "Connect" requests will not succeed.

But this is TCP, right? So if the connection is defuncted then you’ll hear about it via the usual state update handler mechanism. At that point you can create a new connection and it’ll either connect immediately or, if there’s something wrong with the network, enter the .waiting(…) state.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"