Out-of-band data returned by recv() and read() on socket bound to non-loopback address even when SO_OOBINLINE is disabled

Question

Created May ’24

Replies 7

Boosts 0

Participants 2

I've been investigating an issue with the SO_OOBINLINE socket option. When that option is disabled, the expectation is that out-of-band data that is sent on the socket will not be available through the use of read() or recv() calls on that socket.

What we have been noticing is that when the socket is bound to a non-loopback address (and the communication is happening over that non-loopback address), then even when SO_OOBINLINE is disabled for the socket, the read()/recv() calls both return the out-of-band data. The issue however isn't reproducible with loopback address, and read()/recv() both correctly exclude the out-of-band data.

This issue is only noticed on macos. I have been able to reproduce on macos M1, following version, but the original report which prompted me to look into this was reported on macos x64. My M1 OS version is:

sw_vers
ProductName:		macOS
ProductVersion:		14.3.1
BuildVersion:		23D60

Attached is a reproducer (main.c.txt - rename it to main.c after downloading) that I have been able to develop which reproduces this issue on macos. When you compile and run that:

./a.out

it binds to a non-loopback address by default and you should see the failure log, resembling:

...
ERROR: expected: 1234512345 but received: 12345U12345

To run the same reproducer against loopback address, run it as:

./a.out loopback

and that should succeed (i.e. no out-of-band data) with logs resembling:

...
SUCCESS: completed successfully, expected: 1234512345, received: 1234512345

Is this a bug in the OS? I would have reported this directly through feedback assistant, but my past few open issues (since more than a year) have not even seen an acknowledgement or a reply, so I decided to check here first.

main.c.txt

Answered by ForumsContributor in

Boost

Answer 1

DTS Engineer OP

Apple

May ’24

Folks still care about out-of-band data!?! Fascinating.

Is this a bug in the OS?

I’ve no idea. I’ve been supported networking on macOS for 25+ years and IIRC this is the first time that I’ve ever seen a question about OOB data. So, yeah, I don’t really know what the correct behaviour is O-:

How does this difference in behaviour impact your product?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 2

jaikiran OP

May ’24

Hello Quinn, I don't have data on whether applications still rely on out-of-band or whether applications do send out-of-band data. However, given that this is an option at the socket level, the JDK as part of the Java specification exposes an API on its java.net.Socket class to allow applications to enable or disable this option https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/net/Socket.html#setOOBInline(boolean). As part of verifying that the Java API works as expected (on all OS), we run tests for that API. In fact, that's what prompted me to look into this issue when we noticed it fail (only) on macos. For context, here's the JDK issue https://bugs.openjdk.org/browse/JDK-8279920 where we have been tracking this.

Specifically here, if some peer ends up sending a out-of-band data (for whatever reason) and even if the application has decided it isn't interested in out-of-band data (which is the default), then the application can still end up receiving this out-of-band data unexpectedly.

0

Answer 3

DTS Engineer OP

Apple

May ’24

Accepted Answer

Ah, interesting. Thanks for the backstory.

I recommend that you file a bug about this. BSD Sockets’ primary reason to exist is for compatibility with cross-platform code like this, and it’s a problem if that’s not working.

Please post your bug number, just for the record.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 4

jaikiran OP

May ’24

Thank you Quinn for your help so far. I have now filed FB13799990 to track this issue and attached the same reproducer to it.

On a different note, during the past couple of years I have filed a few issues through feedback assistant, 2 of them can be classified as belonging to networking area. They are still open and haven't seen any acknowledgement or any response. In some other channels, I have been told (by people who aren't from Apple) that when such issues in feedback assistant don't see any response, it most likely means that they haven't been triaged and the suggestion is to refile them afresh. I don't know if I should be doing that. The feedback ids for those other issues are FB12128351, FB12016446 and FB9997771. Would you have any inputs on whether or not I should refile these other issues or just leave them alone and hope someone responds to them?

0

Answer 5

DTS Engineer OP

Apple

May ’24

I have now filed FB13799990 to track this issue

Thanks for that!

The feedback ids for those other issues are FB12128351, FB12016446 and FB9997771.

So, that last one FB9997771 was reported as fixed in the 2022 OS rereleases (so, macOS 13 and friends). That should’ve been communicated to you but wasn’t. I’m not sure why. I’ll follow-up on that internally, just for my own understanding, but this is all about Apple internal processes so I probably won’t post any more details here.

The other two are still under investigation; I don’t have any further info to share.

when such issues in feedback assistant don't see any response, it most likely means that they haven't been triaged

That’s not true in general.

Bug Reporting: How and Why? has a bunch of hints and tips on this front, but probably the best important is this one:

If you’re filing a bug against an API, choose Developer Technologies & SDKs at the top level.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 6

jaikiran OP

May ’24

So, that last one FB9997771 was reported as fixed in the 2022 OS rereleases (so, macOS 13 and friends).

Thank you for that detail, Quinn. I am very happy to hear that that one is officially fixed. That had caused really odd intermittent failures in unexpected areas within the JDK and had taken us a long time to narrow it down. I will run our tests against macosx 13 and higher to verify the fix.

That should’ve been communicated to you but wasn’t. I’m not sure why. I’ll follow-up on that internally, just for my own understanding, but this is all about Apple internal processes so I probably won’t post any more details here.

I understand.

The other two are still under investigation; I don’t have any further info to share.

Thank you for that update.

Bug Reporting: How and Why? has a bunch of hints and tips on this front, but probably the best important is this one: If you’re filing a bug against an API, choose Developer Technologies & SDKs at the top level.

This is useful. So far I've been filing it under "Something else not in this list" category for several of the bugs that I've opened either related to network APIs or kernel APIs. Including the one that I opened in this discussion. Henceforth, I'll keep that category in mind.

Overall, thank you again for all the help and responses you have been providing, not just in this thread but other previous discussions too. Not receiving any updates/responses on feedback assistance issues is demotivating but seeing the responses in the developer forums here and being assured that the feedback assistant issues have been noticed and are being investigated does encourage in filing new ones.

0

Answer 7

jaikiran OP

May ’24

Hello Quinn,

So, that last one FB9997771 was reported as fixed in the 2022 OS rereleases (so, macOS 13 and friends).

Thank you for that detail, Quinn. I am very happy to hear that that one is officially fixed. That had caused really odd intermittent failures in unexpected areas within the JDK and had taken us a long time to narrow it down. I will run our tests against macosx 13 and higher to verify the fix.

I ran our reproducer against macos 12.x, 13.x and 14.x versions of macos aarch64. I can confirm that the issue is fixed and no longer reproducible in 13.x and 14.x versions. The issue continues to reproduce in 12.x versions of macos aarch64.

0