SecureTransport Generates SSL Continuation Message Instead of TLS Client Hello on M1

Yep so for asynchronous operation I capture the pointer value of the void* data parameter of my SSLWriteFunc callback so I can send the client hello on my socket write pipeline after returning from SSLHandshake. On Intel the correct bytes for the TLS client hello are still at the address upon returning from from SSLHandshake but on M1 all zeroes :(

Yes I am aware SecureTransport is deprecated in favor of Network. In Wireshark on the Intel MBP the 151 bytes are observed as a TLS 1.2 client hello while on M1 it is observed as an SSL continuation message and that is the last message observed.

I don't want to sound like a broken record because it sounds like you are aware that you should be using Network framework, but this discrepancy that you have run into may be a good opportunity to try out what using Network framework' Security options and see what that gives you cross platform. The reasoning here is not only will this update your code, but it will also give you more leverage to open a bug report if something is not right cross platform. Right now I suspect if you open a bug report it will be returned with a message to update to Network framework's Security options. If you find an issue with Network framework's Security options, it will be reviewed.

The cause of the error has been identified:

For asynchronous operation I capture the data pointer (to avoid memcpy) provided by my SSLWriteFunc callback triggered by SSLHandshake calls so I can queue the bytes to be sent on my library's custom kernel queueing pipeline after returning from SSLHandshake. On Intel and iOS arm products prior to M1 support I observe the bytes at this address are still valid after returning from SSLWriteFunc callback but on M1 these bytes are already zeroed.

Despite that it doesn't help me in the present I thank you for your response. However, as long as there is production code somewhere still relying on SecureTransport and doing truly asynchronous writes/reads with the underlying socket this matters to the world just not to Apple. What I'm hearing is even though we are savvy enough as a company to make no guarantee we wouldn't do this, we broke expected empirical behavior between the way the library used to operate with CPUs in our old products and the fact that you are developer that spent his slim amount of hard earned disposable income on our new architecture specifically to port his libraries and applications means little to us. But that was always expected and I thought I'd give the forum a try anyway.

The reason I'm not ready to tackle updating to Network yet is because I am in the middle of a long winded cross-platform feature addition resulting in significant architectural changes that originated on the other platforms I support. Again, I haven't had the professional cycles across all the fields I am obligated to dig into yet but one of the things I'm skeptical of is that it's going to provide road blocks for async non-blocking + direct BSD socket access that SecureTransport didn't and I know you are Apple and all but I still like to verify the integrity of each and every modern and legacy library you've published independently, and if you left something out of a "modern" replacement that I require I'm going to reach for the deprecated one.

I still intended to expose SecureTransport as an option for client code relying on that path after I integrate with Network. I'm not just going to throw that code away. After I shown that I've integrated with Network then will I be worthy a non-boiler plate response when filling a bug report re SecureTransport?

So I've looked into Network.Framework a little further. If Apple wants to anger cross-platform devs by "deprecating" the socket paradigm that's fine and I was even prepared to drink your cool aid despite the weak arguments made during its wwdc introduction about how it's so superior to sockets. However, where you've completely lost me is the inability to queue network operations for sending/receiving and dequeue completion notifications of those operations via kqueue on our own thread pools to avoid your GCD rails. I've been developing for your platforms for long enough and am familiar with the internals of GCD such as to be sure that I don't want your GCD implementation indirectly managing kevents for me. By forcing use of Network you have forsaken any library that wishes to do scheduling itself and I don't really think Apple is qualified to dictate what a network architecture for a multiplayer game engine should look like nor should that be the prerogative of anyone but the engine developer.

The high level goal for Network framework was to try and improve developer experience by getting code out of the kernel, away from sockets and into user space. And for the vast majority of Developers, this has worked nicely. I think at this point you should open a bug report about what you are seeing and please include any Developer ergonomics issues that you are seeing with Network framework, as that would be our ultimate goal, to make sure that you are able to use Network framework in your product and not have to fall back to SecureTransport with Sockets.

Very good, thank you, I will do that. To better pose my report what is the best resource to get an understanding of how Network operates under the hood? Can you point me to the source? Is it even available?

To clarify you are saying the entire transport stack from application send/recv to NIC has been moved into userspace such that there are no kernel transitions as in a custom HFT solution? Was AIO completion a la FreeBSD ever once considered?

To clarify you are saying the entire transport stack from application send/recv to NIC has been moved into userspace

For more on this, see the WWDC video on Network Framework here.

Regarding:

what is the best resource to get an understanding of how Network operates under the hood? Can you point me to the source? Is it even available?

Network Framework is not open source, but to better understand it more checkout the WWDC video above or the sample code and documentation here. I can tell you that Network framework will not work for you cross platform, but it is the defacto Networking library for all Apple platforms, so if you find something that does not work for you about it outside of the kernel, please open a bug report about it for review.

Sigh I know that you know I have already watched that video and I know you know the answer to my question regarding whether the userspace transport code operates without going through a driver extension/kext to interface with the NIC is not addressed in it.

None of this really matters. Regardless of whether all operation is in userspace, the userspace applications still needs to dequeue results with kqueue whether it reaches into the kernel or not on whatever thread pool I choose as the userspace application architect. Duh. Duh, right?

GCD uses kqueue internally so it's not even compatible with GCD. Terry spent years porting FreeBSD kernel code to XNU for you. Your OS even has kqueue extensions that FreeBSD doesn't (albeit to my irritation no AIO) used for priority queueing by GCD. Why did you just abandon all that?

The conclusion that I've come to with Network.Framework is going to be that which most other opensource network lib devs have arrived at:

https://github.com/nanomsg/nng/issues/497

You can't provide any legitimate reasons that it's actually superior to sockets as you claim. I'm not going to waste my time filing a bug report that you have all but guaranteed will never be addressed. Instead of abandoning all the work I've done on my library I will just abandon support for natively provided security on Darwin platforms in favor of 3rd party encryption like I am already using even though it hurts my soul to have to exclude Darwin platforms from full cross-platform feature parity.

https://github.com/3rdGen-Media/CoreTransport

Ok my mistake, at ~50 minutes in the the video deterministically expresses that you've moved the Transport layer into userspace but not the entire stack. So you are taking liberties and fibbing when you say you've moved operation into userspace bc you haven't entirely. My point remains, you could have just implemented all this as a pure c api wherein kqueue reads from userspace memory based on some endpoint file descriptor. There are active FreeBSD kernel extensions that do so.

I come back to this a few months later and all traces of the video WWDC 2018 Introducing Network.framework: A modern alternative to Sockets and sample projects have completely been removed from the internet.

have completely been removed from the internet.

You are correct that that video is no longer available from Apple, but my experience is that nothing is ever completely removed from the Internet. To aid in your search for the video I’m going to post the original URL here: https://developer.apple.com/videos/play/wwdc2018/715

On the sample code front, the sample for that session was nwcat, which has been rolled into the main documentation. There are a bunch of other Network framework examples available; start your search on Network framework page.

IMPORTANT Network framework has Swift and C APIs, and the samples only show up if you select the corresponding language at the top right. The two APIs are conceptually very similar, so it’s relatively easy to map from one to the other [1]. I recommend you look at both.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] For an example of that, consider the various snippets on this thread.