Is DEXT Driver supporting these Networking Features?

Tx/Rx Multiqueue,

Yes. A DEXT can have multiple IOUserNetworkRxSubmissionQueue/IOUserNetworkRxCompletionQueue and IOUserNetworkTxSubmissionQueue/IOUserNetworkTxCompletionQueue.

RSS

The queue architecture above handles this. Every queue has an associated dispatch queue, which lets you control parallel packet processing.

NS/ARP offload

Your DEXT is directly controlling your hardware so, yes, I think this is something you could configure your hardware to do.

RSC, PTP or packet timestamping and TSN.

I'm not entirely sure how all of those are handled but it might be helpful to look at the NetworkingDriverKit Constants list.

__
Kevin Elliott
DTS Engineering, CoreOS/Hardware

Hi, @DTS Engineer

The queue architecture above handles this. Every queue has an associated dispatch queue, which lets you control parallel packet processing.

In the RSS feature of Windows (e.g., as described on the website https://learn.microsoft.com/en-us/windows-hardware/drivers/netcx/rsc-offload), the driver developer uses NET_ADAPTER_RECEIVE_SCALING_HASH_SECRET_KEY to request a hash key from the system. This hash key is then used with an algorithm to calculate a hash value, which determines the specific QueueID.

Then, I reviewed this document IOUserNetworkRxSubmissionQueue/withpool

As I understand it, if I need 4 queues, they can be created using withPool to initialize IOUserNetworkPacketQueueId queueId[0~3]. Each queue extracts packets from the same IOUserNetworkPacketBufferPool, with capacity and bufferCount set as uint32_t capacity, uint32_t bufferCount. However, I am uncertain how to declare or assign values to the two parameters, OSObject * target and DequeueAction dequeueAction. If possible, please provide more information or detailed documentation.

Additionally, are there predefined functionality or value lists for the two parameters void * refCon and IOOptionBits options that can be used for specific feature configurations?

I would like to know the mechanism by which this method determines which QueueID a packet should be placed into or dequeued from for both Tx and Rx. I hope to get a detailed explanation of the exact usage, sample code, and whether this aligns with your statement, "The queue architecture above handles this."

Thanks

First off, I need to be up front on this point:

If possible, please provide more information or detailed documentation.

This isn't directly stated but, practically speaking, but implementing a DriverKit DEXT basically requires a solid understanding of IOKit, as much of DriverKit's actual implementation is directly copied or derived from IOKit. This hasn't been an issue for most DEXT developers because they already had quite deep understanding of IOKit, as they're typically "porting" an existing IOKit KEXT to DriverKit, not actually creating a truly new DEXT from "scratch".

It sounds like you're coming into this as a new developer and I'm afraid that's likely to require a more difficult process of education and indepedant investigation. I can try and provide some broad guidance and direction but, as detailed documentation and sample code are both somewhat limited. However, NetworkingDriverKit does have the sample "Connecting a network driver" built on PCIDriverKit*, which you may have overlooked.

*Keep in mind that a NetworkingDriverKit DEXT typically involves at least two DEXT frameworks, one for the underlying hardware transport (PCI/USB) which is then presented to the networking system through NetworkingDriverKit.

In terms of getting started with this, IOKit has actually been fairly well documented. That documentation is relatively old, however, it's core API has changed very little and it's fundamental architecture hasn't changed at all.

So, the core IOKit documentation sources are:

• IOKit Fundamentals-> This describes the broad architecture and design of IOKit.

• IOKit Device Driver Design Guidelines-> This is a more detailed, practical guide to IOKit an KEXT development. It's not a directly relevant as I/O Kit Fundamentals, but it's still worth understanding.

• Accessing Hardware From Applications-> This covers the user space IOKit API, which are exactly the same APIs DriverKit use.

In addition, much of the kernel and many IOKit families are available in our opensource releases. DriverKit itself not available, but it can be helpful to see how an IOKit KEXT is implemented.

Looking at specifics:

However, I am uncertain how to declare or assign values to the two parameters, OSObject * target

By convention, "target" basicallymeans "the object this message/method is for". There are many places in IOKit where it's necessary to create a C function callback for something that would (ideally) be represented as a C++ method and the "target" method is provided to act as the "this" pointer in those functions.

and DequeueAction dequeueAction.

DequeAction is a function pointer typedef'ed in IOUserNetworkPacketQueue.h. Here is that definition:

typedef uint32_t (*DequeueAction)(OSObject *target,
                                      IOUserNetworkPacketQueue *queue,
                                      IOUserNetworkPacket **packetArray,
                                      uint32_t packetCount, void *refCon);

Additionally, are there predefined functionality or value lists for the two parameters void * refCon

By convention, a "refCon" pointer is an arbitrary value the system accepts from you at some earlier configuration point and then returns to you in each callback. It has no defined type (indeed, it doesn't even need to be a pointer) and is not managed by the system in ANY way. If you want to "attach" some value/data to the callback, you can use the refCon. If you won't want to... then don't.

and IOOptionBits options that can be used for specific feature configurations?

I'm not sure which method your actually looking at here, but "IOOptionBits" is a generic IOKit type used to indicate that a specific value is a bit field. In many places it's included in the function definition to provide for future API evolution, not because any options have actually been defined. As far as I can tell, that true of most the IOOptionBits in NetworkingDriverKt, with the exception of allocatePacket. That method defaults to "kIOUserNetworkNonBlocking" but can also be set to "kIOUserNetworkBlocking".

I'm not sure that this universal, but the default value of "0" like this:

    virtual bool
    initWithPool(IOUserNetworkPacketBufferPool *pool,
                 IOUserNetworkPacketDescriptor *descriptor,
                 IOOptionBits options = 0) LOCALONLY;

...often indicates that not options have been defined. If/when we introduce options in a method that had none, we often do so by defining a new "0" option (which preserves the existing behavior) alongside the new options we're adding, then change the default to new "0" option.

I would like to know the mechanism by which this method determines which QueueID a packet should be placed into or dequeued from for both Tx and Rx. I hope to get a detailed explanation of the exact usage, sample code, and whether this aligns with your statement, "The queue architecture above handles this."

Take a look at "Connecting a network driver" and see if that clarifies things.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

@DTS Engineer

We are indeed developing the PCIe DEXT from scratch, as we do not have any prior KEXT products related to this. I will take the time to thoroughly understand IOKit before consulting you on technical questions.

However, I’d like to start by discussing the concept with you to understand whether the functionality I am aiming for can be achieved on Apple’s platform.

To begin with, let’s assume that I have created four independent queues in the driver: IOUserNetworkTxSubmissionQueue/IOUserNetworkTxCompletionQueue, namely Queue[0], Queue[1], Queue[2], and Queue[3], where Queue[3] has the highest priority, and Queue[0] has the lowest.

Would it be possible for a user, through an application, to determine which specific application-generated packets should be processed and transmitted via one of the driver-created queues [Queue[0] ~ Queue[3]] based on priority? Our goal is to implement functionality on macOS similar to the "Realtek Bandwidth Control" feature on Windows, where the transmission priority of different applications can be managed to achieve traffic shaping by application priority.

The queue architecture above handles this. Every queue has an associated dispatch queue, which lets you control parallel packet processing.

I would like to provide additional information regarding the points mentioned in my previous response. The workflow of RSS (Receive-Side Scaling) on Windows is roughly as follows:

"RSS calculates the queue number for each packet based on a hash function and an indirection table. The upper layer uses this to assign processes to corresponding CPU cores. The hash value is calculated using TCP/UDP ports or IPv4/IPv6 addresses, and the hardware uses hash value bits [6:0] to look up the indirection table to obtain the queue number."

Does "The queue architecture above handles this" imply that the example I previously mentioned—"setting network packet transmission/receiving priorities for applications via an application"—is achievable?

Additionally, in macOS, can a user application transmit a specific value to the hardware device to enable packets from a particular application to be received by a specific Queue ID?

I am doing my best to clearly articulate the functionality we hope to achieve with Multi-queue, and I hope that our understanding of the issue is aligned.