Debug logs for UDP communication?

Question

Created Jun ’22

Replies 4

Boosts 0

Views 1.1k

Participants 2

We have some extensive tests which exercise UDP communication. Some of these tests fail fairly often due to the UDP packet being dropped by the kernel (or related reasons). These tests use loopback interface for communication. I have been looking to see if there's a way to pinpoint or narrow down exactly why a particular packet was dropped by the kernel. Looking at the kernel code, like here https://github.com/apple-opensource/xnu/blob/master/bsd/netinet/udp_usrreq.c#L1463 it appears that there are log message that get written out during some of this communication. However, looking at what KERNEL_DEBUG stands for, it appears that it's:

/*
 * Traced only on debug kernels.
 */
#define KDBG_DEBUG(x, ...) KDBG_(_DEBUG, x, ## __VA_ARGS__, 4, 3, 2, 1, 0)

So I don't think these logs get generated in a regular release build of the OS.

Are there any other ways we can generate similar logs or any other tools that will give a clearer picture of why the packet might be drop?

Boost

Answer 1

Systems Engineer OP

Apple

Jun ’22

it appears that there are log message that get written out during some of this communication

What are the log messages that are written out during this time? Also are there any errno logs generated here?

0

Answer 2

jaikiran OP

Jun ’22

Thank you for responding @meaton.

What are the log messages that are written out during this time? Also are there any errno logs generated here?

What I meant by the log message being written out was that the source code (the one I'm guessing is the right one https://github.com/apple-opensource/xnu/blob/master/bsd/netinet/udp_usrreq.c#L1463?) has log statements like the KERNEL_DEBUG. However, I haven't found any kind of log messages in the system logs, that relate to UDP communication when these packets go missing. So I was wondering if there was some way to get any kind of logging generated at runtime on these systems?

You note above errno logs in your reply. Is there some specific file/location where those get logged, so that I can take a look?

Not every single run shows up this issue, but it does happen consistently enough that I think understanding the real reason for them being dropped would help ruling out any unexpected bugs either in the application or some other place.

0

Answer 3

Systems Engineer OP

Apple

Jun ’22

So I was wondering if there was some way to get any kind of logging generated at runtime on these systems? You note above errno logs in your reply. Is there some specific file/location where those get logged, so that I can take a look?

The errno values usually show up in a sysdiagnose, if you take a look at the logarchive file around the time the issue took place. See the section under sysdiagnose here.

0

Answer 4

jaikiran OP

Jun ’22

The errno values usually show up in a sysdiagnose, if you take a look at the logarchive file around the time the issue took place. See the section under sysdiagnose here.

@meaton, thank you very much for that input. That certainly helped in understanding one UDP related test failure on my local setup. The logarchive had a couple of lines of log messages which provided the necessary hint that there was a system level content filter extension which was getting involved in this flow and that was causing the unexpected odd failure. Dealing with the system level content filter extension to not interfere with this test, helped the test pass.

Of course, there are several other tests that fail intermittently (and in some cases there's no content filter involved). So I'll will be reviewing those failures through these sysdiagnose data. Hopefully, I'll find the necessary data to narrow them down too. Thanks again.

0