setsockopt not setting the correct IP_TTL when called in a loop

I am working on a program on macOS that sends UDP probes in batches with increasing TTLs using the following code (stripped down)

int sd = socket(AF_INET, SOCK_DGRAM, 0);
if (sd < 0) { // error handling }

sockaddr_in saddr;
memset(&saddr, 0, sizeof (saddr));
saddr.sin_family = AF_INET;
saddr.sin_port = htons(port_num);
if (bind(sd, (const sockaddr*)&saddr, sizeof(saddr)) < 0) { // error handling }

for (int i = 1; i <= 5; ++i) {
    int ttl = i;
    if(setsockopt(sd, IPPROTO_IP, IP_TTL, &ttl, sizeof(ttl)) < 0) {
        // error handling
    }

    char probe[64] = {0};
    int cc = sendto(sd, probe, sizeof(probe), 0, &dest_addr, sizeof(dest_addr));
    if (cc < 0 || cc != sizeof(probe)) {
       // error handling
    }
}

I would expect the code to send five packets with TTLs from 1 to 5, however, if I inspect the network activity in Wireshark all the packets are sent with TTL set to 5. I have not set the socket as non-blocking and none of the API calls reports any error.

Is it incorrect to call setsockopt between sendto calls like this? Moreover, if I add a 10ms sleep between iterations, it works as expected.

This is a weird one. The setsockopt operation is synchronous but the sendto operation is not, so all the packets get queued and, as they come out of the queue, they all pick up the final TTL value. Honestly, I’m not sure if that’s expected behaviour or not.

If you wait for the socket buffer to empty before queuing the next send, do things work as expected? You can monitor this using SO_NWRITE.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

When you call setsockopt() in a loop to set the IP_TTL option on a socket, the IP_TTL value may not be set correctly because the socket is not closed and reopened for each iteration of the loop. The setsockopt() function sets options on a socket. In the case of setting the IP_TTL option, it specifies the time-to-live value for outgoing IP packets. However, the socket needs to be closed and reopened after each call to setsockopt() to ensure that the changes take effect. If the socket is not closed and reopened, subsequent calls to setsockopt() will not change the IP_TTL value because the previous socket is still in use with the previous value. This can result in unexpected behavior, such as packets not being sent with the expected TTL value. To fix this issue, you should close the socket and reopen it for each iteration of the loop where you call setsockopt() to set the IP_TTL value. Alternatively, you can use a separate socket for each iteration of the loop to avoid this problem. Here is an example of how you could modify your code to ensure that the IP_TTL value is set correctly: c

Copy code #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <string.h> #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <arpa/inet.h> int main() { int sock, ttl, i, optval; struct sockaddr_in addr; sock = socket(AF_INET, SOCK_DGRAM, 0); if (sock < 0) { perror("socket"); exit(EXIT_FAILURE); } memset(&addr, 0, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); addr.sin_port = htons(1234); for (i = 1; i <= 5; i++) { ttl = i; optval = ttl; if (setsockopt(sock, IPPROTO_IP, IP_TTL, &optval, sizeof(optval)) < 0) { perror("setsockopt"); exit(EXIT_FAILURE); } if (sendto(sock, "test", 4, 0, (struct sockaddr *)&addr, sizeof(addr)) < 0) { perror("sendto"); exit(EXIT_FAILURE); } close(sock); sock = socket(AF_INET, SOCK_DGRAM, 0); if (sock < 0) { perror("socket"); exit(EXIT_FAILURE); } } close(sock); exit(EXIT_SUCCESS); } In this modified code, the socket is closed and reopened for each iteration of the loop where setsockopt() is called to set the IP_TTL value. This ensures that the changes take effect and the correct TTL value is used for each packet.

https://apunkagames.cc/

@eskimo The value of the socket option SO_NWRITE just after the sendto call is always 0.

int value = 0;
socklen_t len = sizeof(value);

int res = getsockopt(sd, SOL_SOCKET, SO_NWRITE, &value, &len);

Bummer. So much for that plan.

The only other comparable scenario to this is traceroute, and it’s never going to hit this problem because:

  • Either it gets a reply, in which case it knows that the previous datagram hit the wire

  • Or it does not, in which case it only continues after a huge timeout

If you do what Khan65566 suggested, closing and re-opening the socket, does that resolve the issue?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@eskimo Closing and reopening the socket does work, but I don't know if it's a good idea to create a socket in every iteration. I also tried to set the IP_TTL option using ancillary data (similar to IP_TOS) but I guess that is not supported. Is there any other option (other than writing my own IP header)?

> I don't know if it's a good idea to create a socket in every iteration.

How many iterations are you doing? The example you posted has 5, and open and closing a socket 5 times is not going to be a problem. Neither is 50 times. Doing 5000 times OTOH, you’d want to think twice about (-:

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@eskimo The number of iterations is actually configurable, but I wouldn't expect it to be too high (< 100). However, I would still like to explore if there are any other options.

You have two options so far:

  • Add a delay

  • Close and re-open the socket

Of these, I have a strong preference for the second one.

The only other option I can think of is to switch to NWConnection but, honestly, I’m not confident that’ll work at all and, even if it does, that it’ll help in all scenarios. Remember that in some scenarios NWConnection is actually backed by BSD Sockets [1].

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] In other scenarios it uses the user-space networking stack.

setsockopt not setting the correct IP_TTL when called in a loop
 
 
Q