TSocket Option: kSendBuffer/kRecvBuffer

[augustin]

Hi,

I have a question regarding the buffer sizes of a TSocket object on a Linux machine. The following code, when used in the hserv/hclient example found in tutorials/net/, shows a strange behaviour.

Int_t val;

val = -1;
sock->GetOption(kSendBuffer, val);
printf("sendbuffer size: %d\n", val);

Int_t status = sock->SetOption(kSendBuffer, val);
printf("set sendbuffer size returned: %d\n", status);

val = -1;
sock->GetOption(kSendBuffer, val);
printf("sendbuffer size: %d\n", val);

I get the following output from this:

sendbuffer size: 2626560
set sendbuffer size returned: 0
sendbuffer size: 425984

The result 0 in the second line means that SetOption claimed to have successfully changed the value. Furthermore, it seems 425984 is the maximum amount of memory I can assign here.

After some research, I found that when this value is set, the Linux kernel doubles it. This means the actual maximum value is 425984/2 = 212992. This maximum is given through the values in sysctl through:

net.core.rmem_max
net.core.wmem_max

…which are 212992 by default on my machine. Changing these values changes the maximum buffer size that can be set via TSocket::SetOption.

So, my actual question is:
1. How can the initial size of the buffer, as seen in the first output line above, be much larger than this maximum value?

And as bonus questions:
2. Is it possible to set such a larger value despite (i.e., ignoring) what is set in sysctl?
3. Why does TSocket::SetOption return a 0, even though the desired value has not been set?

Best regards & many thanks in advance!
Sven

[ganis]

Hi,

As you may have seen, those ROOT calls are just wrappers around the relevant setsockopt/getsockopt system calls.
I have tried on Ubuntu and I get the same results as you.
The question is therefore why the first call to getsockopt returns a value inconsistent with the system default settings (for ubuntu they are under /proc/sys/net/core/wmem_default and /proc/sys/net/core/wmem_max).
One should check if on other systems is the same.

@augustin

3. Why does TSocket::SetOption return a 0, even though the desired value has not been set?

According to its man page, setsockopt may or may not raise an error if the value is not valid (which is your case); ROOT returns an error only when setsockopt does. This needs to be at least documented.

Perhaps the best is, when relevant, to cross check the setting as you do in your test macro.

G Ganis

[augustin]

Hi,

@ganis

The question is therefore why the first call to getsockopt returns a value inconsistent with the system default settings (for ubuntu they are under /proc/sys/net/core/wmem_default and /proc/sys/net/core/wmem_max).
One should check if on other systems is the same.

I think, I can now answer that from my research since yesterday:
The Linux network stack uses TCP autotuning, which tries to adapt these buffer sizes to the network’s performance. So, this much higher value is the actual value that is in use. However, it does not honour the maximum set by net.core.*mem_max. Instead it honours the values set in net.ipv4.tcp_*mem!

@ganis

@augustin

3. Why does TSocket::SetOption return a 0, even though the desired value has not been set?

According to its man page, setsockopt may or may not raise an error if the value is not valid (which is your case); ROOT returns an error only when setsockopt does. This needs to be at least documented.

Ah, okay. I had not found this in the manual …
You are right, the ROOT docs should let the user know about this.

Thanks so far!
Sven