network plumbing (was: UDP question)
On Fri, 16 Sep 2005, Rosanna wrote:
> The only way I got a better UDP throughput (around 91 Mbps) was
> changing the socket buffer size (-w) and the datagram length (-l). I
> used the following commands to do so:
>
> iperf -c serveraddress -u -b 100G -w 128k -l 32k
>
> iperf -s -u -w 128k -l 32k
>
> I will also appreciate if someone knows what is happening.
There are a number of queues in the protocol stacks of operating
systems. A very appromixative view is something like: 1) the socket
buffer; 2) one queue for IP packets (TCP, UDP, etc. multiplexed/mixed;
3) one software queue for the network interface (IPv4, IPv6, IPX,
appletalk, whatever mixed together), 4) another staging area on the
network chip itself. The reality may be more complex, and
system-dependent. Then you have a number of system threads (pieces of
code) pumping for all this plumbing.
There is (at least) one good paper describing the linux TCP/IP stack
from this point of view
http://datatag.web.cern.ch/datatag/publications.html
"A Map of the Networking Code in Linux Kernel 2.4.20"
It's obviously harder to get such information for closed systems like
the ones from Microsoft.
You may also find this bookmarks list interesting
<http://www.csm.ornl.gov/~dunigan/netperf/netlinks.html>
When these queues become full (and some of them do when you stress
your system), the pumping threads have a problem. They may solve it
either by just stopping to pump (blocking), or by dropping (leaking)
the packet they were about to move, among others because they do not
want to hurt the latency...
Please note that, depending on the relative size of the queues (think
iperf -w) and the relative "pumping" speed (including transient speeds
and events!), you may have leaks at some points whereas the overall
throughput is still below 100Mb/s.
TCP does a good job of hiding all this complexity (to a certain
point).
I am afraid you won't find any simple answer to your questions. But
you may enjoy the journey :-)
Cheers,
Marc.
--
So einfach wie möglich. Aber nicht einfacher -- Albert Einstein
http://marc.herbert.free/noq/