RE: [Network-qos-l] Iperf hangs
Should be 3306.
Connie
-----Original Message-----
From: Cottrell, Les
Sent: Monday, September 18, 2006 3:22 PM
To: Logg, Connie A.; Dimitrios Katramatos
Cc: iperf-users --at-- dast.nlanr.net; iepm-group; Network QoS list
Subject: RE: [Network-qos-l] Iperf hangs
Hi Dimitrios. How can you tell the sockets are associated with mysql?
Netstat -a shows the hostname and ports, but not the application. Is there some other indicator?
-----Original Message-----
From: owner-iepm-group --at-- lists1.slac.stanford.edu [mailto:owner-iepm-group --at-- lists1.slac.stanford.edu] On Behalf Of Logg, Connie A.
Sent: Monday, September 18, 2006 2:06 PM
To: Dimitrios Katramatos
Cc: iperf-users --at-- dast.nlanr.net; iepm-group; Network QoS list
Subject: RE: [Network-qos-l] Iperf hangs
I have been suspecting mysql, but I think it is not a mysql problem. The number of sockets allocated to mysql is not considered a lot. I have contacted several mysql lists and have been told that the mysql sockets are a very reasonable number. I am suspecting iperf/TCP at this point, but am looking into it further.
-----Original Message-----
From: Dimitrios Katramatos [mailto:dkat --at-- bnl.gov]
Sent: Monday, September 18, 2006 11:51 AM
To: Logg, Connie A.
Cc: iperf-users --at-- dast.nlanr.net; iepm-group; Network QoS list
Subject: Re: [Network-qos-l] Iperf hangs
Connie,
Well, the error message probably refers to the general state of the machine, and I see a ton of tcp and udp sockets open, not necessarily needed.
Most tcp sockets are in wait state and related with mysql and (old?) processes all on this machine.
The udp sockets are occupied by the same process 32529, pathload_snd, and have identical local and foreign addresses (localhost.localdomain:xxx to localhost.localdomain:xxx) (!?)
I suspect some iepm programs don't exit gracefully/clean up properly before exiting.
pathload_snd seems to definitely have gone nuts.
I would kill 32529 and restart MySQL for starters.
Ultimately reboot (after coordination with all users) if the above doesn't fix things.
Regards,
Dimitri
> [root --at-- iepmbw ~]# netstat -p
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign
> Address State PID/Program name
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39940 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39941 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39942 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39943 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39936 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39939 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39967 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39960 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39961 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39962 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39956 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39957 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39958 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39959 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39955 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39978 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39979 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39972 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39973 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39974 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39968 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39969 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39970 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39971 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39997 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39993 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39994 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39995 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:mysql
> iepmbw.bnl.org:39986 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39994
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39995
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39992
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39993
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39998
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39996
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39997
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39986
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39987
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39984
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39985
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39990
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39991
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39988
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39989
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39978
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39979
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39976
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39977
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39982
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39983
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39980
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39981
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39970
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39971
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39968
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39969
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39974
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39975
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39972
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39973
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39962
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39963
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39960
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39961
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39966
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39967
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39964
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39965
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39955
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39952
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39958
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39959
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39956
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39957
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39951
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39948
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39949
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39938
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39939
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39936
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39937
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39942
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39943
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39940
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:39941
> iepmbw.bnl.org:mysql TIME_WAIT -
> tcp 0 1 iepmbw.bnl.org:39804
> iepm.desy.de:5003 SYN_SENT 3215/tlaytcp
> tcp 0 42653736 iepmbw.bnl.org:39927
> iepmbw.twaren.net:5000 ESTABLISHED 3648/iperf
> tcp 0 0 iepmbw.bnl.org:39953
> cache3.bnl.gov:http TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:5000
> snv1-10g.ultralight.o:39381 ESTABLISHED 32309/bw-iperf-serv
> tcp 0 3058820 iepmbw.bnl.org:39639
> iepm-bw.slac.stanford.:5003 ESTABLISHED 2567/tlaytcp
> tcp 0 1 iepmbw.bnl.org:39707
> rainbow.inp.nsk.su:5000 SYN_SENT 2887/iperf
> tcp 0 144 iepmbw.bnl.org:ssh
> ::ffff:130.199.80.71:1451 ESTABLISHED 11520/4
> tcp 0 0 iepmbw.bnl.org:ssh
> dhcp-25-58.slac.stanf:56375 ESTABLISHED 6382/sshd: ytl [pri
> tcp 0 0 localhost.localdomain:39954
> localhost.localdomain:smtp TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:ssh
> dhcp-25-58.slac.stanf:56278 ESTABLISHED 978/sshd: ytl [priv
> tcp 0 0 iepmbw.bnl.org:http
> charon.s3.bnl.org:62378 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:http
> charon.s3.bnl.org:62379 TIME_WAIT -
> tcp 0 0 iepmbw.bnl.org:ssh
> dhcp-25-58.slac.stanf:51848 ESTABLISHED 13116/sshd: ytl [pr
> udp 0 0 localhost.localdomain:56704
> localhost.localdomain:56704 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54913
> localhost.localdomain:54913 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57346
> localhost.localdomain:57346 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:33027
> localhost.localdomain:33027 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:60548
> localhost.localdomain:60548 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55813
> localhost.localdomain:55813 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:34822
> localhost.localdomain:34822 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57095
> localhost.localdomain:57095 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56712
> localhost.localdomain:56712 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57097
> localhost.localdomain:57097 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56714
> localhost.localdomain:56714 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:34955
> localhost.localdomain:34955 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54668
> localhost.localdomain:54668 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:34957
> localhost.localdomain:34957 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:34446
> localhost.localdomain:34446 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:34959
> localhost.localdomain:34959 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57360
> localhost.localdomain:57360 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55057
> localhost.localdomain:55057 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55314
> localhost.localdomain:55314 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54675
> localhost.localdomain:54675 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56212
> localhost.localdomain:56212 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55189
> localhost.localdomain:55189 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56086
> localhost.localdomain:56086 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57495
> localhost.localdomain:57495 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:33944
> localhost.localdomain:33944 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56217
> localhost.localdomain:56217 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55578
> localhost.localdomain:55578 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54683
> localhost.localdomain:54683 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55452
> localhost.localdomain:55452 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56093
> localhost.localdomain:56093 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55326
> localhost.localdomain:55326 ESTABLISHED 32529/pathload_snd
> udp 0 0 iepmbw.bnl.org:54303
> wanmon1.cern.ch:55001 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:58528
> localhost.localdomain:58528 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55329
> localhost.localdomain:55329 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:60450
> localhost.localdomain:60450 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54819
> localhost.localdomain:54819 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56100
> localhost.localdomain:56100 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55589
> localhost.localdomain:55589 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:60582
> localhost.localdomain:60582 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55207
> localhost.localdomain:55207 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:60456
> localhost.localdomain:60456 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:36393
> localhost.localdomain:36393 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:38058
> localhost.localdomain:38058 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:60203
> localhost.localdomain:60203 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:35372
> localhost.localdomain:35372 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55086
> localhost.localdomain:55086 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54447
> localhost.localdomain:54447 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55984
> localhost.localdomain:55984 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:35249
> localhost.localdomain:35249 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:39730
> localhost.localdomain:39730 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57651
> localhost.localdomain:57651 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56628
> localhost.localdomain:56628 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55861
> localhost.localdomain:55861 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57910
> localhost.localdomain:57910 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54583
> localhost.localdomain:54583 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54584
> localhost.localdomain:54584 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56505
> localhost.localdomain:56505 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56762
> localhost.localdomain:56762 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55227
> localhost.localdomain:55227 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55996
> localhost.localdomain:55996 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55485
> localhost.localdomain:55485 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54462
> localhost.localdomain:54462 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54975
> localhost.localdomain:54975 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55488
> localhost.localdomain:55488 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56641
> localhost.localdomain:56641 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55106
> localhost.localdomain:55106 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:60867
> localhost.localdomain:60867 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57284
> localhost.localdomain:57284 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55749
> localhost.localdomain:55749 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:36422
> localhost.localdomain:36422 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54471
> localhost.localdomain:54471 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54728
> localhost.localdomain:54728 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:58057
> localhost.localdomain:58057 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:35402
> localhost.localdomain:35402 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57291
> localhost.localdomain:57291 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57292
> localhost.localdomain:57292 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57293
> localhost.localdomain:57293 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:58446
> localhost.localdomain:58446 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:60879
> localhost.localdomain:60879 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54352
> localhost.localdomain:54352 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57681
> localhost.localdomain:57681 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56530
> localhost.localdomain:56530 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56531
> localhost.localdomain:56531 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:60628
> localhost.localdomain:60628 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57813
> localhost.localdomain:57813 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57686
> localhost.localdomain:57686 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:57943
> localhost.localdomain:57943 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:58200
> localhost.localdomain:58200 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54745
> localhost.localdomain:54745 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54490
> localhost.localdomain:54490 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56027
> localhost.localdomain:56027 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55644
> localhost.localdomain:55644 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54621
> localhost.localdomain:54621 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55646
> localhost.localdomain:55646 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:33503
> localhost.localdomain:33503 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:58848
> localhost.localdomain:58848 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56033
> localhost.localdomain:56033 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54626
> localhost.localdomain:54626 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:58979
> localhost.localdomain:58979 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56804
> localhost.localdomain:56804 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55525
> localhost.localdomain:55525 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55142
> localhost.localdomain:55142 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:59239
> localhost.localdomain:59239 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56168
> localhost.localdomain:56168 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:58985
> localhost.localdomain:58985 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56170
> localhost.localdomain:56170 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54635
> localhost.localdomain:54635 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:58732
> localhost.localdomain:58732 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55021
> localhost.localdomain:55021 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:60654
> localhost.localdomain:60654 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56432
> localhost.localdomain:56432 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:59249
> localhost.localdomain:59249 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:60658
> localhost.localdomain:60658 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:35699
> localhost.localdomain:35699 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:58612
> localhost.localdomain:58612 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:60917
> localhost.localdomain:60917 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:59126
> localhost.localdomain:59126 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:55160
> localhost.localdomain:55160 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:59641
> localhost.localdomain:59641 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:59642
> localhost.localdomain:59642 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:46716
> localhost.localdomain:46716 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:46588
> localhost.localdomain:46588 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:56829
> localhost.localdomain:56829 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:33022
> localhost.localdomain:33022 ESTABLISHED 32529/pathload_snd
> udp 0 0 localhost.localdomain:54911
> localhost.localdomain:54911 ESTABLISHED 32529/pathload_snd Active UNIX
> domain sockets (w/o servers)
> Proto RefCnt Flags Type State I-Node PID/Program
> name Path
> unix 13 [ ] DGRAM 5506
> 2381/syslogd /dev/log
> unix 2 [ ] DGRAM 6154
> 2702/hald @/var/run/hal/hotplug_socket
> unix 2 [ ] DGRAM 3799
> 1340/udevd @udevd
> unix 2 [ ] DGRAM 33280398
> 3929/netstat
> unix 2 [ ] DGRAM 31490596
> 694/owampd
> unix 2 [ ] DGRAM 31415300
> 32449/thrulayd
> unix 3 [ ] STREAM CONNECTED 28895933
> 2683/dbus-daemon-1 /var/run/dbus/system_bus_socket
> unix 3 [ ] STREAM CONNECTED 28895932
> 20684/cupsd
> unix 3 [ ] STREAM CONNECTED 14357730 13136/sshd:
> ytl --at-- not
> unix 3 [ ] STREAM CONNECTED 14357729
> 13137/sftp-server
> unix 3 [ ] STREAM CONNECTED 14357728 13136/sshd:
> ytl --at-- not
> unix 3 [ ] STREAM CONNECTED 14357727
> 13137/sftp-server
> unix 3 [ ] STREAM CONNECTED 14357715 13116/sshd:
> ytl [pr
> unix 3 [ ] STREAM CONNECTED 14357714 13136/sshd:
> ytl --at-- not
> unix 3 [ ] STREAM CONNECTED 339345 6382/sshd:
> ytl [pri
> unix 3 [ ] STREAM CONNECTED 339344
> 6407/3
> unix 3 [ ] STREAM CONNECTED 264943 978/sshd: ytl
> [priv
> unix 3 [ ] STREAM CONNECTED 264942
> 1011/0
> unix 3 [ ] STREAM CONNECTED 6152
> 2683/dbus-daemon-1 /var/run/dbus/system_bus_socket
> unix 3 [ ] STREAM CONNECTED 6151
> 2702/hald
> unix 2 [ ] DGRAM 6125
> 2693/rhnsd
> unix 3 [ ] STREAM CONNECTED 6110
> 2683/dbus-daemon-1
> unix 3 [ ] STREAM CONNECTED 6109
> 2683/dbus-daemon-1
> unix 2 [ ] DGRAM 6072
> 2657/xfs
> unix 2 [ ] DGRAM 6013
> 2627/crond
> unix 2 [ ] DGRAM 5969
> 2608/gpm
> unix 2 [ ] DGRAM 5955
> 2598/clientmqueue
> unix 2 [ ] DGRAM 5936
> 2590/sendmail: acce
> unix 2 [ ] DGRAM 5857
> 2571/ntpd
> unix 2 [ ] DGRAM 5828
> 2556/xinetd
> unix 2 [ ] DGRAM 5514 2385/klogd
Logg, Connie A. wrote:
> I have been having problems with 'too many orphaned sockets', and
> after watching the count for several days I never seen A lot (mostly 0 and < 5).
>
> However while watching the machine I also notice that at times there are several (usually 8) connections from iepmbw.bnl.org to port 5000 on iepm-bw.slac.stanford.edu.
>
> I just ran a multistream iperf from the command line on the bnl node to the slac node and it appears to hang when done and not return to the command line. I waited over a minute. Here is the output:
>
> iepm --at-- iepmbw ~/v3src]$ /home/iepm/v3src/bin/iperf -c 134.79.240.38 -p 5000 -i 5 -t 20 -w 8M -P 8
> ------------------------------------------------------------
> Client connecting to 134.79.240.38, TCP port 5000 TCP window size:
> 16.0 MByte (WARNING: requested 8.00 MByte)
> ------------------------------------------------------------
> [ 10] local 192.203.218.61 port 33094 connected with 134.79.240.38
> port 5000 [ 4] local 192.203.218.61 port 33088 connected with
> 134.79.240.38 port 5000 [ 3] local 192.203.218.61 port 33087
> connected with 134.79.240.38 port 5000 [ 5] local 192.203.218.61 port
> 33089 connected with 134.79.240.38 port 5000 [ 6] local
> 192.203.218.61 port 33090 connected with 134.79.240.38 port 5000 [ 7]
> local 192.203.218.61 port 33091 connected with 134.79.240.38 port 5000
> [ 8] local 192.203.218.61 port 33092 connected with 134.79.240.38
> port 5000 [ 9] local 192.203.218.61 port 33093 connected with
> 134.79.240.38 port 5000 [ 5] 0.0- 5.0 sec 34.0 MBytes 57.0
> Mbits/sec [ 7] 0.0- 5.0 sec 19.8 MBytes 33.2 Mbits/sec [ 9] 0.0-
> 5.0 sec 32.3 MBytes 54.1 Mbits/sec [ 8] 0.0- 5.0 sec 23.8 MBytes
> 39.9 Mbits/sec [ 4] 0.0- 5.0 sec 38.4 MBytes 64.4 Mbits/sec [ 8]
> 5.0-10.0 sec 16.4 MBytes 27.5 Mbits/sec
> [ 7] 5.0-10.0 sec 76.8 MBytes 129 Mbits/sec
> [ 5] 5.0-10.0 sec 47.9 MBytes 80.3 Mbits/sec [ 4] 5.0-10.0 sec
> 53.9 MBytes 90.4 Mbits/sec [ 9] 5.0-10.0 sec 20.2 MBytes 33.9
> Mbits/sec
> [ 7] 10.0-15.0 sec 68.2 MBytes 114 Mbits/sec
> [ 4] 10.0-15.0 sec 43.6 MBytes 73.1 Mbits/sec [ 5] 10.0-15.0 sec
> 42.2 MBytes 70.9 Mbits/sec [ 9] 10.0-15.0 sec 20.6 MBytes 34.6
> Mbits/sec [ 8] 10.0-15.0 sec 36.9 MBytes 62.0 Mbits/sec [ 4]
> 15.0-20.0 sec 46.1 MBytes 77.3 Mbits/sec
> [ 4] 0.0-20.3 sec 182 MBytes 75.3 Mbits/sec
> [ 8] 15.0-20.0 sec 37.5 MBytes 62.9 Mbits/sec
> [ 8] 0.0-20.3 sec 115 MBytes 47.4 Mbits/sec
> [ 5] 15.0-20.0 sec 38.3 MBytes 64.2 Mbits/sec
> [ 5] 0.0-20.4 sec 162 MBytes 66.8 Mbits/sec
> [SUM] 0.0-20.4 sec 607 MBytes 250 Mbits/sec
> [ 9] 15.0-20.0 sec 28.3 MBytes 47.5 Mbits/sec
> [ 9] 0.0-20.7 sec 101 MBytes 41.0 Mbits/sec
> [ 7] 15.0-20.0 sec 13.6 MBytes 22.8 Mbits/sec
> [ 7] 0.0-21.2 sec 178 MBytes 70.4 Mbits/sec
> [iepm --at-- iepmbw ~/v3src]$ I did a <ctrl c> to get out of it.
>
> Does anyone have any ideas what may be causing this?
>
> I am running the iperf again, and it is hung before it finishes. I get the following output:
> [iepm --at-- iepmbw bin]$ /home/iepm/v3src/bin/iperf -c 134.79.240.38 -p 5000
> -i 5 -t 20 -w 8M -P 8
> ------------------------------------------------------------
> Client connecting to 134.79.240.38, TCP port 5000 TCP window size:
> 16.0 MByte (WARNING: requested 8.00 MByte)
> ------------------------------------------------------------
> [ 10] local 192.203.218.61 port 34526 connected with 134.79.240.38
> port 5000 [ 3] local 192.203.218.61 port 34519 connected with
> 134.79.240.38 port 5000 [ 4] local 192.203.218.61 port 34520
> connected with 134.79.240.38 port 5000 [ 5] local 192.203.218.61 port
> 34521 connected with 134.79.240.38 port 5000 [ 8] local
> 192.203.218.61 port 34524 connected with 134.79.240.38 port 5000 [ 7]
> local 192.203.218.61 port 34523 connected with 134.79.240.38 port 5000
> [ 9] local 192.203.218.61 port 34525 connected with 134.79.240.38
> port 5000 [ 6] local 192.203.218.61 port 34522 connected with
> 134.79.240.38 port 5000 [ 8] 0.0- 5.0 sec 24.3 MBytes 40.7
> Mbits/sec [ 7] 0.0- 5.0 sec 24.5 MBytes 41.1 Mbits/sec [ 7]
> 5.0-10.0 sec 20.4 MBytes 34.2 Mbits/sec [ 8] 5.0-10.0 sec 15.8
> MBytes 26.5 Mbits/sec [ 7] 10.0-15.0 sec 33.0 MBytes 55.3
> Mbits/sec [ 8] 10.0-15.0 sec 24.4 MBytes 40.9 Mbits/sec [ 8]
> 15.0-20.0 sec 32.5 MBytes 54.5 Mbits/sec [ 8] 0.0-20.3 sec 96.9
> MBytes 40.1 Mbits/sec [ 7] 15.0-20.0 sec 37.4 MBytes 62.8
> Mbits/sec
> [ 7] 0.0-20.4 sec 115 MBytes 47.5 Mbits/sec
>
> And it just hangs there while I see on the SLAC end
> 48cal --at-- iepm-bw: netstat -a | grep bnl
> tcp 0 0 iepm-bw.slac.stanford:33962 iepmbw.bnl.org:ssh ESTABLISHED
> tcp 0 0 iepm-bw.slac.stanford.:5000 iepmbw.bnl.org:34519 ESTABLISHED
> tcp 0 0 iepm-bw.slac.stanford.:5000 iepmbw.bnl.org:34526 ESTABLISHED
> tcp 0 0 iepm-bw.slac.stanford.:5000 iepmbw.bnl.org:34525 ESTABLISHED
> tcp 0 0 iepm-bw.slac.stanford.:5000 iepmbw.bnl.org:34522 ESTABLISHED
> tcp 0 0 iepm-bw.slac.stanford.:5000 iepmbw.bnl.org:34520 ESTABLISHED
> tcp 0 0 iepm-bw.slac.stanford.:5000 iepmbw.bnl.org:34521 ESTABLISHED
> tcp 0 0 iepm-bw.slac.stanford.:5000 iepmbw.bnl.org:33087 ESTABLISHED
> tcp 0 0 iepm-bw.slac.stanford.:5000 iepmbw.bnl.org:33094 ESTABLISHED
> tcp 0 0 iepm-bw.slac.stanford.:5000 iepmbw.bnl.org:33090 ESTABLISHED
>
> I eventually killed the iperf command runnning from BNL, and the
> sockets are still allocated Several minutes later on the slac node.
>
>
> Any ideas?
>
> Thanks Connie
> Connie Logg, Network Analyst
> Stanford Linear Accelerator Center
> ph: 650-926-2879
> "Happiness is found along the way, not at the end of the road, and 'IF' is the middle word in life."
> _______________________________________________
> Network-qos-l mailing list
> Network-qos-l --at-- lists.bnl.gov
> http://lists.bnl.gov/mailman/listinfo/network-qos-l
>
--
Dimitrios Katramatos, Ph.D.
RCF/ATLAS Group
Physics Department
Brookhaven National Laboratory
Building 510M, office M2-36
Upton, NY 11793-5000
email: dkat --at-- bnl.gov
tel: 631-344-4008
fax: 631-344-7616