Ping responses missing but can see them in tcpdump
up vote
0
down vote
favorite
Redhat 2.6.32-754.el6.x86_64
I have an FPGA card acting as a NIC with an associated driver.
On my RH host, if I run ping through the FPGA NIC I see lots of missing responses. However, if I also run tcpdump and filter on icmp I can see the responses coming through.
Here's an example output from ping (note the many many missing sequence numbers):
from 172.16.1.9: icmp_seq=465 ttl=128 time=0.600 ms
64 bytes from 172.16.1.9: icmp_seq=467 ttl=128 time=0.490 ms
64 bytes from 172.16.1.9: icmp_seq=480 ttl=128 time=0.565 ms
64 bytes from 172.16.1.9: icmp_seq=482 ttl=128 time=0.590 ms
64 bytes from 172.16.1.9: icmp_seq=516 ttl=128 time=0.448 ms
64 bytes from 172.16.1.9: icmp_seq=526 ttl=128 time=0.649 ms
64 bytes from 172.16.1.9: icmp_seq=528 ttl=128 time=0.534 ms
64 bytes from 172.16.1.9: icmp_seq=539 ttl=128 time=0.424 ms
64 bytes from 172.16.1.9: icmp_seq=546 ttl=128 time=0.606 ms
64 bytes from 172.16.1.9: icmp_seq=562 ttl=128 time=0.521 ms
64 bytes from 172.16.1.9: icmp_seq=569 ttl=128 time=0.651 ms
64 bytes from 172.16.1.9: icmp_seq=591 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=617 ttl=128 time=0.652 ms
64 bytes from 172.16.1.9: icmp_seq=642 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=643 ttl=128 time=0.672 ms
64 bytes from 172.16.1.9: icmp_seq=644 ttl=128 time=0.443 ms
64 bytes from 172.16.1.9: icmp_seq=657 ttl=128 time=0.427 ms
64 bytes from 172.16.1.9: icmp_seq=668 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=704 ttl=128 time=0.332 ms
64 bytes from 172.16.1.9: icmp_seq=741 ttl=128 time=0.486 ms
64 bytes from 172.16.1.9: icmp_seq=742 ttl=128 time=0.478 ms
64 bytes from 172.16.1.9: icmp_seq=751 ttl=128 time=0.513 ms
64 bytes from 172.16.1.9: icmp_seq=753 ttl=128 time=0.511 ms
From 172.16.0.156 icmp_seq=788 Destination Host Unreachable
From 172.16.0.156 icmp_seq=789 Destination Host Unreachable
From 172.16.0.156 icmp_seq=790 Destination Host Unreachable
From 172.16.0.156 icmp_seq=792 Destination Host Unreachable
From 172.16.0.156 icmp_seq=793 Destination Host Unreachable
From 172.16.0.156 icmp_seq=794 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=798 ttl=128 time=0.671 ms
64 bytes from 172.16.1.9: icmp_seq=799 ttl=128 time=0.608 ms
64 bytes from 172.16.1.9: icmp_seq=801 ttl=128 time=0.538 ms
64 bytes from 172.16.1.9: icmp_seq=814 ttl=128 time=0.402 ms
64 bytes from 172.16.1.9: icmp_seq=923 ttl=128 time=0.458 ms
From 172.16.0.156 icmp_seq=952 Destination Host Unreachable
From 172.16.0.156 icmp_seq=953 Destination Host Unreachable
From 172.16.0.156 icmp_seq=954 Destination Host Unreachable
From 172.16.0.156 icmp_seq=956 Destination Host Unreachable
From 172.16.0.156 icmp_seq=957 Destination Host Unreachable
From 172.16.0.156 icmp_seq=958 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=966 ttl=128 time=0.472 ms
From 172.16.0.156 icmp_seq=979 Destination Host Unreachable
From 172.16.0.156 icmp_seq=980 Destination Host Unreachable
From 172.16.0.156 icmp_seq=981 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=993 ttl=128 time=0.586 ms
^C
--- 172.16.1.9 ping statistics ---
997 packets transmitted, 96 received, +15 errors, 90% packet loss, time 996823ms
rtt min/avg/max/mdev = 0.332/31.837/2001.563/226.238 ms, pipe 3
And here's a snippet from tcpdump that was running at the same time:
16:27:24.300566 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 990, length 64
0x0000: 4500 0054 571e 0000 8001 89c5 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 e573 de11 03de ecd8 f65b .......s.......[
0x0020: 0000 0000 9294 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:25.300102 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 991, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 5073 de11 03df edd8 f65b ......Ps.......[
0x0020: 0000 0000 1e94 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:25.300683 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 991, length 64
0x0000: 4500 0054 571f 0000 8001 89c4 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 5873 de11 03df edd8 f65b ......Xs.......[
0x0020: 0000 0000 1e94 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:26.300166 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 992, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 0a72 de11 03e0 eed8 f65b .......r.......[
0x0020: 0000 0000 6394 0400 0000 0000 1011 1213 ....c...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:26.300766 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 992, length 64
0x0000: 4500 0054 5720 0000 8001 89c3 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 1272 de11 03e0 eed8 f65b .......r.......[
0x0020: 0000 0000 6394 0400 0000 0000 1011 1213 ....c...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:27.300141 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 993, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 2071 de11 03e1 efd8 f65b .......q.......[
0x0020: 0000 0000 4c94 0400 0000 0000 1011 1213 ....L...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:27.300694 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 993, length 64
0x0000: 4500 0054 5721 0000 8001 89c2 ac10 0109 E..TW!..........
0x0010: ac10 009c 0000 2871 de11 03e1 efd8 f65b ......(q.......[
0x0020: 0000 0000 4c94 0400 0000 0000 1011 1213 ....L...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
^C
1034 packets captured
1036 packets received by filter
0 packets dropped by kernel
1034 packets captured
1036 packets received by filter
0 packets dropped by kernel
So hopefully from that you can see the echo replies in the tcpdump output with sequence numbers 992, 991 and 990 which are completely missed by ping.
If I swap over to the built in NIC, ping works as expected. This made me suspect my driver. However, if the driver was to blame would tcpdump also not see the icmp replies?
rhel drivers ping tcpdump
asked Nov 22 at 16:51
Adi
1614
add a comment |
up vote
0
down vote
favorite
Redhat 2.6.32-754.el6.x86_64
I have an FPGA card acting as a NIC with an associated driver.
On my RH host, if I run ping through the FPGA NIC I see lots of missing responses. However, if I also run tcpdump and filter on icmp I can see the responses coming through.
Here's an example output from ping (note the many many missing sequence numbers):
from 172.16.1.9: icmp_seq=465 ttl=128 time=0.600 ms
64 bytes from 172.16.1.9: icmp_seq=467 ttl=128 time=0.490 ms
64 bytes from 172.16.1.9: icmp_seq=480 ttl=128 time=0.565 ms
64 bytes from 172.16.1.9: icmp_seq=482 ttl=128 time=0.590 ms
64 bytes from 172.16.1.9: icmp_seq=516 ttl=128 time=0.448 ms
64 bytes from 172.16.1.9: icmp_seq=526 ttl=128 time=0.649 ms
64 bytes from 172.16.1.9: icmp_seq=528 ttl=128 time=0.534 ms
64 bytes from 172.16.1.9: icmp_seq=539 ttl=128 time=0.424 ms
64 bytes from 172.16.1.9: icmp_seq=546 ttl=128 time=0.606 ms
64 bytes from 172.16.1.9: icmp_seq=562 ttl=128 time=0.521 ms
64 bytes from 172.16.1.9: icmp_seq=569 ttl=128 time=0.651 ms
64 bytes from 172.16.1.9: icmp_seq=591 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=617 ttl=128 time=0.652 ms
64 bytes from 172.16.1.9: icmp_seq=642 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=643 ttl=128 time=0.672 ms
64 bytes from 172.16.1.9: icmp_seq=644 ttl=128 time=0.443 ms
64 bytes from 172.16.1.9: icmp_seq=657 ttl=128 time=0.427 ms
64 bytes from 172.16.1.9: icmp_seq=668 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=704 ttl=128 time=0.332 ms
64 bytes from 172.16.1.9: icmp_seq=741 ttl=128 time=0.486 ms
64 bytes from 172.16.1.9: icmp_seq=742 ttl=128 time=0.478 ms
64 bytes from 172.16.1.9: icmp_seq=751 ttl=128 time=0.513 ms
64 bytes from 172.16.1.9: icmp_seq=753 ttl=128 time=0.511 ms
From 172.16.0.156 icmp_seq=788 Destination Host Unreachable
From 172.16.0.156 icmp_seq=789 Destination Host Unreachable
From 172.16.0.156 icmp_seq=790 Destination Host Unreachable
From 172.16.0.156 icmp_seq=792 Destination Host Unreachable
From 172.16.0.156 icmp_seq=793 Destination Host Unreachable
From 172.16.0.156 icmp_seq=794 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=798 ttl=128 time=0.671 ms
64 bytes from 172.16.1.9: icmp_seq=799 ttl=128 time=0.608 ms
64 bytes from 172.16.1.9: icmp_seq=801 ttl=128 time=0.538 ms
64 bytes from 172.16.1.9: icmp_seq=814 ttl=128 time=0.402 ms
64 bytes from 172.16.1.9: icmp_seq=923 ttl=128 time=0.458 ms
From 172.16.0.156 icmp_seq=952 Destination Host Unreachable
From 172.16.0.156 icmp_seq=953 Destination Host Unreachable
From 172.16.0.156 icmp_seq=954 Destination Host Unreachable
From 172.16.0.156 icmp_seq=956 Destination Host Unreachable
From 172.16.0.156 icmp_seq=957 Destination Host Unreachable
From 172.16.0.156 icmp_seq=958 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=966 ttl=128 time=0.472 ms
From 172.16.0.156 icmp_seq=979 Destination Host Unreachable
From 172.16.0.156 icmp_seq=980 Destination Host Unreachable
From 172.16.0.156 icmp_seq=981 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=993 ttl=128 time=0.586 ms
^C
--- 172.16.1.9 ping statistics ---
997 packets transmitted, 96 received, +15 errors, 90% packet loss, time 996823ms
rtt min/avg/max/mdev = 0.332/31.837/2001.563/226.238 ms, pipe 3
And here's a snippet from tcpdump that was running at the same time:
16:27:24.300566 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 990, length 64
0x0000: 4500 0054 571e 0000 8001 89c5 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 e573 de11 03de ecd8 f65b .......s.......[
0x0020: 0000 0000 9294 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:25.300102 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 991, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 5073 de11 03df edd8 f65b ......Ps.......[
0x0020: 0000 0000 1e94 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:25.300683 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 991, length 64
0x0000: 4500 0054 571f 0000 8001 89c4 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 5873 de11 03df edd8 f65b ......Xs.......[
0x0020: 0000 0000 1e94 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:26.300166 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 992, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 0a72 de11 03e0 eed8 f65b .......r.......[
0x0020: 0000 0000 6394 0400 0000 0000 1011 1213 ....c...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:26.300766 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 992, length 64
0x0000: 4500 0054 5720 0000 8001 89c3 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 1272 de11 03e0 eed8 f65b .......r.......[
0x0020: 0000 0000 6394 0400 0000 0000 1011 1213 ....c...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:27.300141 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 993, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 2071 de11 03e1 efd8 f65b .......q.......[
0x0020: 0000 0000 4c94 0400 0000 0000 1011 1213 ....L...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:27.300694 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 993, length 64
0x0000: 4500 0054 5721 0000 8001 89c2 ac10 0109 E..TW!..........
0x0010: ac10 009c 0000 2871 de11 03e1 efd8 f65b ......(q.......[
0x0020: 0000 0000 4c94 0400 0000 0000 1011 1213 ....L...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
^C
1034 packets captured
1036 packets received by filter
0 packets dropped by kernel
1034 packets captured
1036 packets received by filter
0 packets dropped by kernel
So hopefully from that you can see the echo replies in the tcpdump output with sequence numbers 992, 991 and 990 which are completely missed by ping.
If I swap over to the built in NIC, ping works as expected. This made me suspect my driver. However, if the driver was to blame would tcpdump also not see the icmp replies?
rhel drivers ping tcpdump
asked Nov 22 at 16:51
Adi
1614
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Redhat 2.6.32-754.el6.x86_64
I have an FPGA card acting as a NIC with an associated driver.
On my RH host, if I run ping through the FPGA NIC I see lots of missing responses. However, if I also run tcpdump and filter on icmp I can see the responses coming through.
Here's an example output from ping (note the many many missing sequence numbers):
from 172.16.1.9: icmp_seq=465 ttl=128 time=0.600 ms
64 bytes from 172.16.1.9: icmp_seq=467 ttl=128 time=0.490 ms
64 bytes from 172.16.1.9: icmp_seq=480 ttl=128 time=0.565 ms
64 bytes from 172.16.1.9: icmp_seq=482 ttl=128 time=0.590 ms
64 bytes from 172.16.1.9: icmp_seq=516 ttl=128 time=0.448 ms
64 bytes from 172.16.1.9: icmp_seq=526 ttl=128 time=0.649 ms
64 bytes from 172.16.1.9: icmp_seq=528 ttl=128 time=0.534 ms
64 bytes from 172.16.1.9: icmp_seq=539 ttl=128 time=0.424 ms
64 bytes from 172.16.1.9: icmp_seq=546 ttl=128 time=0.606 ms
64 bytes from 172.16.1.9: icmp_seq=562 ttl=128 time=0.521 ms
64 bytes from 172.16.1.9: icmp_seq=569 ttl=128 time=0.651 ms
64 bytes from 172.16.1.9: icmp_seq=591 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=617 ttl=128 time=0.652 ms
64 bytes from 172.16.1.9: icmp_seq=642 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=643 ttl=128 time=0.672 ms
64 bytes from 172.16.1.9: icmp_seq=644 ttl=128 time=0.443 ms
64 bytes from 172.16.1.9: icmp_seq=657 ttl=128 time=0.427 ms
64 bytes from 172.16.1.9: icmp_seq=668 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=704 ttl=128 time=0.332 ms
64 bytes from 172.16.1.9: icmp_seq=741 ttl=128 time=0.486 ms
64 bytes from 172.16.1.9: icmp_seq=742 ttl=128 time=0.478 ms
64 bytes from 172.16.1.9: icmp_seq=751 ttl=128 time=0.513 ms
64 bytes from 172.16.1.9: icmp_seq=753 ttl=128 time=0.511 ms
From 172.16.0.156 icmp_seq=788 Destination Host Unreachable
From 172.16.0.156 icmp_seq=789 Destination Host Unreachable
From 172.16.0.156 icmp_seq=790 Destination Host Unreachable
From 172.16.0.156 icmp_seq=792 Destination Host Unreachable
From 172.16.0.156 icmp_seq=793 Destination Host Unreachable
From 172.16.0.156 icmp_seq=794 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=798 ttl=128 time=0.671 ms
64 bytes from 172.16.1.9: icmp_seq=799 ttl=128 time=0.608 ms
64 bytes from 172.16.1.9: icmp_seq=801 ttl=128 time=0.538 ms
64 bytes from 172.16.1.9: icmp_seq=814 ttl=128 time=0.402 ms
64 bytes from 172.16.1.9: icmp_seq=923 ttl=128 time=0.458 ms
From 172.16.0.156 icmp_seq=952 Destination Host Unreachable
From 172.16.0.156 icmp_seq=953 Destination Host Unreachable
From 172.16.0.156 icmp_seq=954 Destination Host Unreachable
From 172.16.0.156 icmp_seq=956 Destination Host Unreachable
From 172.16.0.156 icmp_seq=957 Destination Host Unreachable
From 172.16.0.156 icmp_seq=958 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=966 ttl=128 time=0.472 ms
From 172.16.0.156 icmp_seq=979 Destination Host Unreachable
From 172.16.0.156 icmp_seq=980 Destination Host Unreachable
From 172.16.0.156 icmp_seq=981 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=993 ttl=128 time=0.586 ms
^C
--- 172.16.1.9 ping statistics ---
997 packets transmitted, 96 received, +15 errors, 90% packet loss, time 996823ms
rtt min/avg/max/mdev = 0.332/31.837/2001.563/226.238 ms, pipe 3
And here's a snippet from tcpdump that was running at the same time:
16:27:24.300566 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 990, length 64
0x0000: 4500 0054 571e 0000 8001 89c5 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 e573 de11 03de ecd8 f65b .......s.......[
0x0020: 0000 0000 9294 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:25.300102 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 991, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 5073 de11 03df edd8 f65b ......Ps.......[
0x0020: 0000 0000 1e94 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:25.300683 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 991, length 64
0x0000: 4500 0054 571f 0000 8001 89c4 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 5873 de11 03df edd8 f65b ......Xs.......[
0x0020: 0000 0000 1e94 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:26.300166 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 992, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 0a72 de11 03e0 eed8 f65b .......r.......[
0x0020: 0000 0000 6394 0400 0000 0000 1011 1213 ....c...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:26.300766 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 992, length 64
0x0000: 4500 0054 5720 0000 8001 89c3 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 1272 de11 03e0 eed8 f65b .......r.......[
0x0020: 0000 0000 6394 0400 0000 0000 1011 1213 ....c...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:27.300141 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 993, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 2071 de11 03e1 efd8 f65b .......q.......[
0x0020: 0000 0000 4c94 0400 0000 0000 1011 1213 ....L...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:27.300694 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 993, length 64
0x0000: 4500 0054 5721 0000 8001 89c2 ac10 0109 E..TW!..........
0x0010: ac10 009c 0000 2871 de11 03e1 efd8 f65b ......(q.......[
0x0020: 0000 0000 4c94 0400 0000 0000 1011 1213 ....L...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
^C
1034 packets captured
1036 packets received by filter
0 packets dropped by kernel
1034 packets captured
1036 packets received by filter
0 packets dropped by kernel
So hopefully from that you can see the echo replies in the tcpdump output with sequence numbers 992, 991 and 990 which are completely missed by ping.
If I swap over to the built in NIC, ping works as expected. This made me suspect my driver. However, if the driver was to blame would tcpdump also not see the icmp replies?
rhel drivers ping tcpdump
asked Nov 22 at 16:51
Adi
1614
Redhat 2.6.32-754.el6.x86_64
I have an FPGA card acting as a NIC with an associated driver.
On my RH host, if I run ping through the FPGA NIC I see lots of missing responses. However, if I also run tcpdump and filter on icmp I can see the responses coming through.
Here's an example output from ping (note the many many missing sequence numbers):
from 172.16.1.9: icmp_seq=465 ttl=128 time=0.600 ms
64 bytes from 172.16.1.9: icmp_seq=467 ttl=128 time=0.490 ms
64 bytes from 172.16.1.9: icmp_seq=480 ttl=128 time=0.565 ms
64 bytes from 172.16.1.9: icmp_seq=482 ttl=128 time=0.590 ms
64 bytes from 172.16.1.9: icmp_seq=516 ttl=128 time=0.448 ms
64 bytes from 172.16.1.9: icmp_seq=526 ttl=128 time=0.649 ms
64 bytes from 172.16.1.9: icmp_seq=528 ttl=128 time=0.534 ms
64 bytes from 172.16.1.9: icmp_seq=539 ttl=128 time=0.424 ms
64 bytes from 172.16.1.9: icmp_seq=546 ttl=128 time=0.606 ms
64 bytes from 172.16.1.9: icmp_seq=562 ttl=128 time=0.521 ms
64 bytes from 172.16.1.9: icmp_seq=569 ttl=128 time=0.651 ms
64 bytes from 172.16.1.9: icmp_seq=591 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=617 ttl=128 time=0.652 ms
64 bytes from 172.16.1.9: icmp_seq=642 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=643 ttl=128 time=0.672 ms
64 bytes from 172.16.1.9: icmp_seq=644 ttl=128 time=0.443 ms
64 bytes from 172.16.1.9: icmp_seq=657 ttl=128 time=0.427 ms
64 bytes from 172.16.1.9: icmp_seq=668 ttl=128 time=0.503 ms
64 bytes from 172.16.1.9: icmp_seq=704 ttl=128 time=0.332 ms
64 bytes from 172.16.1.9: icmp_seq=741 ttl=128 time=0.486 ms
64 bytes from 172.16.1.9: icmp_seq=742 ttl=128 time=0.478 ms
64 bytes from 172.16.1.9: icmp_seq=751 ttl=128 time=0.513 ms
64 bytes from 172.16.1.9: icmp_seq=753 ttl=128 time=0.511 ms
From 172.16.0.156 icmp_seq=788 Destination Host Unreachable
From 172.16.0.156 icmp_seq=789 Destination Host Unreachable
From 172.16.0.156 icmp_seq=790 Destination Host Unreachable
From 172.16.0.156 icmp_seq=792 Destination Host Unreachable
From 172.16.0.156 icmp_seq=793 Destination Host Unreachable
From 172.16.0.156 icmp_seq=794 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=798 ttl=128 time=0.671 ms
64 bytes from 172.16.1.9: icmp_seq=799 ttl=128 time=0.608 ms
64 bytes from 172.16.1.9: icmp_seq=801 ttl=128 time=0.538 ms
64 bytes from 172.16.1.9: icmp_seq=814 ttl=128 time=0.402 ms
64 bytes from 172.16.1.9: icmp_seq=923 ttl=128 time=0.458 ms
From 172.16.0.156 icmp_seq=952 Destination Host Unreachable
From 172.16.0.156 icmp_seq=953 Destination Host Unreachable
From 172.16.0.156 icmp_seq=954 Destination Host Unreachable
From 172.16.0.156 icmp_seq=956 Destination Host Unreachable
From 172.16.0.156 icmp_seq=957 Destination Host Unreachable
From 172.16.0.156 icmp_seq=958 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=966 ttl=128 time=0.472 ms
From 172.16.0.156 icmp_seq=979 Destination Host Unreachable
From 172.16.0.156 icmp_seq=980 Destination Host Unreachable
From 172.16.0.156 icmp_seq=981 Destination Host Unreachable
64 bytes from 172.16.1.9: icmp_seq=993 ttl=128 time=0.586 ms
^C
--- 172.16.1.9 ping statistics ---
997 packets transmitted, 96 received, +15 errors, 90% packet loss, time 996823ms
rtt min/avg/max/mdev = 0.332/31.837/2001.563/226.238 ms, pipe 3
And here's a snippet from tcpdump that was running at the same time:
16:27:24.300566 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 990, length 64
0x0000: 4500 0054 571e 0000 8001 89c5 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 e573 de11 03de ecd8 f65b .......s.......[
0x0020: 0000 0000 9294 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:25.300102 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 991, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 5073 de11 03df edd8 f65b ......Ps.......[
0x0020: 0000 0000 1e94 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:25.300683 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 991, length 64
0x0000: 4500 0054 571f 0000 8001 89c4 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 5873 de11 03df edd8 f65b ......Xs.......[
0x0020: 0000 0000 1e94 0400 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:26.300166 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 992, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 0a72 de11 03e0 eed8 f65b .......r.......[
0x0020: 0000 0000 6394 0400 0000 0000 1011 1213 ....c...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:26.300766 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 992, length 64
0x0000: 4500 0054 5720 0000 8001 89c3 ac10 0109 E..TW...........
0x0010: ac10 009c 0000 1272 de11 03e0 eed8 f65b .......r.......[
0x0020: 0000 0000 6394 0400 0000 0000 1011 1213 ....c...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
16:27:27.300141 IP 172.16.0.156 > 172.16.1.9: ICMP echo request, id 56849, seq 993, length 64
0x0000: 4500 0054 0000 4000 4001 e0e3 ac10 009c E..T..@.@.......
0x0010: ac10 0109 0800 2071 de11 03e1 efd8 f65b .......q.......[
0x0020: 0000 0000 4c94 0400 0000 0000 1011 1213 ....L...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
16:27:27.300694 IP 172.16.1.9 > 172.16.0.156: ICMP echo reply, id 56849, seq 993, length 64
0x0000: 4500 0054 5721 0000 8001 89c2 ac10 0109 E..TW!..........
0x0010: ac10 009c 0000 2871 de11 03e1 efd8 f65b ......(q.......[
0x0020: 0000 0000 4c94 0400 0000 0000 1011 1213 ....L...........
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 0000 4567..
^C
1034 packets captured
1036 packets received by filter
0 packets dropped by kernel
1034 packets captured
1036 packets received by filter
0 packets dropped by kernel
So hopefully from that you can see the echo replies in the tcpdump output with sequence numbers 992, 991 and 990 which are completely missed by ping.
If I swap over to the built in NIC, ping works as expected. This made me suspect my driver. However, if the driver was to blame would tcpdump also not see the icmp replies?
rhel drivers ping tcpdump
rhel drivers ping tcpdump
asked Nov 22 at 16:51
Adi
1614
asked Nov 22 at 16:51
Adi
1614
asked Nov 22 at 16:51
Adi
1614
asked Nov 22 at 16:51
Adi
1614
asked Nov 22 at 16:51
Adi
1614
1614
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
It was my driver! Got confused with NET_IP_ALIGN offset related stuff and the destination MAC address of the ICMP response was having the first two bytes mangled by the driver. Obviously tcpdump doesn't care about the MAC address (promiscuous mode) but ping does.
Interestingly, ping didn't see the replies if the first two bytes of the destination MAC address were 0x00 but it appeared that if they were anything else then they would come through! So they didn't appear to have to be the correct values, just not 0x00. Again this may be an issue with my driver.
answered 2 days ago
Adi
1614
Just to add, adding -e to the tcpdump command then displayed the MAC address and made the issue obvious.
– Adi
2 days ago
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
It was my driver! Got confused with NET_IP_ALIGN offset related stuff and the destination MAC address of the ICMP response was having the first two bytes mangled by the driver. Obviously tcpdump doesn't care about the MAC address (promiscuous mode) but ping does.
Interestingly, ping didn't see the replies if the first two bytes of the destination MAC address were 0x00 but it appeared that if they were anything else then they would come through! So they didn't appear to have to be the correct values, just not 0x00. Again this may be an issue with my driver.
answered 2 days ago
Adi
1614
Just to add, adding -e to the tcpdump command then displayed the MAC address and made the issue obvious.
– Adi
2 days ago
add a comment |
up vote
0
down vote
It was my driver! Got confused with NET_IP_ALIGN offset related stuff and the destination MAC address of the ICMP response was having the first two bytes mangled by the driver. Obviously tcpdump doesn't care about the MAC address (promiscuous mode) but ping does.
Interestingly, ping didn't see the replies if the first two bytes of the destination MAC address were 0x00 but it appeared that if they were anything else then they would come through! So they didn't appear to have to be the correct values, just not 0x00. Again this may be an issue with my driver.
answered 2 days ago
Adi
1614
Just to add, adding -e to the tcpdump command then displayed the MAC address and made the issue obvious.
– Adi
2 days ago
add a comment |
up vote
0
down vote
up vote
0
down vote
It was my driver! Got confused with NET_IP_ALIGN offset related stuff and the destination MAC address of the ICMP response was having the first two bytes mangled by the driver. Obviously tcpdump doesn't care about the MAC address (promiscuous mode) but ping does.
Interestingly, ping didn't see the replies if the first two bytes of the destination MAC address were 0x00 but it appeared that if they were anything else then they would come through! So they didn't appear to have to be the correct values, just not 0x00. Again this may be an issue with my driver.
answered 2 days ago
Adi
1614
It was my driver! Got confused with NET_IP_ALIGN offset related stuff and the destination MAC address of the ICMP response was having the first two bytes mangled by the driver. Obviously tcpdump doesn't care about the MAC address (promiscuous mode) but ping does.
Interestingly, ping didn't see the replies if the first two bytes of the destination MAC address were 0x00 but it appeared that if they were anything else then they would come through! So they didn't appear to have to be the correct values, just not 0x00. Again this may be an issue with my driver.
answered 2 days ago
Adi
1614
answered 2 days ago
Adi
1614
answered 2 days ago
Adi
1614
answered 2 days ago
Adi
1614
1614
Just to add, adding -e to the tcpdump command then displayed the MAC address and made the issue obvious.
– Adi
2 days ago
add a comment |
Just to add, adding -e to the tcpdump command then displayed the MAC address and made the issue obvious.
– Adi
2 days ago
Just to add, adding -e to the tcpdump command then displayed the MAC address and made the issue obvious.
– Adi
2 days ago
Just to add, adding -e to the tcpdump command then displayed the MAC address and made the issue obvious.
– Adi
2 days ago
add a comment |
draft saved
draft discarded
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f483491%2fping-responses-missing-but-can-see-them-in-tcpdump%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
