Re: Severe packet loss on fxp interfaces with the new bridging code...

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>]

:On an SMP machine without any of the new mpsafe sysctls active, bridging
:over fxp interfaces barely works.  Maximum bandwidth is somewhere around
:100K/s, and I am seeing up to 80% packet loss.  (as measured with a
:simultaneous scp and ping)  After a bit of detective work, I have found
:the location of these mysteriously vanishing packets, but I do not
:understand the true nature of the problem.

    Ok, I've run some tests.  I haven't been able to reproduce any 
    packet loss testing between nv0 and em0.  I have one of my shuttle
    SMP boxes (test28) set up to bridge to test29:

    (p.s. if you want to submit a patch to fix the output packet count
    too that would be great!)

test28# netstat -I bridge0 -in 1
            input      (bridge0)           output
   packets  errs      bytes    packets  errs      bytes colls
     29392     0    2880416          0     0          0     0
     27542     0    2699116          0     0          0     0
     25886     0    2536828          0     0          0     0

office1# ping -f test29
PING test29.backplane.com (216.240.41.29): 56 data bytes
. .......^.
--- test29.backplane.com ping statistics ---
110597 packets transmitted, 110589 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.080/0.254/10.473/0.074 ms

    On your ping test, make sure that you have set 'net.inet.icmp.icmplim'
    to some high number on the target box so it doesn't throw away the
    ping packets.

    Now, of course, neither of these is using the fxp driver.  But I 
    haven't found anything wrong in particular with the fxp driver so
    also check your mbuf clusters:

    test28# sysctl -a | fgrep nmb
    kern.ipc.nmbclusters: 6656
    kern.ipc.nmbufs: 13312
    test28# 

    Maybe your kern.ipc.nmbclusters is too low.  If so, set it in
    /boot/loader.conf (it has to be set at boot-time, not afterwords).

    If that turns out to be the problem the only thing I'll have to do
    is nudge the defaults.

					-Matt