recent network related changes

["Sepherosa Ziehau" <sepherosa@xxxxxxxxx>]

Hi,

[Only applies to MP safe tests]

The effect of my recent changes to interface<->ethernet is at:
http://leaf.dragonflybsd.org/~sephe/fwd/comp00_0123.png
http://leaf.dragonflybsd.org/~sephe/fwd/comp01_0123.png

I think the above two cases is the most common use cases, and the
current throughput almost doubles the old one.

The ETHER_CHAIN_INPUT effect is not obvious in above two cases, but
its effect is quite obvious, when if_start does not contend:
http://leaf.dragonflybsd.org/~sephe/fwd/comp00_1.png
http://leaf.dragonflybsd.org/~sephe/fwd/comp01_2.png

The results of various tests mentioned in:
http://leaf.dragonflybsd.org/~sephe/fwd/readme.txt
are shown at:
http://leaf.dragonflybsd.org/~sephe/fwd/data00.png
http://leaf.dragonflybsd.org/~sephe/fwd/data01.png

I also compared fastforwarding and non-fastforwarding on HEAD:
http://leaf.dragonflybsd.org/~sephe/fwd/data_ff_nff.png
NOTE: There IS performance regression introduced in fastforwarding
when reducing if_start serializer contention:
http://leaf.dragonflybsd.org/~sephe/fwd/compff.png
But even with the best fastforwarding result (max at 520Kpps), it is
still _quite_ slower than non-fastforwarding.

The comparison of two major changes under various tests are at:
http://leaf.dragonflybsd.org/~sephe/fwd/comp0[01]_xxx.png

There are two regression under specific tests:
1) em0 on cpu0, em1 on cpu1, stream target cpu1:
http://leaf.dragonflybsd.org/~sephe/fwd/comp01_1.png
2) em0 on cpu0, em1 on cpu1, stream target cpu0:
http://leaf.dragonflybsd.org/~sephe/fwd/comp00_0.png

I have studied 1) with ktr:
in udp_thread, each packet enqueueing triggers if_start, and if_start
interlock checking happens for each packets.
I have experimented one solution, which utilizes our lwkt scheduling feature:
schedule if_start on the same CPU but in different thread.
This solution is not as good as the current one in most cases,
especially when packets are spread evenly on each CPUs.

No matter what, I think I currently have a relative good ground to
keep moving on :)

Best Regards,
sephe

-- 
Live Free or Die