Re: Interesting ubench scores for FreeBSD 4.11, 5.4, 6.0beta3 and DFly-Preview

[Kris Kennaway <kkenn@xxxxxxxxxxxxxxxxxx>]

On 2005-09-03, Toma¾ Bor¹tnar <tomaz.borstnar@xxxxxxxx> wrote:
> Kris Kennaway wrote:
>>>Summary
>>>Latest DFly-preview has best Ubench AVG of 106030, because of 2nd best memory score (124353) and balanced CPU score 
>>>(87707). Also good was FreeBSD 6beta3/amd64 with Ubench AVG of   101448, because of balanced CPU (102455) and memory 
>>>score (100441). Seems like extra registers in long mode help quite a bit. Third was FreeBSD 5.4/i386 with Ubench AVG of 
>>>    94304 - mostly because of best memory score (130602) and not so good CPU score (58006). Hardly behind was FreeBSD 4.11
>>>with Ubench AVG of 94296 - decent, but slower CPU score (84431) and very nice memory score (104161). FreeBSD 6beta3/i386
>>>was severly behind all of them with Ubench AVG of 55968 - with cpu of 59138 and memory of 52799.
>>>
>>>I wish DFly had better CPU score :) But it still has best score among 32bit OS systems!
>>>
>>>As usual, YMMV :)
>> 
>> 
>> Did you remember to disable the debugging features in FreeBSD 6
>> (WITNESS, INVARIANTS, malloc debugging)?  If not, you're incurring a
>> significant (usually >30%) performance penalty, which is consistent
>> with your numbers.
>
> I checked FreeBSD 6beta3/i386 again - i forgot to reboot with new
> kernel which I did this time. The only visible change is much better
> memory score - 98040 this time(!), while CPU was 59062. Almost 100%
> more from 52799 with GENERIC debug kernel! But average score is
> still low compared to others - 78551.

I did some tests on a UP pentium 3 machine, and confirmed that most of
the difference between FreeBSD 4.x and later versions is due to the
different compiler being used to compile the code (gcc 2.95 vs 3.4).
gcc 3.x is known to have wildly different performance characteristics
than 2.x (better on some code, worse on other).  When I retested 5.x
and above with a FreeBSD 4.x binary (statically linked), I found
somewhat different results.

I ran at least 10 tests on each platform and then used the ministat
tool (/usr/src/tools/ministat on freebsd) to perform a statistical
comparison.

When using the same binary, the CPU scores are statistically
indistinguishable between the different FreeBSD versions.  This makes
sense since there's little kernel involvment in running userland
integer/FP computations.  When running the gcc 2.95 binary all
versions of FreeBSD were 31% *faster* on this test than when running a
gcc 3 binary (both compiled with -O only).

FreeBSD 5.x and above show a 6.3% drop on the memory test relative to
4.x (with the same 4.x binary).  I reran ubench with kernel profiling
enabled and found that this drop is mostly due to the vm locking
present in FreeBSD 5 and above (via vm_fault).  This locking is also
responsible for the dramatic performance increases on SMP machines
seen in other benchmarks, so it would be more interesting to test on
SMP machines.  I'm not set up to do this on my hardware though.

The memory test showed about a 14.6% benefit from using the gcc 3
binary vs gcc 2.95.  This is presumably due to better code generation
in e.g. memset/memcpy.

In summary: the CPU test is a priori not very useful for comparing
performance of different OSes, because it is largely a test of the
code generated by the compiler (unless you can eliminate this
variable).  The lesson is that if your application includes a lot of
CPU intensive code, you should carefully benchmark its performance
with different compilers and optimizations to see what works best.
Thus, the 'average' number is also not meaningful as a point of
comparison since it is tainted by this fact.

The memory test is a bit more meaningful since it contains a kernel
component, but it's still influenced quite a lot by the compiler, so
it's also hard to compare directly unless you can eliminate that
variable.

Kris