Re: DragonFlyBSD Thread on osnews

["Dmitri Nikulin" <dnikulin@xxxxxxxxx>]

On 2/3/07, Petr Janda <elekktretterr@exemail.com.au> wrote:
>  From what I gathered across this mailing list. DF can't scale nowhere
> close to Linux at the moment because DF still operates under the BGL.

And we won't know how well it scales until it supports the whole
architecture necessary for those 4096-CPU systems, then find somebody
who's willing and able to administer and fairly benchmark the system.

It's not even enough to optimize the scaling. Lock contention is also
a factor of the raw performance - more attempts at locking mean a
higher rate of lock contention, but then if those locks run for less
time there's less chance of an actual block, and indeed less harm when
the block occurs. You still get the necessary instructions (and bus
lock cycles, the real killers) to do the locking, but the emergent
effect is different. Really short critical paths can use spinlocks
sanely, and it's not clear what can be locked by a spinlock until it's
optimized enough. Even DragonFly's parallel architecture requires a
sub-architecture of fine grained locking.

What I'm saying is, even if the BGL disappeared overnight, there'd be
a lot more optimizing and profiling left to make DragonFly bench
against Linux without humiliation. And Linux has years of head start
with a much larger developer base, including countless corporate and
governmental interests, plus very good technologies only it can use
legally because of the patent system which pretty much guarantee its
victory.

The best DragonFly can do is be so much better at SSI clustering that
Linux' high-end SMP performance is considered an expensive
alternative, rather than actually viable. That means also beating
Linux' years of clustering development.

It's probably not too far off that if a corporation decided Linux'
clustering needed improving in a way similar to DragonFly's design, it
would be hacked in over a few months and would perform faster than
DragonFly would for years just because of the foundation. It doesn't
matter to them if it's maintainable or beautiful code, because on
average for these projects the extra maintenance work is less than the
work to make the code easy to maintain. And Linux' components, from
the VM to the ATA to firewire to schedulers, are so often re-developed
that they often have multiple alternatives for each system in the tree
at any one time, with easy compile time switching. Even if nobody
wants to maintain a crappy part of the system, somebody will just
write an alternative and it's generally better.

This haphazard and decidedly chaotic strategy maps so well to actual
real-world development that it is the obvious underpinning of Linux'
success. Any look at the code will make it abundantly clear that it's
not software quality that gets it places, but being so welcoming to
actual development by basement hackers and corporations alike. And
that's something I'm not confident any other project can repeat now.
Call me pessimistic... but I'd rather be optimistic about Linux'
quality improving, than hoping for another project to replace a
significant share of it.

---
Dmitri Nikulin

Centre for Synchrotron Science
Monash University
Victoria 3800, Australia