DragonFly users List (threaded) for 2008-03
Re: FreeBSD 7, DragonFly's status
Dave Hayes wrote:
Does an objective metric of stability actually exist? ( If you say
"uptime" I'll take that as a "no" ;) ) If it does, I would really like
to learn what that metric is. Do you know of any current
low-project-bias work that has been done in this area?
Thanks in advance. :)
It's easiest to define "stability" by "lack of instability", i.e.
"system does not crash no matter what you do to it".
The best method I know to evaluate this is by brute force (other
techniques like static code analysis and formal model checking can
help). You have to try really hard to put the system through all kinds
of bizarre contortions in the workloads you care about (which is
"everything" for a general OS developer) until you find something that
breaks. Then fix it and try again.
You have to put serious effort into it though, because after you fix all
the "obvious" panics that can be reproduced in a few minutes of testing,
you end up trying to cause extremely low probability events that can
(and do) nevertheless pop up on real systems given the right combination
of circumstances. If you don't put in the work, the bugs will usually
not get fixed until they crash a user's system and they bother to report
the bug. This doesn't always happen, often they just curse you out.
Once you can no longer trigger bugs no matter how hard you try (assuming
you are achieving good coverage of the system), I think it's reasonable
to provisionally award the label of "pretty stable" to the aspects of
the system you have been testing. There will always be more bugs than
those you found (especially with particular hardware configurations),
but at least you've made a concerted effort to find them.
This is basically what stress2 and other tools try to help automate,
although it can never replace human-driven QA. It's literally a full
time job to do properly.