From: Robert Oliver <oliverb@...>
Date: 2001-12-11T05:00:14+09:00
Subject: [ruby-talk:28132] Re: John Roth dolt (  Re: A challenge to proponents of Unit Testing. )


> Kent Beck wrote:
>
> > What defects am I missing with the above 5 tests? Put another way, what
> > further tests could improve the MTBF of my function?

"Joe Pallas (pallas at cs dot stanford dot edu)"
<pallas@cs_dot_stanford.edu> wrote in message
news:3C14FD90.9F8987B3@cs_dot_stanford.edu...
> What a bizarre notion.  It is not meaningful to talk about the MTBF of
> software, because software does not fail.  Software is either correct or
> incorrect, and that does not change over time.  Requirements change, and
> execution environments change, but the program itself is not subject to
> "bit rot."

Joe,

I'd just like to discuss the idea of (Mean Time Between Failure) MTBF for
software.

Firstly, software fails,  it just fails differently than hardware.  A few
examples:

A function is tested and meets all functional requirements, but it
accumulates a count internally which overflows about once per week in normal
use.  The function is always broken, but since it only fails once per week,
it has an MTBF of 1 week.

Another function was tested but one obscure test case was overlooked.  The
data with which this function is used contains the obscure combination which
shows the failure about once per month.  The MTBF of the software is about 1
month. It could be different with different data.  It's still broken.

Yet another function has overlooked a test case.  This time, the developers
have not used the function in a way to cause it to fail.  Yes, it's still
broken, and someday we will probably find out how.  It's MTBF is (for now) a
long time.

Secondly, MTBF is a useful concept that relates to observed behaviour.
Which is more reliable, Windows or Linux?  Many of us have an opinion to
this question and it is almost always defined in terms of how long the
system will run before it crashes, in other words, its MTBF.

Thirdly, MTBF can be used to weed out bad systems.  We may not be able to
prove a good given system has a high MTBF, but we can almost surely show a
bad system has a poor MTBF.  In all likelihood, both systems have some bugs.

Regards,

Bob Oliver