From: Frank Mitchell <fmitchell@...>
Date: 2004-04-16T07:49:10+09:00
Subject: Re: test / unit question:  add "Warnings" to Failures andErrors

Bret Pettichord wrote:
> In fact i even think the distinction between failures and errors is 
> tedious and unnecessary.

It depends, if it allows you to distinguish broken test code from broken 
production code.  For example:

- Test code violating preconditions (in the Design by Contract sense) 
should be "errors", while violated postconditions or invariants should 
be "failures".  For example, a method is only valid when its receiver is 
in a particular state; if the test calls that method without verifying 
or ensuring the receiver's state, the test is at fault.  (Of course, 
sometimes it's a regression bug, and designers have to decide whether to 
patch the test or redesign to remove the dependency.)

- Failures in the fixture setup, particularly reading test input files 
or setting up Mock Objects, should be "errors", while failures during 
the execution of production code should be "failures".

- Exceptions caused by external flaws, like an IOException while loading 
test data or the unanticipated death of a remote server, should be 
"errors".  Of course, the proper design of a unit test would inline test 
data and mock out servers, but I've seen a number of "real-world" tests 
that don't; some test writers find the task too hard given the current 
design, and others are trying to prove that a third-party API works as 
expected underneath their own wrapper.

I agree that current tools don't make the distinction very well, and 
maybe it's even theoretically impossible in the general case.