From: Bil Kleb <W.L.Kleb@...>
Date: 2004-03-02T01:43:06+09:00
Subject: Re: Test::Unit: metadata

David Garamond wrote:
>
> We'd like to put several metadata into the test collection to get a 
> better picture of the state of the tests.

Personally, I am usually only interested in the number of tests failing.

> 1. total number of test methods, cases, and suites.

I guess function points would be somewhere between cases and suites?

> 2. LOC per each test case and test suite, and total LOC dedicated to 
> testing (this will eventually be compared to total LOC of code, LOC of 
> comments, LOC of documentation, etc); aside from LOC, number of 
> chars/bytes will also be used as another unit of measure.

AFAIK, LOC is out as a worthful measure---see DeMarco's Slack and
Poppendiecks' Lean Software Development.

> 3. number (and, as % of LOC) of tests that are interactive (i.e. GUI 
> tests).

Sounds like challenge for Phlip (to reduce the number of interactive GUI tests).

> And there will also be lots of measurements to be put into the database 
> about the results after each run, e.g.: number of assertions, number of 
> test methods/cases that succeed/failed, actual run time of tests, etc.

Have you seen BulkTestRunner class that is part of Rubicon?  It produces a
console summary like the following:

========================================================================
Mobility Test Summary         Test Results
========================================================================
                  Name   OK?   Tests  Asserts      Failures   Errors
             ------------------------------------------------------------
             Array2DUT             3        6
     BoundaryProfileUT             7       31
       CartesianFaceUT             4       16
                CaseUT             3        7
      CellCollectionUT             8       47
                CellUT             3        9
         CmdLineOptsUT             4        6
                FaceUT             5       13
           GridLevelUT             7       20
                GridUT             3        4
HyperbolicFunctionsUT             3        9
        InteriorCellUT   FAIL     13       32          1
             LimiterUT             4       10
             LineFitUT             1        2
             PhysicsUT             3        8
              ScalarUT             3        3
              VectorUT             6        7
========================================================================
          All 17 files   FAIL     80      230          1        0
========================================================================

                              Failure Report
========================================================================

InteriorCellUT:
---------------
     ./InteriorCellUT.rb:49:in `testParameterLookupByReynoldsNumber'
     ....<1> expected but was
<2>.

========================================================================

If you search blade, you should be able to find my stand-alone version of it.

Regards,
-- 
Bil Kleb, NASA, Hampton, Virginia, USA