From: Caleb Clausen <vikkous@...>
Date: 2008-09-12T08:41:41+09:00
Subject: Re: How to use this Combinatorial testing technique

On 9/10/08, Surya <surya5kn@gmail.com> wrote:
> Hi Friends,
>
> I have been in to testing for 3 yrs. In my project there are 6000 test
> cases present and some of them ought to be non-value adding to the
> testing effort. Around 80% of the total test cases were automated and
> would like to run execersie to reduce the un important test cases from
> the pool. Can any one suggest how to go about this and apply
> Combinatorial/pairwise testing effectively to my situation.

I suffer from an excess of test cases as well, but I don't have a
ready solution.
I have heard before of tools to help identify redundant test cases,
but none for
use with ruby.

I imagine such a tool would work like this:
1) run each test case by itself within a coverage analyzer, to get a
list of lines executed.
2) now work thru the list of lists of lines covered per test case,
eliminating test cases where each line covered by that case is covered
also in some other case. (A number of algorithms are possible here.)

Writing a tool like that wouldn't be too hard. Getting it to be
efficient might be challenging...

Of course there's a problem with this; line coverage is less than
perfect.  So if there's a
line whose first half is tested by one test case, and the second half
tested by another
test case, the second case may be flagged as redundant when it isn't.
Expression
coverage is the gold standard when it comes to code coverage. So
ideally, you'd write the
tool above to work expression by expression rather than line by line.
There are no expression-based coverage tools for ruby that I know of.

Even expression coverage will only give you complete code coverage;
data coverage is
another matter. Is there a tool or metric for assessing data coverage?
I have no idea.