From: Mauricio Fernandez <mfp@...>
Date: 2006-08-05T05:53:59+09:00
Subject: Re: Automatically finding gaps in unit test coverage ?

On Sat, Aug 05, 2006 at 03:50:35AM +0900, Tomasz Wegrzanowski wrote:
> On 8/4/06, Robert Klemme <shortcutter@googlemail.com> wrote:
> >I don't know of any mechanism built into RubyUnit or a code coverage
> >tool.  But this is something you could do: use set_trace_func to trace
> >all methods invoked, then at the end of a single class's test case or
> >when all tests are done go through your classes and check their
> >#instance_methods to verify that all of them were called.  But that
> >still does not give you info whether you had a proper test case for the
> >method or the method was called by some other method etc.  Maybe you can
> >extend the tracing to record only methods called from a unit test.
[...]

You can do that in rcov with the  --test-unit-only option.

> Well, one cannot have everything :-). rcov is pretty useful for finding
> missing tests. My only problem with it so far is its inability to test
> coverage of metaprogrammed code, but I guess that can't be helped.

Are you referring to this?

$ rcov --no-html --text-coverage --no-color a.rb 
================================================================================
a.rb
================================================================================
   
   module StupidMeta
     def ugly_make_meth(meth)
       module_eval <<-EOF, __FILE__, __LINE__+1
         def #{meth}(a,b)
           a + #{meth.to_s.inspect} + b
         end 
       EOF
     end
   
     def make_meth(meth)
       define_method(meth) do |a,b|
!!       a + meth.to_s + b
!!     end
     end
   
   end
   
   class X
     extend StupidMeta
     ugly_make_meth :foo
     make_meth :bar
   end
   
   # we don't actually call them, so they should both be uncovered
   #x = X.new
   #x.foo("method ", " here")
   #x.bar("method ", " here")


As you can see, Module#define_method will work better, if you can use it at
all (that is, if you don't need a block argument, under 1.8, and don't care
about the closure and the speed hit). Another advantage is that it will tell
you if the code is well formed (syntactically correct) much earlier.

The problem with the string forms of eval is that I've had to go to great
lengths to make sure rcov marks heredocs correctly, and the above annotation
*is* correct.

Indeed, in the above snippet the heredoc in ugly_make_meth has been
"executed", in some sense (this interpretation gains some weight due to the
interpolation going on; there's actually something being evaluated there).

But there are two interpretations for that string: as a mere heredoc, and as
executable code that defines another method. There are thus also two ways to
think about it as far as coverage is concerned.

I'm thinking that maybe some sort of pragma would make sense:

  def ugly_make_meth(meth)
    module_eval <<-EOF, __FILE__, __LINE+1__ # rcov:noheredoc
      def #{meth}(a,b)
        a + #{meth.to_s.inspect} + b
      end
    EOF
  end


As for other limitations, I think I know how to handle generated code that
cannot be meaningfully associated to your sources, but I feel it's not worth
the effort.

Is there anything else you'd like to see addressed?

-- 
Mauricio Fernandez  -   http://eigenclass.org   -  singular Ruby