[ruby-core:61246] [ruby-trunk - Feature #9508] Add method coverage and branch coverage metrics

Issue #9508 has been updated by Sam Rawlins.


Hi Eric,

I'd actually like to keep the format of Coverage.result as the new format (Coverage.result values are each a Hash with :lines, :methods, and :decisions keys), rather than the existing Ruby 2.1.0 format, for two reasons:

1. Currently, the call to Coverage.result is very destructive: it immediately freezes the Coverage results, and clears the line coverage arrays. I don't think we can cleanly retrieve the line coverage, then maybe later the method coverage, always leaving the collective coverage results in an indeterminate "some cleared and some not cleared" state... and not really remember (or even let be discoverable) which results have been wiped and which haven't. [1]

2a. The very top of the Coverage documentation reads "Coverage provides coverage measurement feature for Ruby. This feature is experimental, so these APIs may be changed in future." so tools should be ready for a change.

2b. This change should be very easy to sniff out. For example, in SimpleCov, only one small change is needed in simple_cov/result.rb [1] in order to remain compatible with old Coverage and new:

       def initialize(original_result)
         @original_result = original_result.freeze
         @files = SimpleCov::FileList.new(original_result.map do |filename, coverage|
    -      SimpleCov::SourceFile.new(filename, coverage) if File.file?(filename)
    +      if coverage.is_a? Array  # Ruby < 2.2.0
    +        SimpleCov::SourceFile.new(filename, coverage) if File.file?(filename)
    +      else                     # Ruby >= 2.2.0
    +        SimpleCov::SourceFile.new(filename, coverage[:lines]) if File.file?(filename)
    +      end
         end.compact.sort_by(&:filename))
         filter!
       end

Of course the decision isn't up to me, but multiple methods like Coverage.result2 or Coverage.result(metric = :lines) or something feels bad... maybe there is another solution.

I'll amend my code with fewer rb_hash_lookups. My code there is super ugly... your suggestion will be much better. And #9425 is looking very promising!

[1] footnote: I'm not a huge fan of the Coverage.result methodology... but I think it was written this way so that the returned Coverage hash of results is now eligible for garbage collection. I'd rather a method called maybe Coverage.end that ends coverage (and neatly pairs with Coverage.start) and returns results that it does today. AND that those results can again be retrieved later, at will, by calling Coverage.result. Those results would be valid and unchanged until Coverage.start is called again, at which point the results are cleared (rather than when Coverage.end is called). But this would be for another ticket, if its even worth opening soon.
[2] https://github.com/colszowka/simplecov/blob/master/lib/simplecov/result.rb#L26

----------------------------------------
Feature #9508: Add method coverage and branch coverage metrics
https://bugs.ruby-lang.org/issues/9508#change-45588

* Author: Sam Rawlins
* Status: Open
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
----------------------------------------
Since the Coverage extension was introduced in Ruby 1.9, Ruby has had built-in line code coverage. Ruby should support more of the basic code coverage metrics [1]. I have a pull request on GitHub ( https://github.com/ruby/ruby/pull/511 ) to add Method Coverage (Function Coverage) and Branch Coverage. I'd love feedback to improve it.

Currently, with this feature, Coverage.result would look like:

    {"/Users/sam/code/ruby/cov_method.rb" => {
      lines: [1, 2, 2, 20, nil, nil, 2, 2, 2, nil, 0, nil, nil, nil, 1, 0, nil, nil, 1, 1, nil, nil, 1],
      methods: {1=>2, 15=>0, 19=>1},
      branches: {8=>2, 11=>0}
    }}

which includes
* the current Ruby line coverage report,
* as well as a method report (The method defined on line 1 was called 2 times; the method on line 15 was called 0 times; ...),
* and a branch report (the branch on line 8 was called 2 times; the branch on line 11 was called 0 times).

Branches
--------

Branches include the bodies of if, elsif, else, unless, and when statements, which are all tracked with this new feature. However, this feature is not aware of void bodies, for example:

    if foo
      :ok
    end

will report that only one branch exists in the file. It would be better to declare that there is a branch body on line 2, and a void branch body on line 3, or perhaps line 1. This would require the keys of the [:branch] Hash to be something other than line numbers. Perhaps label_no? Perhaps nd_type(node) paired with line or label_no?

More Coverage
-------------

I think that Statement Coverage, and Condition Coverage could be added to this feature, using the same techniques.

Caveats
-------

I was not very clear on the bit-arrays used in ruby.h, and just used values for the new macros that seemed to work.

Also, I would much rather use Ranges to identify a branch, so that a Coverage analyzer like SimpleCov won't need any kind of Ruby parser to identify and highlight a full chunk of code as a tested branch, or a not tested branch. I'm trying to find how that could be implemented...

[1] Wikipedia has good definitions: http://en.wikipedia.org/wiki/Code_coverage

---Files--------------------------------
pull-request-511.patch (26.7 KB)
pull-request-511.patch (38.5 KB)
pull-request-511.patch (57 KB)


-- 
http://bugs.ruby-lang.org/