From: hunter_spawn@...
Date: 2017-11-04T01:09:34+00:00
Subject: [ruby-core:83666] [Ruby trunk Feature#13901] Add branch coverage

Issue #13901 has been updated by MaxLap (Maxime Lapointe).


marcandre (Marc-Andre Lafortune) wrote:
> mame (Yusuke Endoh) wrote:
> > I see...  It is a very interesting approach, but it seems to completely ignore asynchronous interrupt, such as signal and `Thread#raise`.  I wonder if it is not a good design decision for the embedded coverage measurement library.  I'll think about it.  Thank you for teaching me, anyway.
> 
> Indeed, it ignores that completely. I can not even remotely imagine a case where it could possibly matter.

To add a little bit to the discussion: The only information that you lose when an asynchronous interrupt like Thread.raise happens are the details on which ones of multiple sequential things that can never raise were done or not (if the interrupt happened during the execution of the sequence). By that I mean things such as:

~~~ ruby
my_var = 1
['here is', 'a big' , 'array', ['of', 6, 'literals']]
other_var = my_var
a_hash = {1: 3}
~~~

It doesn't matter, in the sense of Coverage, weither the assignment of one variable to the other happened or not, or that the array creation was completed or not. These are things without size effects for the program that literally cannot fail (aside from asynchronous interrupt and OutOfMemory).

As soon as something "interesting" can happen, we have trackers to know if the thing was interrupted or not. By interesting, I mean literally anything that can raise an exception or change control flow: send, define a method, if, case, loops, constants, rescue, ensure, etc...

----------------------------------------
Feature #13901: Add branch coverage 
https://bugs.ruby-lang.org/issues/13901#change-67692

* Author: mame (Yusuke Endoh)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
I plan to add "branch coverage" (and "method coverage") as new target types of coverage.so, the coverage measurement library.  I'd like to introduce this feature for Ruby 2.5.0.  Let me to hear your opinions.

## Basic Usage of the Coverage API

The sequence is the same as the current: (1) require "coverage.so", (2) start coverage measurement by `Coverage.start`, (3) load a program being measured (typically, a test runner program), and (4) get the result by `Coverage.result`.

When you pass to `Coverage.start` with keyword argument "`branches: true`", branch coverage measurement is enabled.

test.rb

~~~
require "coverage"
Coverage.start(lines: true, branches: true)
load "target.rb"
p Coverage.result
~~~

target.rb

~~~
1: if 1 == 0
2:   p :match
3: else
4:   p :not_match
5: end
~~~

By measuring coverage of target.rb, the result will be output (manually formatted):

~~~
$ ruby test.rb
:not_match
{".../target.rb" => {
  :lines => [1, 0, nil, 1, nil],
  :branches => {
    [:if, 0, 1] => { [:then, 1, 2] => 0, [:else, 2, 4] => 1 }
  }
}
~~~

`[:if, 0, 1]` reads "if branch at Line 1", and `[:then, 1, 2]` reads "then clause at Line 2".  So, `[:if,0,1] => { [:then,1,2]=>0, [:else,2,4]=>0 }` reads "the branch from Line 1 to Line 2 has never executed, and the branch from Line 1 to Line 4 has executed once."

The second number (`0` of `[:if, 0, 1]`) is a unique ID to avoid conflict, just in case where multiple branches are written in one line.  This format of a key is discussed in "Key format" section.

## Why needed

Traditional coverage (line coverage) misses a branch in one line.  Branch coverage is useful to find such untested code.  See the following example.

target.rb

~~~
p(:foo) unless 1 == 0
p(1 == 0 ? :foo : :bar)
~~~

The result is:

~~~
{".../target.rb" => {
  :lines => [1, 1],
  :branches => {
    [:unless, 0, 1] => { [:else, 1, 1] => 0, [:then, 2, 1] => 1 },
    [:if, 3, 2] => { [:then, 4, 2] => 0, [:else, 5, 2] => 1 }
  }
}}
~~~

Line coverage tells coverage 100%, but branch coverage shows that the `unless` statement of the first line has never taken true and that the ternary operator has never taken true.

## Current status

I've already committed the feature in trunk as an experimental feature.  To enable the feature, you need to set the environment variable `COVERAGE_EXPERIMENTAL_MODE` = `true`.  I plan to activate this feature by default by Ruby 2.5 release, if there is no big problem.

## Key format

The current implementation uses `[<label>, <unique ID>, <lineno>]`, like `[:if, 0, 1]`, to represent the site of branch.  `<unique ID>` is required for the case where multiple branches are in one line.

I think this format is arguable.  I thought of some other candidates:

* `[<label>, <lineno>, <column-no>]`: A big problem, how should we handle TAB character?
* `[<label>, <offset from file head>]`: Looks good for machine readability.
* Are `<label>` and `<lineno>` needed? They are useful for human, but normally, this result will be processed by a visualization script (such as SimpleCov).

What do you think?  I'm now thinking that `[<label>, <lineno>, <offset from file head>]` is reasonable, but it is hard for me to implement.  I'll try later but parse.y is so difficult... (A patch is welcome!)

## Compatibility

This API is 100% compatible.  If no keyword argument is given, the result will be old format, i.e., a hash from filename to an array that represents line coverage.

~~~
# compatiblie mode
Coverage.start
load "target.rb"
p Coverage.result
#=> {".../target.rb" => [1, 1, 1, ...] }

# If "lines: true" is given, the result format differs slightly
Coverage.start(lines: true)
load "target.rb"
p Coverage.result
#=> {".../target.rb" => { :lines => [1, 1, 1, ...] } }
~~~

## Method coverage

Method coverage is also supported.  You can measure it by using `Coverage.start(methods: true)`.

target.rb

~~~
 1: def foo
 2: end
 3: def bar
 4: end
 5: def baz
 6: end
 7:
 8: foo
 9: foo
10: bar
~~~

result (manually formatted)

~~~
{".../target.rb"=> {
  :methods => {
    [:foo, 0, 1] => 2,
    [:bar, 1, 3] => 1,
    [:baz, 2, 5] => 0,
  }
}}
~~~

## Notes

* `if` statements whose condtion is literal, such as `if true` and `if false`, are not considered as a branch.

* `while`, `until`, and `case` are also supported.  See Examples 2 and 3.

* This proposal is based on #9508.  The proposal has [some spec-level issues](https://github.com/ruby/ruby/pull/511#issuecomment-328753499), but the work was really inspiring me.

## Future work

* Someone may want to know how many times an one-line block is executed, such as `n.times {  }`.

* Someone may want to know how many times each method call is executed, such as `obj.foo.bar.baz` (For example, if method `foo` always raises an exception, calls to `bar` and `baz` are not executed.)

## Some examples

### Example 1

target.rb

~~~
1: if 1 == 0
2:   p :match1
3: elsif 1 == 0
4:   p :match2
5: else
6:   p :not_match
7: end
~~~

result (manually formatted)

~~~
{"target.rb" => {
  :lines => [1, 0, 1, 0, nil, 1, nil],
  :branches => {
    [:if, 1] => { [:then, 2] => 0, [:else, 3] => 1 },
    [:if, 3] => { [:then, 4] => 0, [:else, 5] => 1 },
  }
}
~~~

### Example 2

target.rb

~~~
 1:case :BOO
 2:when :foo then p :foo
 3:when :bar then p :bar
 4:when :baz then p :baz
 5:else p :other
 6:end
 7:
 8:x = 3
 9:case
10:when x == 0 then p :foo
11:when x == 1 then p :bar
12:when x == 2 then p :baz
13:else p :other
14:end
~~~

result (manually formatted)

~~~
{"target.rb" => {
  :lines => [1, 0, 0, 0, 1, nil, nil, 1, 1, 1, 1, 1, 1, nil],
  :branches => {
    [:case, 1] => {
      [:when, 2] => 0,
      [:when, 3] => 0,
      [:when, 4] => 0,
      [:else, 5] => 1
    },
    [:case, 8] => {
      [:when, 9] => 0,
      [:when, 10] => 0,
      [:when, 11] => 0,
      [:else, 12] => 1
    }
  }
}
~~~

### Example 3

target.rb

~~~
 1:n = 0
 2:while n < 100
 3:  n += 1
 4:end
~~~

result (manually formatted)

~~~
{"target.rb" => {
  :lines => [1, 101, 100, nil],
  :branches => {
    [:while, 2] => {
      [:body, 3] => 100,
      [:end, 5] => 1
    }
  }
}
~~~



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>