From: ruby-core@... Date: 2017-11-18T05:26:40+00:00 Subject: [ruby-core:83811] [Ruby trunk Feature#13901] Add branch coverage Issue #13901 has been updated by marcandre (Marc-Andre Lafortune). We're thinking about an output format (for simplecov in particular) and I wanted to suggest for branch, method and "callsite" coverage to separate the "map" data from the "runs" data. What I call the "map" data is all the information that specifies what the branches/methods/callsites are (type, line, column, ...), while the "runs" data is the number of time it is run (or eventually `nil` if it is not executable). ``` { "path" => { :callsite_map => [range, other_range, ���], :callsite_runs => [runs, other_runs, ���], :branch_map => [branch, other_branch, ���], :branch_runs => [[runs, ���], [other_runs, ���], ���], ... }, "other_path" ��� } ``` The reason to separate the run data from the map is to allow for easier merging, and more efficient output in different files more efficiently too. The map can be written once while the runs can be written as many times as needed and merged at the end. Imagine a big app that runs a test suite in 42 parallel processes, and that callsite coverage is generated. One could generate the callsite map once, write it down, and each of the 42 process could write the runs at the end. The map for each callsite would probably be at least 4 integers (start_line, start_column, end_line, end_column), with maybe a type (:send, :def, etc.), so would be quite a bit larger than each runs file. The 43 files would be easier to merge later on too. This splitting is what Istanbul (the coverage tool of Google's Babel) is doing. It is the direction that DeepCover is likely to take too. Note that this is "compatible" with the existing line coverage in the sense that for line coverage, the "map" is trivial and thus not needed; only the runs are output. It's only for the other coverage types that the map has to be specified. ---------------------------------------- Feature #13901: Add branch coverage https://bugs.ruby-lang.org/issues/13901#change-67844 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal * Assignee: * Target version: ---------------------------------------- I plan to add "branch coverage" (and "method coverage") as new target types of coverage.so, the coverage measurement library. I'd like to introduce this feature for Ruby 2.5.0. Let me to hear your opinions. ## Basic Usage of the Coverage API The sequence is the same as the current: (1) require "coverage.so", (2) start coverage measurement by `Coverage.start`, (3) load a program being measured (typically, a test runner program), and (4) get the result by `Coverage.result`. When you pass to `Coverage.start` with keyword argument "`branches: true`", branch coverage measurement is enabled. test.rb ~~~ require "coverage" Coverage.start(lines: true, branches: true) load "target.rb" p Coverage.result ~~~ target.rb ~~~ 1: if 1 == 0 2: p :match 3: else 4: p :not_match 5: end ~~~ By measuring coverage of target.rb, the result will be output (manually formatted): ~~~ $ ruby test.rb :not_match {".../target.rb" => { :lines => [1, 0, nil, 1, nil], :branches => { [:if, 0, 1] => { [:then, 1, 2] => 0, [:else, 2, 4] => 1 } } } ~~~ `[:if, 0, 1]` reads "if branch at Line 1", and `[:then, 1, 2]` reads "then clause at Line 2". So, `[:if,0,1] => { [:then,1,2]=>0, [:else,2,4]=>0 }` reads "the branch from Line 1 to Line 2 has never executed, and the branch from Line 1 to Line 4 has executed once." The second number (`0` of `[:if, 0, 1]`) is a unique ID to avoid conflict, just in case where multiple branches are written in one line. This format of a key is discussed in "Key format" section. ## Why needed Traditional coverage (line coverage) misses a branch in one line. Branch coverage is useful to find such untested code. See the following example. target.rb ~~~ p(:foo) unless 1 == 0 p(1 == 0 ? :foo : :bar) ~~~ The result is: ~~~ {".../target.rb" => { :lines => [1, 1], :branches => { [:unless, 0, 1] => { [:else, 1, 1] => 0, [:then, 2, 1] => 1 }, [:if, 3, 2] => { [:then, 4, 2] => 0, [:else, 5, 2] => 1 } } }} ~~~ Line coverage tells coverage 100%, but branch coverage shows that the `unless` statement of the first line has never taken true and that the ternary operator has never taken true. ## Current status I've already committed the feature in trunk as an experimental feature. To enable the feature, you need to set the environment variable `COVERAGE_EXPERIMENTAL_MODE` = `true`. I plan to activate this feature by default by Ruby 2.5 release, if there is no big problem. ## Key format The current implementation uses `[<label>, <unique ID>, <lineno>]`, like `[:if, 0, 1]`, to represent the site of branch. `<unique ID>` is required for the case where multiple branches are in one line. I think this format is arguable. I thought of some other candidates: * `[<label>, <lineno>, <column-no>]`: A big problem, how should we handle TAB character? * `[<label>, <offset from file head>]`: Looks good for machine readability. * Are `<label>` and `<lineno>` needed? They are useful for human, but normally, this result will be processed by a visualization script (such as SimpleCov). What do you think? I'm now thinking that `[<label>, <lineno>, <offset from file head>]` is reasonable, but it is hard for me to implement. I'll try later but parse.y is so difficult... (A patch is welcome!) ## Compatibility This API is 100% compatible. If no keyword argument is given, the result will be old format, i.e., a hash from filename to an array that represents line coverage. ~~~ # compatiblie mode Coverage.start load "target.rb" p Coverage.result #=> {".../target.rb" => [1, 1, 1, ...] } # If "lines: true" is given, the result format differs slightly Coverage.start(lines: true) load "target.rb" p Coverage.result #=> {".../target.rb" => { :lines => [1, 1, 1, ...] } } ~~~ ## Method coverage Method coverage is also supported. You can measure it by using `Coverage.start(methods: true)`. target.rb ~~~ 1: def foo 2: end 3: def bar 4: end 5: def baz 6: end 7: 8: foo 9: foo 10: bar ~~~ result (manually formatted) ~~~ {".../target.rb"=> { :methods => { [:foo, 0, 1] => 2, [:bar, 1, 3] => 1, [:baz, 2, 5] => 0, } }} ~~~ ## Notes * `if` statements whose condtion is literal, such as `if true` and `if false`, are not considered as a branch. * `while`, `until`, and `case` are also supported. See Examples 2 and 3. * This proposal is based on #9508. The proposal has [some spec-level issues](https://github.com/ruby/ruby/pull/511#issuecomment-328753499), but the work was really inspiring me. ## Future work * Someone may want to know how many times an one-line block is executed, such as `n.times { }`. * Someone may want to know how many times each method call is executed, such as `obj.foo.bar.baz` (For example, if method `foo` always raises an exception, calls to `bar` and `baz` are not executed.) ## Some examples ### Example 1 target.rb ~~~ 1: if 1 == 0 2: p :match1 3: elsif 1 == 0 4: p :match2 5: else 6: p :not_match 7: end ~~~ result (manually formatted) ~~~ {"target.rb" => { :lines => [1, 0, 1, 0, nil, 1, nil], :branches => { [:if, 1] => { [:then, 2] => 0, [:else, 3] => 1 }, [:if, 3] => { [:then, 4] => 0, [:else, 5] => 1 }, } } ~~~ ### Example 2 target.rb ~~~ 1:case :BOO 2:when :foo then p :foo 3:when :bar then p :bar 4:when :baz then p :baz 5:else p :other 6:end 7: 8:x = 3 9:case 10:when x == 0 then p :foo 11:when x == 1 then p :bar 12:when x == 2 then p :baz 13:else p :other 14:end ~~~ result (manually formatted) ~~~ {"target.rb" => { :lines => [1, 0, 0, 0, 1, nil, nil, 1, 1, 1, 1, 1, 1, nil], :branches => { [:case, 1] => { [:when, 2] => 0, [:when, 3] => 0, [:when, 4] => 0, [:else, 5] => 1 }, [:case, 8] => { [:when, 9] => 0, [:when, 10] => 0, [:when, 11] => 0, [:else, 12] => 1 } } } ~~~ ### Example 3 target.rb ~~~ 1:n = 0 2:while n < 100 3: n += 1 4:end ~~~ result (manually formatted) ~~~ {"target.rb" => { :lines => [1, 101, 100, nil], :branches => { [:while, 2] => { [:body, 3] => 100, [:end, 5] => 1 } } } ~~~ -- https://bugs.ruby-lang.org/ Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe> <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>