From: mame@... Date: 2017-11-03T23:38:48+00:00 Subject: [ruby-core:83664] [Ruby trunk Feature#13901] Add branch coverage Issue #13901 has been updated by mame (Yusuke Endoh). marcandre (Marc-Andre Lafortune) wrote: > mame (Yusuke Endoh) wrote: > > `obj.foo.foo` has two NODE_CALLs. It is difficult to distinguish them by only the beginning, unless we have other information about the node in question. Honestly, I'm unsure if we can distinguish all relevant NODEs by only a pair of the beginning and the end. > > It depends what you consider the beginning position, but if you return the beginning of the method sending (4 vs 8 in your example, or 5 vs 9), then it's easy to distinguish between the two NODE_CALLs. These correspond to `dot`, or `selector` range in `parser` (try `ruby-parser -L -e "obj.foo.foo"` :-) The parser gem produces a good work. > > I had no thought of counting trivial evaluation such as a literal, but if your pure-Ruby approach does not cause performance issue, now I'd like to consider supporting it. It would be difficult (for me) to implement all in 2.5, though. > > Just to be clear: literals (or anything that can not change control flow) do not cause *any performance loss at all* for us. > > We always deduce their execution from normal control flow. E.g. in `var = [1, 2, 3]`, to know if `3` was executed, we check if it's previous sibbling (`2`) was executed. To know that, we check if `1` was executed. `1` has no previous sibbling, so we check the parent `[]`. It itself checks the parent `a = `, which finally asks bring us to our parent root. Only this parent introduces a small performance hit with a counter `$global[][] += 1`. So we get the execution count of the literals, the array creation and the variable assignment for free. > > To summarize our strategy: a Node is not responsible to know how many times it was entered by control flow, that is the responsibility of the parent node. A Node's responsibility is to know how many times control flow exited it normally. For many nodes like literals, exit = entry so there's nothing special to be done. A "send" node must add a $tracker+=1 after the call to know that. I need to credit Maxime Lapointe for pointing us in that direction early on. I see... It is a very interesting approach, but it seems to completely ignore asynchronous interrupt, such as signal and `Thread#raise`. I wonder if it is not a good design decision for the embedded coverage measurement library. I'll think about it. Thank you for teaching me, anyway. ---------------------------------------- Feature #13901: Add branch coverage https://bugs.ruby-lang.org/issues/13901#change-67690 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal * Assignee: * Target version: ---------------------------------------- I plan to add "branch coverage" (and "method coverage") as new target types of coverage.so, the coverage measurement library. I'd like to introduce this feature for Ruby 2.5.0. Let me to hear your opinions. ## Basic Usage of the Coverage API The sequence is the same as the current: (1) require "coverage.so", (2) start coverage measurement by `Coverage.start`, (3) load a program being measured (typically, a test runner program), and (4) get the result by `Coverage.result`. When you pass to `Coverage.start` with keyword argument "`branches: true`", branch coverage measurement is enabled. test.rb ~~~ require "coverage" Coverage.start(lines: true, branches: true) load "target.rb" p Coverage.result ~~~ target.rb ~~~ 1: if 1 == 0 2: p :match 3: else 4: p :not_match 5: end ~~~ By measuring coverage of target.rb, the result will be output (manually formatted): ~~~ $ ruby test.rb :not_match {".../target.rb" => { :lines => [1, 0, nil, 1, nil], :branches => { [:if, 0, 1] => { [:then, 1, 2] => 0, [:else, 2, 4] => 1 } } } ~~~ `[:if, 0, 1]` reads "if branch at Line 1", and `[:then, 1, 2]` reads "then clause at Line 2". So, `[:if,0,1] => { [:then,1,2]=>0, [:else,2,4]=>0 }` reads "the branch from Line 1 to Line 2 has never executed, and the branch from Line 1 to Line 4 has executed once." The second number (`0` of `[:if, 0, 1]`) is a unique ID to avoid conflict, just in case where multiple branches are written in one line. This format of a key is discussed in "Key format" section. ## Why needed Traditional coverage (line coverage) misses a branch in one line. Branch coverage is useful to find such untested code. See the following example. target.rb ~~~ p(:foo) unless 1 == 0 p(1 == 0 ? :foo : :bar) ~~~ The result is: ~~~ {".../target.rb" => { :lines => [1, 1], :branches => { [:unless, 0, 1] => { [:else, 1, 1] => 0, [:then, 2, 1] => 1 }, [:if, 3, 2] => { [:then, 4, 2] => 0, [:else, 5, 2] => 1 } } }} ~~~ Line coverage tells coverage 100%, but branch coverage shows that the `unless` statement of the first line has never taken true and that the ternary operator has never taken true. ## Current status I've already committed the feature in trunk as an experimental feature. To enable the feature, you need to set the environment variable `COVERAGE_EXPERIMENTAL_MODE` = `true`. I plan to activate this feature by default by Ruby 2.5 release, if there is no big problem. ## Key format The current implementation uses `[