[ruby-core:84881] [Ruby trunk Bug#9572] Restarting Coverage does not produce correct coverage result

Issue #9572 has been updated by danmayer (Dan Mayer).


> Pause / Resume just seems like ignoring coverage information for some code. 

Yes, the goal would be to ignore it for some code, but with reduced performance costs.

My thought was much of the performance cost was related to tracking the events, if most of the cost is incurred at compile time (as I don't think there is anything I could do about that) the feature might not be very helpful. I was primarily thinking if one could pause the event listeners, and performance would go nearly back to normal.

~~~ c
rb_remove_event_hook((rb_event_hook_func_t) update_line_coverage);
rb_remove_event_hook((rb_event_hook_func_t) update_branch_coverage);
rb_remove_event_hook((rb_event_hook_func_t) update_method_coverage);
~~~

You can see the POC that I am using to test this idea here: https://github.com/danmayer/ruby/compare/trunk...danmayer:feature/coverage_pause

I get that to get any coverage data `Coverage.start` would need to be called before requiring files.

> Especially being able to continue getting coverage results for files already required

What I was getting at here is currently if you call Coverage.result then Coverage.start again, it stops tracking previously tracked files. Only NEWLY required files after the second start are tracked. I think with your current `peek_result` this issue can already be considered solved. There just isn't a good reason to ever call Coverage.start a second time.

I guess the question and value of the change depends on if the act of turning on and off the above event hooks can result in worthwhile performance benefits. Or if most cost is added once we compile files after calling `Coverage.start`. Does that make more sense?

I expanded my sample application here: https://github.com/danmayer/coverage-bug/blob/master/example.rb to try to show running

* without coverage at all
* traditional coverage support as it exists today
* being able to pause / resume coverage during execution

The results are below, but I think I need to come up with a more realistic example to get a better idea of the performance impacts. I added some comments to try to make it a bit more clear

~~~ text
# coverage.start is never called
[master][~/projects/coverage-bug] ruby example.rb
       user     system      total        real
  11.261481   0.212573  11.474054 ( 11.548027)
coverage
{}
done
#coverage.start and coverage.result are used
[master][~/projects/coverage-bug] COVERAGE=true ruby example.rb
       user     system      total        real
  11.673899   0.198156  11.872055 ( 11.946569)
coverage
{"/usr/local/lib/ruby/2.6.0/bigdecimal/math.rb"=>[nil, 1, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 1, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 0, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, nil, 0, nil, nil, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, nil, 0, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, nil, 0, nil, nil, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, nil, 0, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, nil, 0, 0, 0, 0, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 2001, 2001, 2001, 2001, 2001, nil, 2001, 2001, nil, 2001, nil, 2001, 2001, 2001, 1456728, 1452726, 1452726, 1452726, 1452726, 1452726, nil, nil, 2001, 2001, 2001, 432216, 428214, 428214, 428214, 428214, 
 428214, nil, 2001, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 0, 0, nil, nil], "/Users/danmayer/projects/coverage-bug/app.rb"=>[1, 1, 2, 2, nil, 2, 2, 2001, nil, nil, nil], "/Users/danmayer/projects/coverage-bug/app_proxy.rb"=>[1, 1, 2, 2, 2, 2, nil, 2, 2, 1, 0, 0, nil, 1, nil, nil]}
done
# coverage.start, coverage.pause, and coverage.resume used along with coverage.peek_result
[master][~/projects/coverage-bug] ENHANCED_COVERAGE=true ruby example.rb 
       user     system      total        real
  11.467796   0.196386  11.664182 ( 11.729636)
  12.113387   0.185672  12.299059 ( 12.365711)
coverage
{"/usr/local/lib/ruby/2.6.0/bigdecimal/math.rb"=>[nil, 1, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 1, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 0, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, nil, 0, nil, nil, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, nil, 0, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, nil, 0, nil, nil, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, nil, 0, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, nil, 0, 0, 0, 0, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 2000, 2000, 2000, 2000, 2000, nil, 2000, 2000, nil, 2000, nil, 2000, 2000, 2000, 1456000, 1452000, 1452000, 1452000, 1452000, 1452000, nil, nil, 2000, 2000, 2000, 432000, 428000, 428000, 428000, 428000, 
 428000, nil, 2000, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, 1, 0, 0, nil, nil], "/Users/danmayer/projects/coverage-bug/app.rb"=>[1, 1, 1, 1, nil, 1, 1, 2000, nil, nil, nil], "/Users/danmayer/projects/coverage-bug/app_proxy.rb"=>[1, 1, 0, 0, 0, 1, nil, 1, 1, 1, 1, 0, nil, 0, nil, nil]}
done
~~~


----------------------------------------
Bug #9572: Restarting Coverage does not produce correct coverage result
https://bugs.ruby-lang.org/issues/9572#change-69588

* Author: sean_ferguson (Sean Ferguson)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-01-13 trunk 61811) [x86_64-darwin15]
* Backport: 
----------------------------------------
Feature 4796(https://www.ruby-forum.com/topic/1811306#1001864) appears
to have a bug in it.

While the test listed there does pass the following test does not:
<pre>
require "test/unit"
require "coverage"
require 'tmpdir'
class TestCoverage < Test::Unit::TestCase
  def test_restarting_coverage
    Dir.mktmpdir {|tmp|
      Dir.chdir(tmp) {
        File.open("test.rb", "w") do |f|
          f.puts <<-EOS
            def coverage_test_method
              puts :ok
            end
          EOS
        end
        Coverage.start
        require tmp + '/test.rb'
        Coverage.result
        Coverage.start
        coverage_test_method
        result = Coverage.result
        assert_equal 1, result.size
        assert_equal [0, 1, nil], result.first[1] # coverage stats show an empty array here
      }
    }
  end
end
</pre>
It appears that while the coverage is finding the correct files it is
not giving any coverage stats for those files. Knowing this information would be very helpful in determining test coverage data for individual test files. I'm not very familiar
with how the coverage library works, but if you can point me at where to
look I can give fixing it a try.

Thanks,

Sean Ferguson



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>