From: Benoit Daloze Date: 2015-02-17T13:32:26+01:00 Subject: [ruby-core:68140] Re: Future of test suites for Ruby --089e013c5d24a07298050f47e7cd Content-Type: text/plain; charset=UTF-8 Hello, I think we all agree both test suites are useful. The big question remains of what do we do with the existing RubySpec tests. I do not think it is reasonable to migrate one suite entirely over the other as this is too much work and there are various reasons to prefer one over the other in different contexts. But, I would like to have the possibility to still contribute to RubySpec in a way that any implementation may use it and to merge efforts coming from different sources to improve RubySpec. - The first step is to bring nurse/rubyspec up to date with rubyspec/rubyspec archive's branch. I am willing to do that if no one has an objection against it. Any help is welcome of course! (just email me) - The second step is really to choose a canonical RubySpec repository, to avoid "death by too much forks". This repository should only contain RubySpec tests for practical reasons. We should allow many specs contributors to take part in merging changes and maintaining specs. I think this was a fatal flaw of rubyspec/rubyspec in that too few people had the large burden of merging and maintaining the specs. The main existing repository I see today is nurse/rubyspec. I am thinking the process could be similar to handling pull requests on ruby/ruby in that some contributors would provide feedback and merge them. The CI is very useful in this regard to ensure MRI is not broken inadvertently. - The third step is to decide what to do about new specs which are not contributed to the canonical repository directly. This is worth another discussion and I think it is wiser to first achieve the two first steps before discussing this in more details. I have been contributing to MRI, JRuby, RubySpec, and more recently on a new Ruby implementation. I really think we should not let RubySpec rot, as it is a valuable test suite. I spoke to people from each main Ruby implementation and they all wish for a bright future for Ruby tests and specs, so let's do this! What do you think? On 5 January 2015 at 23:18, Charles Oliver Nutter wrote: > I'll try to be brief so we can discuss all this. tl;dr: RubySpec is > valuable, MRI tests are valuable, we need to better utilize both of > them. > > As you may have heard, Brian Shirai officially ended the RubySpec > project last week: > http://rubini.us/2014/12/31/matz-s-ruby-developers-don-t-use-rubyspec/ > > I believe this is a good opportunity for us to address concerns about > how Ruby behavior is being tested over time. > > I think we can agree on the following facts: > > * ruby-core devs *have* been using RubySpec on and off, and at least > some of us see value in continuing to do so. naruse already maintains > a fork at github/naruse/rubyspec. > > * MRI's suite is not nearly as bad as the post claims. I have done > some coverage measurements that show it doing at least as well as > RubySpec for core classes, and significantly better for standard > libraries. > > * MRI's tests are often hard to read, too large, undercommented. > There's a lot of room for improvement, but this is obviously an > organic codebase that has had many contributors. > > * Ugly tests are harder for users (and contributors) to grok. Ugly > tests are less attractive targets for contribution. > > * RubySpec represents a valuable set of tests for Ruby behavior. There > will be a large overlap with MRI's tests, but RubySpec does things > that MRI's suite does not (and vice versa). > > * RubySpec is not gone; Rubinius will still need a suite to verify > Ruby compatibility. RubySpec lives in the Rubinius repository now. > > There's a number of paths forward for us right now. > > * Do nothing. Run the last public RubySpec dump periodically to ensure > we don't regress, as has been done with naruse's fork. New tests > continue to go into MRI suite. > > This is the status quo, and I don't think anyone benefits from it > other than folks who dislike change. > > * Begin an effort to bring missing tests from RubySpec into MRI's > suite and to clean up and improve MRI's suite. Abandon RubySpec at > some point. > > I think *some* effort should be made to improve MRI's suite in the > short term, since we obviously aren't going to throw it away. I don't > like the idea of spending a lot of time porting tests one way or the > other...at least until we've picked a winner. I don't like the idea of > abandoning RubySpec, since it has many valuable features. > > * Take over RubySpec as a fork and begin to maintain it as a secondary > suite for MRI. Encourage contributors and committers to write specs > rather than tests. > > I think this is possible. The barrier to contributing to RubySpec is > much lower if we're maintaining a fork of it, and I think that > eliminates many obstacles. I also feel like the future needs to be in > either RubySpec or a much cleaner version of MRI's suite, and there's > less work to start using the former now. > > ... > > So that's what I have for now. Give me your thoughts...anything goes. > --089e013c5d24a07298050f47e7cd Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello,
I think we all agree both test suites are useful.

= The big question remains of what do we do with the existing RubySpec tests.=
I do not think it is reasonable to migrate one suite entirely ove= r the other as this is too much work and there are various reasons to prefe= r one over the other in different contexts.

But, I would like = to have the possibility to still contribute to RubySpec in a way that any i= mplementation may use it
and to merge efforts coming from different sour= ces to improve RubySpec.

- The first step is to bring nurse/ru= byspec up to date with rubyspec/rubyspec archive's branch.
I a= m willing to do that if no one has an objection against it. Any help is wel= come of course! (just email me)

- The second step is really to= choose a canonical RubySpec repository, to avoid "death by too much f= orks".
This repository should only contain RubySpec tests for= practical reasons.
We should allow many specs contributors t= o take part in merging changes and maintaining specs.
I think this was a= fatal flaw of rubyspec/rubyspec in that too few people had the large burde= n of merging and maintaining the specs.

The main ex= isting repository I see today is nurse/rubyspec.
I am thinking the proce= ss could be similar to handling pull requests on ruby/ruby in that some con= tributors would provide feedback and merge them.
The CI is very useful i= n this regard to ensure MRI is not broken inadvertently.

=
- The third step is to decide what to do about new specs which a= re not contributed to the canonical repository directly.
This= is worth another discussion and I think it is wiser to first achieve the t= wo first steps before discussing this in more details.

I have been contributing to MRI, JRuby, RubySpec, and more recentl= y on a new Ruby implementation.
I really think we should not let RubySpe= c rot, as it is a valuable test suite.
I spoke to people from each main = Ruby implementation and they all wish for a bright future for Ruby tests an= d specs, so let's do this!

What do you thi= nk?

On= 5 January 2015 at 23:18, Charles Oliver Nutter <headius@headius.com= > wrote:
I'll try to be bri= ef so we can discuss all this. tl;dr: RubySpec is
valuable, MRI tests are valuable, we need to better utilize both of
them.

As you may have heard, Brian Shirai officially ended the RubySpec
project last week:
http://rubini.us/2014/12/31/matz-s-ruby-developer= s-don-t-use-rubyspec/

I believe this is a good opportunity for us to address concerns about
how Ruby behavior is being tested over time.

I think we can agree on the following facts:

* ruby-core devs *have* been using RubySpec on and off, and at least
some of us see value in continuing to do so. naruse already maintains
a fork at github/naruse/rubyspec.

* MRI's suite is not nearly as bad as the post claims. I have done
some coverage measurements that show it doing at least as well as
RubySpec for core classes, and significantly better for standard
libraries.

* MRI's tests are often hard to read, too large, undercommented.
There's a lot of room for improvement, but this is obviously an
organic codebase that has had many contributors.

* Ugly tests are harder for users (and contributors) to grok. Ugly
tests are less attractive targets for contribution.

* RubySpec represents a valuable set of tests for Ruby behavior. There
will be a large overlap with MRI's tests, but RubySpec does things
that MRI's suite does not (and vice versa).

* RubySpec is not gone; Rubinius will still need a suite to verify
Ruby compatibility. RubySpec lives in the Rubinius repository now.

There's a number of paths forward for us right now.

* Do nothing. Run the last public RubySpec dump periodically to ensure
we don't regress, as has been done with naruse's fork. New tests continue to go into MRI suite.

This is the status quo, and I don't think anyone benefits from it
other than folks who dislike change.

* Begin an effort to bring missing tests from RubySpec into MRI's
suite and to clean up and improve MRI's suite. Abandon RubySpec at
some point.

I think *some* effort should be made to improve MRI's suite in the
short term, since we obviously aren't going to throw it away. I don'= ;t
like the idea of spending a lot of time porting tests one way or the
other...at least until we've picked a winner. I don't like the idea= of
abandoning RubySpec, since it has many valuable features.

* Take over RubySpec as a fork and begin to maintain it as a secondary
suite for MRI. Encourage contributors and committers to write specs
rather than tests.

I think this is possible. The barrier to contributing to RubySpec is
much lower if we're maintaining a fork of it, and I think that
eliminates many obstacles. I also feel like the future needs to be in
either RubySpec or a much cleaner version of MRI's suite, and there'= ;s
less work to start using the former now.

...

So that's what I have for now. Give me your thoughts...anything goes.

--089e013c5d24a07298050f47e7cd--