[ruby-core:78634] Re: ruby/spec needs help from CRuby committers

Typo: "are not prefect" =>  "are not perfect"

On Tue, Dec 13, 2016 at 11:39 PM, Benoit Daloze <[e-mail]> wrote:

> Currently, ruby/spec is maintained mostly by individuals and enjoys the
> contribution of a few CRuby committers.
> Unfortunately, the task of making specs for all the new features and
> changes in MRI is a huge task and too much for these people.
> If you look at specs for changes in Ruby 2.3, they were finally completed
> around 1 year after 2.3.0 was out. This is too long and far from ideal.
>
> I believe now is the time to change that.
> If CRuby wants to define a language and allow alternative implementations
> to target the Ruby language,
> then contributing new specs for new features and changes to ruby/spec is
> the best way they can help.
>
> Currently, CRuby committers add tests under test/ or test/ruby/.
> This is very hard to reuse for other implementations,
> and more importantly does not document what the feature is supposed to do.
> Instead, it provides assertions on specific data points.
> This means, every time somebody read these tests, they have to reverse
> engineer these assertions to understand the behavior and/or read MRI source
> code.
> Both of these tasks are taking a large amount of time and there is no
> point for many people to have to do these tasks over and over again when
> the behavior could be documented once clearly.
>
> This is the strongest point for ruby/spec.
> Every spec example ("test") has a description, that says how some
> functionality is supposed to behave.
> If these descriptions are done properly, there is no need to look at MRI
> source or to reverse engineer the assertions to understand how the
> functionality behaves.
> Having this description, it is much faster to implement the correct
> behavior, and test it.
> Spec examples are also typically short and only contain a couple
> assertions, each identifying a part of the behavior.
> This is immensely useful for other implementations trying to implement a
> new feature,
> as they can implement the most important parts first, and come back to the
> tricky edge cases later (and there are many in Ruby).
> Also, they can update specs and then work on new specs one by one while
> still running all previous specs.
> No alternative Ruby implementation to this day passes all of ruby/spec,
> neither MRI tests.
> The ability to keep tests green while tagging (marking) known failures is
> critical. This is built-in in ruby/spec.
>
> There are many other technical considerations,
> but in my opinion these descriptions and focused small spec examples are
> the most important.
> For instance, should vs expect().to() vs assert_equal is just syntactic
> sugar and does not matter at all in comparison.
>
> I will just give one example I know well, tests for File.join:
> MRI tests: https://github.com/ruby/ruby/blob/
> 42d9712e78744e19ac71a7c0eaa099ac9203e1b1/test/ruby/test_
> file_exhaustive.rb#L1224-L1261
> ruby/spec: https://github.com/ruby/spec/blob/
> 2a62f378c0c3151aa5abfe66a8eae2016b29c78a/core/file/join_spec.rb
> Feel free to take another example. I only want to illustrate what I mean
> about descriptions and small examples.
>
> A point of friction which has been observed multiple times is that
> ruby/spec is a different repository and there are synchronization problems.
> For example, if a change happens on MRI trunk and it needs to adapt
> ruby/spec, it will pass either ruby/ruby CI or ruby/spec CI but never both.
> I want to fix this and I want to encourage CRuby developers to write specs.
>
> Therefore I would like to integrate an in-tree copy of ruby/spec under
> spec/rubyspec/.
> This way, implementation and tests can be changed in a single commit, just
> like with MRI tests.
> I will keep synchronizing the changes to ruby/spec, so it can be used
> directly and other people can contribute there.
> I know this synchronization works because I have been doing it for over 2
> years.
> This process is also well documented: https://github.
> com/ruby/spec/wiki/Merging-specs-from-JRuby-and-other-sources
>
> There are many advantages if the person implementing the feature also
> document the behavior:
> The behavior will be documented by the most knowledgable person on the
> subject.
> It will also likely contain details about edge-cases and implementation,
> which are very desirable for ruby/spec.
> Descriptions are written in english, which might be a difficult for some
> people less fluent in english.
> But we are not writing literature, very simple english talking about Ruby
> concepts and implementation is fine!
>
> # So here is my request to the CRuby developers:
> I would like to ask the CRuby developers to add specs in ruby/spec when
> implementing new features, changing existing Ruby behavior and writing new
> tests.
> Of course, nobody likes to write tests twice, so I also propose these
> specs can be done *instead* of MRI tests.
> In my opinion, MRI tests should only be needed for MRI-specific behavior
> such as RubyVM.
> Regression tests are perfectly fine in ruby/spec.
> If one Ruby implementation regresses or fail in some edge case, it's very
> likely another will as well.
>
> ruby/spec and mspec (the runner for ruby/spec) are not prefect and are not
> fixed in stone, they can evolve and improve.
> If you think mspec is too complex, we can simplify it.
> If there is some feature in the MRI test harness not present in mspec, we
> can add it.
> If you think some syntax is confusing, let's discuss it and change it.
>
> Dear CRuby developers, what do you think?
> What are your concerns with this approach?
>

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>