From: Benoit Daloze Date: 2016-12-13T23:58:12+01:00 Subject: [ruby-core:78634] Re: ruby/spec needs help from CRuby committers --===============1724202299== Content-Type: multipart/alternative; boundary=001a11411ef60d2cab0543922a9b --001a11411ef60d2cab0543922a9b Content-Type: text/plain; charset=UTF-8 Typo: "are not prefect" => "are not perfect" On Tue, Dec 13, 2016 at 11:39 PM, Benoit Daloze wrote: > Currently, ruby/spec is maintained mostly by individuals and enjoys the > contribution of a few CRuby committers. > Unfortunately, the task of making specs for all the new features and > changes in MRI is a huge task and too much for these people. > If you look at specs for changes in Ruby 2.3, they were finally completed > around 1 year after 2.3.0 was out. This is too long and far from ideal. > > I believe now is the time to change that. > If CRuby wants to define a language and allow alternative implementations > to target the Ruby language, > then contributing new specs for new features and changes to ruby/spec is > the best way they can help. > > Currently, CRuby committers add tests under test/ or test/ruby/. > This is very hard to reuse for other implementations, > and more importantly does not document what the feature is supposed to do. > Instead, it provides assertions on specific data points. > This means, every time somebody read these tests, they have to reverse > engineer these assertions to understand the behavior and/or read MRI source > code. > Both of these tasks are taking a large amount of time and there is no > point for many people to have to do these tasks over and over again when > the behavior could be documented once clearly. > > This is the strongest point for ruby/spec. > Every spec example ("test") has a description, that says how some > functionality is supposed to behave. > If these descriptions are done properly, there is no need to look at MRI > source or to reverse engineer the assertions to understand how the > functionality behaves. > Having this description, it is much faster to implement the correct > behavior, and test it. > Spec examples are also typically short and only contain a couple > assertions, each identifying a part of the behavior. > This is immensely useful for other implementations trying to implement a > new feature, > as they can implement the most important parts first, and come back to the > tricky edge cases later (and there are many in Ruby). > Also, they can update specs and then work on new specs one by one while > still running all previous specs. > No alternative Ruby implementation to this day passes all of ruby/spec, > neither MRI tests. > The ability to keep tests green while tagging (marking) known failures is > critical. This is built-in in ruby/spec. > > There are many other technical considerations, > but in my opinion these descriptions and focused small spec examples are > the most important. > For instance, should vs expect().to() vs assert_equal is just syntactic > sugar and does not matter at all in comparison. > > I will just give one example I know well, tests for File.join: > MRI tests: https://github.com/ruby/ruby/blob/ > 42d9712e78744e19ac71a7c0eaa099ac9203e1b1/test/ruby/test_ > file_exhaustive.rb#L1224-L1261 > ruby/spec: https://github.com/ruby/spec/blob/ > 2a62f378c0c3151aa5abfe66a8eae2016b29c78a/core/file/join_spec.rb > Feel free to take another example. I only want to illustrate what I mean > about descriptions and small examples. > > A point of friction which has been observed multiple times is that > ruby/spec is a different repository and there are synchronization problems. > For example, if a change happens on MRI trunk and it needs to adapt > ruby/spec, it will pass either ruby/ruby CI or ruby/spec CI but never both. > I want to fix this and I want to encourage CRuby developers to write specs. > > Therefore I would like to integrate an in-tree copy of ruby/spec under > spec/rubyspec/. > This way, implementation and tests can be changed in a single commit, just > like with MRI tests. > I will keep synchronizing the changes to ruby/spec, so it can be used > directly and other people can contribute there. > I know this synchronization works because I have been doing it for over 2 > years. > This process is also well documented: https://github. > com/ruby/spec/wiki/Merging-specs-from-JRuby-and-other-sources > > There are many advantages if the person implementing the feature also > document the behavior: > The behavior will be documented by the most knowledgable person on the > subject. > It will also likely contain details about edge-cases and implementation, > which are very desirable for ruby/spec. > Descriptions are written in english, which might be a difficult for some > people less fluent in english. > But we are not writing literature, very simple english talking about Ruby > concepts and implementation is fine! > > # So here is my request to the CRuby developers: > I would like to ask the CRuby developers to add specs in ruby/spec when > implementing new features, changing existing Ruby behavior and writing new > tests. > Of course, nobody likes to write tests twice, so I also propose these > specs can be done *instead* of MRI tests. > In my opinion, MRI tests should only be needed for MRI-specific behavior > such as RubyVM. > Regression tests are perfectly fine in ruby/spec. > If one Ruby implementation regresses or fail in some edge case, it's very > likely another will as well. > > ruby/spec and mspec (the runner for ruby/spec) are not prefect and are not > fixed in stone, they can evolve and improve. > If you think mspec is too complex, we can simplify it. > If there is some feature in the MRI test harness not present in mspec, we > can add it. > If you think some syntax is confusing, let's discuss it and change it. > > Dear CRuby developers, what do you think? > What are your concerns with this approach? > --001a11411ef60d2cab0543922a9b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Typo: "are not prefe= ct" =3D>=C2=A0=C2=A0"a= re not perfect"

On Tue, Dec 13, 2016 at 11:39 PM, Benoit Daloze <eregon= tp@gmail.com> wrote:
Currently, ruby/spec is maintained mostly by individuals and enj= oys the contribution of a few CRuby=C2=A0committers.
Unfortunately, the= task of making specs for all the new features and changes in MRI is a huge= task and too much for these people.
If you look at specs for cha= nges in Ruby 2.3, they were finally completed around 1 year after 2.3.0 was= out. This is too long and far from ideal.

I believe now= is the time to change that.
If CRuby wants to define a language = and allow alternative implementations to target the Ruby language,
then contributing new specs for new features and changes to ruby/spec is = the best way they can help.

Currently, CRuby commi= tters add tests under test/ or test/ruby/.
This is very hard to r= euse for other implementations,
and more importantly does not doc= ument what the feature is supposed to do.
Instead, it provides as= sertions on specific data points.
This means, every time somebody= read these tests, they have to reverse engineer these assertions to unders= tand the behavior and/or read MRI source code.
Both of these task= s are taking a large amount of time and there is no point for many people t= o have to do these tasks over and over again when the behavior could be doc= umented once clearly.

This is the strongest = point for ruby/spec.
Every spec example ("test") has a = description, that says how some functionality is supposed to behave.
<= div>If these descriptions are done properly, there is no need to look at MR= I source or to reverse engineer the assertions to understand how the functi= onality behaves.
Having this description, it is much faster to im= plement the correct behavior, and test it.
Spec examples are also= typically short and only contain a couple assertions, each identifying a p= art of the behavior.
This is immensely useful for other implement= ations trying to implement a new feature,
as they can implement t= he most important parts first, and come back to the tricky edge cases later= (and there are many in Ruby).
Also, they can update specs and th= en work on new specs one by one while still running all previous specs.
No alternative Ruby implementation to this day passes all of ruby/sp= ec, neither MRI tests.
The ability to keep tests green while tagg= ing (marking) known failures is critical. This is built-in in ruby/spec.

There are many other technical considerations,
=
but in my opinion these descriptions and focused small spec examples a= re the most important.
For instance, should vs expect().to() vs a= ssert_equal is just syntactic sugar and does not matter at all in compariso= n.

I will just give one example I know well, tests= for File.join:
Feel free to take another example. I only want to illust= rate what I mean about descriptions and small examples.

A point of friction which has been observed multiple times is that ru= by/spec is a different repository and there are synchronization problems.
For example, if a change happens on MRI trunk and it needs to adap= t ruby/spec, it will pass either ruby/ruby CI or ruby/spec CI but never bot= h.
I want to fix this and I want to encourage CRuby developers to= write specs.

Therefore I would like to integrate = an in-tree copy of ruby/spec under spec/rubyspec/.
This way, impl= ementation and tests can be changed in a single commit, just like with MRI = tests.
I will keep synchronizing the changes to ruby/spec, so it = can be used directly and other people can contribute there.
I kno= w this synchronization works because I have been doing it for over 2 years.=

There are many ad= vantages if the person implementing the feature also document the behavior:=
The behavior will be documented by the most knowledgable person = on the subject.
It will also likely contain details about edge-ca= ses and implementation, which are very desirable for ruby/spec.
D= escriptions are written in english, which might be a difficult for some peo= ple less fluent in english.
But we are not writing literature, ve= ry simple english talking about Ruby concepts and implementation is fine!

# So here is my request to the CRuby developers:
I would like to ask the CRuby developers=C2=A0to add specs in ruby/= spec when implementing new features, changing existing Ruby behavior and wr= iting new tests.
Of course, nobody likes to write tests twice, so= I also propose these specs can be done *instead* of MRI tests.
I= n my opinion, MRI tests should only be needed for MRI-specific behavior suc= h as RubyVM.
Regression tests are perfectly fine in ruby/spec.
If one Ruby implementation regresses or fail in some edge case, it's v= ery likely another will as well.

ruby/spec and msp= ec (the runner for ruby/spec) are not prefect and are not fixed in stone, t= hey can evolve and improve.
If you think mspec is too complex, we= can simplify it.
If there is some feature in the MRI test harnes= s not present in mspec, we can add it.
If you think some syntax i= s confusing, let's discuss it and change it.

D= ear CRuby developers, what do you think?
What are your concerns w= ith this approach?

--001a11411ef60d2cab0543922a9b-- --===============1724202299== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline Unsubscribe: --===============1724202299==--