From: Benoit Daloze Date: 2016-12-13T23:39:43+01:00 Subject: [ruby-core:78633] ruby/spec needs help from CRuby committers --===============1133943555== Content-Type: multipart/alternative; boundary=94eb2c06cfd0eeefc0054391e760 --94eb2c06cfd0eeefc0054391e760 Content-Type: text/plain; charset=UTF-8 Currently, ruby/spec is maintained mostly by individuals and enjoys the contribution of a few CRuby committers. Unfortunately, the task of making specs for all the new features and changes in MRI is a huge task and too much for these people. If you look at specs for changes in Ruby 2.3, they were finally completed around 1 year after 2.3.0 was out. This is too long and far from ideal. I believe now is the time to change that. If CRuby wants to define a language and allow alternative implementations to target the Ruby language, then contributing new specs for new features and changes to ruby/spec is the best way they can help. Currently, CRuby committers add tests under test/ or test/ruby/. This is very hard to reuse for other implementations, and more importantly does not document what the feature is supposed to do. Instead, it provides assertions on specific data points. This means, every time somebody read these tests, they have to reverse engineer these assertions to understand the behavior and/or read MRI source code. Both of these tasks are taking a large amount of time and there is no point for many people to have to do these tasks over and over again when the behavior could be documented once clearly. This is the strongest point for ruby/spec. Every spec example ("test") has a description, that says how some functionality is supposed to behave. If these descriptions are done properly, there is no need to look at MRI source or to reverse engineer the assertions to understand how the functionality behaves. Having this description, it is much faster to implement the correct behavior, and test it. Spec examples are also typically short and only contain a couple assertions, each identifying a part of the behavior. This is immensely useful for other implementations trying to implement a new feature, as they can implement the most important parts first, and come back to the tricky edge cases later (and there are many in Ruby). Also, they can update specs and then work on new specs one by one while still running all previous specs. No alternative Ruby implementation to this day passes all of ruby/spec, neither MRI tests. The ability to keep tests green while tagging (marking) known failures is critical. This is built-in in ruby/spec. There are many other technical considerations, but in my opinion these descriptions and focused small spec examples are the most important. For instance, should vs expect().to() vs assert_equal is just syntactic sugar and does not matter at all in comparison. I will just give one example I know well, tests for File.join: MRI tests: https://github.com/ruby/ruby/blob/42d9712e78744e19ac71a7c0eaa099ac9203e1b1/test/ruby/test_file_exhaustive.rb#L1224-L1261 ruby/spec: https://github.com/ruby/spec/blob/2a62f378c0c3151aa5abfe66a8eae2016b29c78a/core/file/join_spec.rb Feel free to take another example. I only want to illustrate what I mean about descriptions and small examples. A point of friction which has been observed multiple times is that ruby/spec is a different repository and there are synchronization problems. For example, if a change happens on MRI trunk and it needs to adapt ruby/spec, it will pass either ruby/ruby CI or ruby/spec CI but never both. I want to fix this and I want to encourage CRuby developers to write specs. Therefore I would like to integrate an in-tree copy of ruby/spec under spec/rubyspec/. This way, implementation and tests can be changed in a single commit, just like with MRI tests. I will keep synchronizing the changes to ruby/spec, so it can be used directly and other people can contribute there. I know this synchronization works because I have been doing it for over 2 years. This process is also well documented: https://github.com/ruby/spec/wiki/Merging-specs-from-JRuby-and-other-sources There are many advantages if the person implementing the feature also document the behavior: The behavior will be documented by the most knowledgable person on the subject. It will also likely contain details about edge-cases and implementation, which are very desirable for ruby/spec. Descriptions are written in english, which might be a difficult for some people less fluent in english. But we are not writing literature, very simple english talking about Ruby concepts and implementation is fine! # So here is my request to the CRuby developers: I would like to ask the CRuby developers to add specs in ruby/spec when implementing new features, changing existing Ruby behavior and writing new tests. Of course, nobody likes to write tests twice, so I also propose these specs can be done *instead* of MRI tests. In my opinion, MRI tests should only be needed for MRI-specific behavior such as RubyVM. Regression tests are perfectly fine in ruby/spec. If one Ruby implementation regresses or fail in some edge case, it's very likely another will as well. ruby/spec and mspec (the runner for ruby/spec) are not prefect and are not fixed in stone, they can evolve and improve. If you think mspec is too complex, we can simplify it. If there is some feature in the MRI test harness not present in mspec, we can add it. If you think some syntax is confusing, let's discuss it and change it. Dear CRuby developers, what do you think? What are your concerns with this approach? --94eb2c06cfd0eeefc0054391e760 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Currently, ruby/spec is maintained mostly by individuals a= nd enjoys the contribution of a few CRuby=C2=A0committers.
Unfortunatel= y, the task of making specs for all the new features and changes in MRI is = a huge task and too much for these people.
If you look at specs f= or changes in Ruby 2.3, they were finally completed around 1 year after 2.3= .0 was out. This is too long and far from ideal.

I belie= ve now is the time to change that.
If CRuby wants to define a lan= guage and allow alternative implementations to target the Ruby language,
then contributing new specs for new features and changes to ruby/sp= ec is the best way they can help.

Currently, CRuby= committers add tests under test/ or test/ruby/.
This is very har= d to reuse for other implementations,
and more importantly does n= ot document what the feature is supposed to do.
Instead, it provi= des assertions on specific data points.
This means, every time so= mebody read these tests, they have to reverse engineer these assertions to = understand the behavior and/or read MRI source code.
Both of thes= e tasks are taking a large amount of time and there is no point for many pe= ople to have to do these tasks over and over again when the behavior could = be documented once clearly.

This is the stro= ngest point for ruby/spec.
Every spec example ("test") = has a description, that says how some functionality is supposed to behave.<= /div>
If these descriptions are done properly, there is no need to look= at MRI source or to reverse engineer the assertions to understand how the = functionality behaves.
Having this description, it is much faster= to implement the correct behavior, and test it.
Spec examples ar= e also typically short and only contain a couple assertions, each identifyi= ng a part of the behavior.
This is immensely useful for other imp= lementations trying to implement a new feature,
as they can imple= ment the most important parts first, and come back to the tricky edge cases= later (and there are many in Ruby).
Also, they can update specs = and then work on new specs one by one while still running all previous spec= s.
No alternative Ruby implementation to this day passes all of r= uby/spec, neither MRI tests.
The ability to keep tests green whil= e tagging (marking) known failures is critical. This is built-in in ruby/sp= ec.

There are many other technical considerations,=
but in my opinion these descriptions and focused small spec exam= ples are the most important.
For instance, should vs expect().to(= ) vs assert_equal is just syntactic sugar and does not matter at all in com= parison.

I will just give one example I know well,= tests for File.join:
MRI tests: https://github.com/ruby/ruby/blob/42d9712e78744e1= 9ac71a7c0eaa099ac9203e1b1/test/ruby/test_file_exhaustive.rb#L1224-L1261=
ruby/spec:=C2=A0https://githu= b.com/ruby/spec/blob/2a62f378c0c3151aa5abfe66a8eae2016b29c78a/core/file/joi= n_spec.rb
Feel free to take another example. I only want to i= llustrate what I mean about descriptions and small examples.

=
A point of friction which has been observed multiple times is th= at ruby/spec is a different repository and there are synchronization proble= ms.
For example, if a change happens on MRI trunk and it needs to= adapt ruby/spec, it will pass either ruby/ruby CI or ruby/spec CI but neve= r both.
I want to fix this and I want to encourage CRuby develope= rs to write specs.

Therefore I would like to integ= rate an in-tree copy of ruby/spec under spec/rubyspec/.
This way,= implementation and tests can be changed in a single commit, just like with= MRI tests.
I will keep synchronizing the changes to ruby/spec, s= o it can be used directly and other people can contribute there.
= I know this synchronization works because I have been doing it for over 2 y= ears.
This process is also well documented:=C2=A0ht= tps://github.com/ruby/spec/wiki/Merging-specs-from-JRuby-and-other-sources<= /a>

There are many advantages if the person implem= enting the feature also document the behavior:
The behavior will = be documented by the most knowledgable person on the subject.
It = will also likely contain details about edge-cases and implementation, which= are very desirable for ruby/spec.
Descriptions are written in en= glish, which might be a difficult for some people less fluent in english.
But we are not writing literature, very simple english talking abo= ut Ruby concepts and implementation is fine!

# So = here is my request to the CRuby developers:
I would like to ask t= he CRuby developers=C2=A0to add specs in ruby/spec when implementing new fe= atures, changing existing Ruby behavior and writing new tests.
Of= course, nobody likes to write tests twice, so I also propose these specs c= an be done *instead* of MRI tests.
In my opinion, MRI tests shoul= d only be needed for MRI-specific behavior such as RubyVM.
Regression te= sts are perfectly fine in ruby/spec.
If one Ruby implementation r= egresses or fail in some edge case, it's very likely another will as we= ll.

ruby/spec and mspec (the runner for ruby/spec)= are not prefect and are not fixed in stone, they can evolve and improve.
If you think mspec is too complex, we can simplify it.
I= f there is some feature in the MRI test harness not present in mspec, we ca= n add it.
If you think some syntax is confusing, let's discus= s it and change it.

Dear CRuby developers, what do= you think?
What are your concerns with this approach?
--94eb2c06cfd0eeefc0054391e760-- --===============1133943555== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline Unsubscribe: --===============1133943555==--