From: Benoit Daloze Date: 2017-01-09T22:16:46+01:00 Subject: [ruby-core:79029] Re: ruby/spec needs help from CRuby committers --===============0444697578== Content-Type: multipart/alternative; boundary=001a1144b5c0fa98f90545afe487 --001a1144b5c0fa98f90545afe487 Content-Type: text/plain; charset=UTF-8 Thank you Shyouhei-san for your reply. On Wed, Jan 4, 2017 at 2:52 AM, Urabe Shyouhei wrote: > > I once was a SET (Software Engineer in Test) at a company of > 1,000+ employee. My role there included writing tests of someone > else's products, much like the situation of rubyspec. Why that > company had dedicated test-automation engineers was because > writing non-garbage acceptance tests in general requires a > different skill set from writing a production code. Of course, > people wrote their own unit tests and that's definitely a good > thing. But they did so to _boost_ their development, not to be > conservative about their specifications. > I think the difference is much smaller between ruby/spec and test/ruby, essentially it's the same code as unit tests, but in smaller chunks, each chunk with a plain English description. For me, the "spec" in the name is mostly to refer to these descriptions and the fact it tries to explain the behavior. It is not a formal specification of the Ruby language, it just intends to test the Ruby language, its core and standard libraries and also describes the reason behind the behavior. I think the person implementing a new feature is usually the most knowledgable about the feature and implementation, and therefore can give relevant tests and write helpful descriptions. Nothing is perfect, so there are many cases which are not covered or sometimes even misleading, but I think ruby/spec is a very valuable effort overall. So when it comes to ruby. I don't think test/ruby is the ideal > solution for everyone. I also admit it's ugly, and needs > improvements. But at least I can say it's a wrong idea to > abondon it. Doing so is much like to throw away unit tests and > let Selenium do everything. Just nonsense. Unit test is not a > spec. They exist separately because they have different > purposes. Merging them hurts both sides. The (failed) past > attempts to migrate rubyspec had problems on this point, seems to > me at least. > I do not propose to abandon test/ruby neither to merge it with ruby/spec as that seems unrealistic and of limited value. But, I would like to encourage using ruby/spec instead (or in complement) of test/ruby. I think it is quite similar to write ruby/specs or test/ruby tests, but I think ruby/spec tests have many advantages as described above. Maybe some people dislike or do not want to write ruby/spec, and that is fine. But if a few MRI committers contribute more actively to ruby/spec, it would already be a huge step towards testing and understanding Ruby better. The whole Ruby ecosystem would benefit as alternative implementations would be able to follow changes more efficiently and contribute back specs for additional cases. > That said. I also have to note that tests without specs is a > poor solution. The problems pointed by @eregon in this thread is > worth listening to. The company I worked before solved this > problem by hiring someone to write specs. That's not doable by > us (no such budget) but someone has to do that job somehow. Are > we going to let the core devs do that? Then I think we need to > motivate them. "Spec is a good thing you should do" is a true > assertion but sounds much like a homily. Instead show them it > benefits. For instance, find bugs using spec and show them how > it helps their developments. > Indeed, I found many bugs using ruby/spec. Testing from the perspective of trying to explain the behavior can discover many interesting cases. I reported 70 bugs on the Ruby tracker in 7 years. The majority of them where discovered by specs. Here are a few examples: * Inconsistent Float#round on Windows * Various difference of behavior on Windows * Coverage is leaking in global state * ObjectSpace.each_object exposes internal objects * Deadlock in #autoload * Incorrect encoding for Time#zone * Bugs in keyword arguments, there were many of them back then * Various methods changed the exception they throw in error cases Alternative implementations also have their own test suites. They could each have a different one and duplicate the efforts, or each use the same test suite known as ruby/spec to share the benefits. --001a1144b5c0fa98f90545afe487 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thank you Shyouhei-san for your reply.

On Wed, Jan 4, 2017 at 2:52 AM, U= rabe Shyouhei <shyouhei@ruby-lang.org> wrote: I once was a SET (Software Engineer in Test) at a company of
1,000+ employee.=C2=A0 My role there included writing tests of someone
else's products, much like the situation of rubyspec.=C2=A0 Why that company had dedicated test-automation engineers was because
writing non-garbage acceptance tests in general requires a
different skill set from writing a production code.=C2=A0 Of course,
people wrote their own unit tests and that's definitely a good
thing.=C2=A0 But they did so to _boost_ their development, not to be
conservative about their specifications.

I think the difference is much smaller between ruby/spec and test/ruby,
essentially it's the same code as unit tests, but in smaller c= hunks,
each chunk with a plain English description. For me, the &= quot;spec" in the name is
mostly to refer to these descripti= ons and the fact it tries to explain the behavior.
It is not a fo= rmal specification of the Ruby language, it just intends to test the Ruby l= anguage,
its core and standard libraries and also describes the r= eason behind the behavior.

I think the person impl= ementing a new feature is usually the most knowledgable
about the= feature and implementation, and therefore can give relevant tests and
write helpful descriptions. Nothing is perfect, so there are many cas= es which are
not covered or sometimes even misleading, but I thin= k ruby/spec is a very valuable
effort overall.

So when it comes to ruby.=C2=A0 I don't think test/ruby is the ideal solution for everyone.=C2=A0 I also admit it's ugly, and needs
improvements.=C2=A0 But at least I can say it's a wrong idea to
abondon it.=C2=A0 Doing so is much like to throw away unit tests and
let Selenium do everything.=C2=A0 Just nonsense.=C2=A0 Unit test is not a spec.=C2=A0 They exist separately because they have different
purposes.=C2=A0 Merging them hurts both sides.=C2=A0 The (failed) past
attempts to migrate rubyspec had problems on this point, seems to
me at least.

I do not propose to abando= n test/ruby neither to merge it with ruby/spec
as that seems unre= alistic and of limited value. But, I would like to encourage
usin= g ruby/spec instead (or in complement) of test/ruby. I think it is quite si= milar
to write ruby/specs or test/ruby tests, but I think ruby/sp= ec tests
have many advantages as described above.

<= /div>
Maybe some people dislike or do not want to write ruby/spec, and = that is fine.
But if a few MRI committers contribute more activel= y to ruby/spec,
it would already be a huge step towards testing a= nd understanding Ruby better.
The whole Ruby ecosystem would bene= fit as alternative implementations would
be able to follow change= s more efficiently and contribute back specs for additional cases.
=C2=A0
That said.=C2=A0 I also have to note that tests without specs is a
poor solution.=C2=A0 The problems pointed by @eregon in this thread is
worth listening to.=C2=A0 The company I worked before solved this
problem by hiring someone to write specs.=C2=A0 That's not doable by us (no such budget) but someone has to do that job somehow.=C2=A0 Are
we going to let the core devs do that?=C2=A0 Then I think we need to
motivate them.=C2=A0 "Spec is a good thing you should do" is a tr= ue
assertion but sounds much like a homily.=C2=A0 Instead show them it
benefits.=C2=A0 For instance, find bugs using spec and show them how
it helps their developments.

Indeed, I = found many bugs using ruby/spec.
Testing from the perspective of = trying to explain the behavior can discover many interesting cases.

I reported 70 bugs on the Ruby tracker in 7 years.
<= div>The majority of them where discovered by specs.

Here are a few examples:
* Inconsistent Float#round on Windows<= /div>
* Various difference of behavior on Windows
* Coverage = is leaking in global state
*=C2=A0ObjectSpace.each_object exposes= internal objects
* Deadlock in #autoload
* Incorrect e= ncoding for Time#zone
* Bugs in keyword arguments, there were man= y of them back then
* Various methods changed the exception they = throw in error cases

Alternative implementations a= lso have their own test suites.
They could each have a different = one and duplicate the efforts,
or each use the same test suite kn= own as ruby/spec to share the benefits.
--001a1144b5c0fa98f90545afe487-- --===============0444697578== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline Unsubscribe: --===============0444697578==--