From: Vladimir Sizikov <vsizikov@...>
Date: 2010-02-02T22:18:03+09:00
Subject: [ruby-core:28010] Re: [Bug:trunk] some behavior changes of 	lib/csv.rb between 1.8 and 1.9

Hi,

On Tue, Feb 2, 2010 at 1:17 PM, Yusuke ENDOH <mame@tsg.ne.jp> wrote:
> 2010/2/1 James Edward Gray II <james@graysoftinc.com>:
>> I say that meaning that CSV has a lot of tests in Ruby itself. �Does RubySpec not make an effort to port existing test suites?

I'd say that a lot of tests in Ruby are very good candidates for
porting to RubySpec. But the devil is always in details: which tests
are good RubySpec ones and which are not.

> Here is just my personal opinion: �RubySpec is (at least, aim to be)
> "spec", not test suite.
>
> The library "spec" can be referred to know the standard, strict and
> guaranteed behavior of library. �In the sense, it is similar to
> document, rather than test suites.
> "The code is the spec" philosophy is too cumbersome for the purpose.
> It is difficult to know what behavior is guaranteed. �And, some
> optimization and refactoring may lead to spec change easily.

I have a bit different view here.

Personally, I'd consider anything (reasonable) that any
external/public Ruby library or any other 3rd party Ruby code expects
from the Ruby implementation to be a RubySpec worthy material.
Otherwise, those libs/application could be working differently on
different implementations, which is always not good. :)

At a minimum, Rubyspec should have tests for public API behavior, including:
* Regular use cases and boundaries
  (ideally, with Equivalence Class Partitioning
http://www.testing-world.com/58828/Equivalence-Class-Partitioning)
* Invalid/exceptional use cases (how API/impl react to those
invalid/exceptional situations)
* All the assertions explicitly stated in the public/official
documentation (all *testable* assertions, I mean).

What are the not-so-good candidates for RubySpec:
* white-box tests
* tests for private API, and for internal implementation-details
* most of regression tests (they are implementation specific, typically)
* heavily platfrom-dependent or platfrom-specific behaviors (very hard
to maintain such tests and keep the sanity)
* performance/stress tests

For tests that are in Ruby repository, while there are lots and lots
of good tests that are very suitable for RubySpecs, they often mixed
among other ones which are not ideal candidates, so it could take
quite an effort to detect which ones belong to which category and to
port only them (and to check that RubySpec doesn't have similar tests
already). Big task.

In JRuby, we tend to encourage to write RubySpec tests first, and only
if there is a good reason why such test is not good for RubySpec, only
then we end up with jruby-specific tests. We also use various
3rd-party test suites, which we also trying to move to RubySpec, where
appropriate, to reduce the number of such 3rd-party suites and to
simplify test maintenance.

Thanks,
  --Vladimir

> In addition, test suites may include "white-box test," which is
> involved with the specific implementation.
> For example, when yet another CSV library appears (like FasterCSV
> for old CSV) and aims to be strictly compatibile to lib/csv.rb,
> the test suites may be too implementation-specific to be used as
> comformance test.
>
>
> Well, in [ruby-core:27930], to be exact, I had to ask you "are these
> new behaviors guaranteed (at least in 1.9 series)?"
>
> --
> Yusuke ENDOH <mame@tsg.ne.jp>
>
>