[ruby-core:17346] Re: Release Plan: Ruby 1.9.0-2

Hi,

First, it's OK for me that replacing Test::Unit with
miniunit if miniunit provides high extensibility and will be
well maintained. (e.g. accept new extension API request if
it's needed) But it seems that high extensibility conflicts
with simple implementation.


I make a summary of Ryan's opinion. Ryan, could you confirm
it?

---
= From the maintainer(Ryan)'s point of view

== What is the Test::Unit's problem?

Test::Unit is too complex to maintain.

== Solution

Replace Test::Unit with miniunit.

miniunit is a simple, mini and clean unit testing
framework. And miniunit has Test::Unit compatible layer.

== Benefit

  * ease maintenance.
  * fast.
  * more features rather than Test::Unit.
    e.g.:
    * additional assertions.
    * small spec implementation.
    * small mock implementation.

== Problem

None because miniunit provides Test::Unit compatible layer.
---


Here is my opinion against the maintainer's solution:

---
= From a Test::Unit user(me)'s point of view

== Problem

  * miniunit isn't extensible.

  * miniunit just only provides public API (*1) compatible
    layer. It doesn't provide internal API (*2) compatible
    layer.

    (*1) TestCase#assert_*, TestCase#setup/teardown and
         TestCase#run
    (*2) e.g.: command line option,AutoRunner, TestCase#run
         implementation, TestCase#add_{failure,error,...},
         and so on

  * miniunit just provides minimal features

=== miniunit isn't extensible

Unfortunately Test::Unit doesn't keep improving itself for a
few years. But there are many improvements in the
world. e.g.: RSpec provides BDD style syntax, multiple
setup/teardown (before/after) mechanism, new test (example)
status (pending) and so on. RSpec and Mocha provides a mock
system.

Some advanced users (like ActiveSupport, Mocha and so on)
extend Test::Unit by overriding existing methods and/or with
Test::Unit's internal API because Test::Unit doesn't provide
extensible interface. It causes ugly hacks.

miniunit doesn't provide extensible interface because it
introduces complex mechanism. But we need it to avoid
ugly hacks.(*) If miniunit keep simple, we will be dirty.

I want users to be simple rather than testing framework is
simple.

(*) How do we add new command line option? How do we get
    colorized output? How do we get diff between expected
    and actual values? Need another filter command? Need to
    overriding existing methods? It doesn't conflict with
    other extension?

=== miniunit just provides a public Test::Unit API compatible layer

There are some advanced users that uses Test::Unit's private
API as I mentioned in the above section. They are more users
that use tools developed by some advanced users.

Some advanced users need to implement their extended
features for miniunit and Test::Unit (for backward
compatibility) because miniunit just provides a public
Test::Unit API compatible layer. It means that miniunit will
be simple but tools developed by some advanced users may be
dirty.

Some or many tests may not worked. It's too uneasy
situation. We usually don't have tests for tests.

=== miniunit just provides minimal features

miniunit provides some advanced features: mock system and
BDD style syntax. But they just provides limited
functions. e.g. Mini::Mock can't handle multi expects for
the same name.

Most of users prefer to useful system rather than simple
system but limited features because they want to write their
tests simply like Ryan wants miniunit to be simple.

Most of users will use RSpec rather than Mini::Spec if they
want to use BDD style syntax.

Most of users will use Mocha or RSpec rather than Mini::Mock
if they want to use a mock system.


Yes, I think that miniunit is very good solution for Ruby
implementation developers because it has very small
dependencies. It doesn't use standard libraries like
OptionParser. But users doesn't require very small
dependencies. They want useful tools rather than very small
dependencies.
---


Here is a solution against the maintainer's problem proposed
by me:

---
= From my point of view

== Solution

I maintain Test::Unit.

== Benefit

  * Test::Unit's API isn't broken.
    * Test::Unit extension libraries are kept working.
  * miniunit doesn't need to provide Test::Unit compatible
    API. miniunit will be more simple.
  * Ryan doesn't need to be burdened.

== Problem

  * no dramatic speed up.
  * no dramatic simplification.
  * not trusted maintainer.
---


Here is my opinion about Test::Unit:

---
= From my point of view

== What is the Test::Unit's problem?

Extending Test::Unit causes ugly hacks.

  * Test::Unit isn't extensible.

  * Test::Unit has poor features under present circumstances.

The details of them are mentioned in the above sections.

== Solution

Extending Test::Unit to make it extensible.

Sample implementation:
  http://rubyforge.org/projects/test-unit/

== Benefit

  * reduce ugly hacks.
    * some advanced users will be able to write their
      extensions more simply.
  * provide useful (not limited) features.
    * users will be able to write their test more simply.
  * keep high backward compatibility rather than miniunit.

== Problem

  * will not be as simple as miniunit.
  * will not be as fast as miniunit.
---


In <[e-mail]>
  "[ruby-core:17312] Re: Release Plan: Ruby 1.9.0-2" on Wed, 18 Jun 2008 13:36:19 +0900,
  Ryan Davis <[e-mail]> wrote:

> I support -n and -v. --name is too much typing.

I prefer to short option name on command line. But I prefer
to long option name in script because it's self-describing.

>              I would also love to see more people extending it through  
> gems and have designed it specifically to make extending it easy...  
> something I found incredibly hard to do with test/unit (and why I  
> wrote miniunit to begin with).

In my point of view, miniunit still less
extensibility. e.g. I want to add a feature that C-c
interrupt running test and show result but it seems that I
need to do ugly hack.


> I've spent less than an hour (ie, a small and reasonable amount of  
> time) running 1.9 tests with both test/unit and miniunit and got the  
> results within a reasonable level of sameness. Unfortunately, ruby  
> testing on OSX seems to be very inconsistent, so I can't tell what's  
> because of miniunit (if anything) and what isn't.
> 
> 505 % tail -2 tests.before.txt tests.after.txt
> ==> tests.before.txt <==
> 4650 tests, 1659634 assertions, 37 failures, 79 errors
> ==> tests.after.txt <==
> 4610 tests, 1689855 assertions, 49 failures, 134 errors
> 
> Given this, is there any reason NOT to switch?

Yes.
It seems that replacing in the status causes some
confusions. What is wrong? Test? Implementation? Testing
system?

What about the following work flow if we replace Test::Unit
with miniunit?

  * all tests are passed with Test::Unit
  * replace Test::Unit with miniunit without implementation
    and test changes
  * confirm all tests are still passed
    * miniunit may need to more Test::Unit compatibility
      features temporarily
  * migrate to miniunit API?

> As an aside, I'm don't understand why you're questioning switching 1.9  
> to miniunit. I thought I detailed all of this to you back when you  
> expressed interest in working on test/unit (feb/mar?). None of this  
> should be new (except for specs and mocks).

I'm sorry... It seems that I've misunderstood...
I understood that we do the following work flow:
  * release the current Test::Unit as a gem for backward
    compatibility (done)
  * extend Test::Unit
  * release it as a gem for preview
  * update Test::Unit in the Ruby repository to the extended
    Test::Unit


Thanks,
--
kou