[ruby-core:17389] Re: Release Plan: Ruby 1.9.0-2

On Jun 19, 2008, at 08:48 , Kouhei Sutou wrote:

> I make a summary of Ryan's opinion. Ryan, could you confirm
> it?

sure. being on the same page is a GoodThing(tm).

> = From the maintainer(Ryan)'s point of view
>
> == What is the Test::Unit's problem?
>
> Test::Unit is too complex to maintain.

to maintain or use/extend the internals.

> == Solution
>
> Replace Test::Unit with miniunit.
>
> miniunit is a simple, mini and clean unit testing
> framework. And miniunit has Test::Unit compatible layer.

yes

> == Benefit
>
>  * ease maintenance.
>  * fast.
>  * more features rather than Test::Unit.
>    e.g.:
>    * additional assertions.
>    * small spec implementation.
>    * small mock implementation.

yes

> == Problem
>
> None because miniunit provides Test::Unit compatible layer.
> ---

for TestCase API, yes.

> Here is my opinion against the maintainer's solution:
>
> ---
> = From a Test::Unit user(me)'s point of view
>
> == Problem
>
>  * miniunit isn't extensible.

false. it is ruby. it is just as extendable as test/unit, if not more,  
since it is cleaner.

>  * miniunit just only provides public API (*1) compatible
>    layer. It doesn't provide internal API (*2) compatible
>    layer.
>
>    (*1) TestCase#assert_*, TestCase#setup/teardown and
>         TestCase#run
>    (*2) e.g.: command line option,AutoRunner, TestCase#run
>         implementation, TestCase#add_{failure,error,...},
>         and so on

true

>  * miniunit just provides minimal features

it provides all features of test/unit's TestCase. It doesn't provide  
GUI runners. I think this is a GoodThing(tm). For those that don't  
(and I think they are VERY few and far between), we've released test/ 
unit classic as a gem.

I can imagine that providing a GUI interface to miniunit isn't that  
hard. I don't code GUIs much, but I can imagine something working much  
the same way as miniunit's tests:

     @tu = Mini::Test.new
     @output = StringIO.new("")
     Mini::Test.output = @output

So again, I don't think this is a problem.

I should probably also point out that miniunit is much better tested  
than test/unit can ever be:

lib/mini/mock.rb                                    |    31 |    27 |  
100.0%
lib/mini/spec.rb                                    |    82 |    66 |   
87.9%
lib/mini/test.rb                                    |   436 |   344 |   
96.8%

The only parts not covered are compatibility conditionals for 1.8/1.9/ 
rubinius.

> === miniunit isn't extensible
>
> Unfortunately Test::Unit doesn't keep improving itself for a
> few years. But there are many improvements in the
> world. e.g.: RSpec provides BDD style syntax, multiple
> setup/teardown (before/after) mechanism, new test (example)
> status (pending) and so on. RSpec and Mocha provides a mock
> system.
>
> Some advanced users (like ActiveSupport, Mocha and so on)
> extend Test::Unit by overriding existing methods and/or with
> Test::Unit's internal API because Test::Unit doesn't provide
> extensible interface. It causes ugly hacks.

miniunit is compatible with ActiveSupport already. Chad Fowler did two  
1-line changes to make miniunit work with flexmock. I'm sure mocha  
will take a change or two, but it won't be much.

The ugly hacks are required for test/unit, not for miniunit. Making  
miniunit work with these hacks has been ugly and painful, but it is  
something we can address with the library authors and clean up over  
time.

> miniunit doesn't provide extensible interface because it
> introduces complex mechanism. But we need it to avoid
> ugly hacks.(*) If miniunit keep simple, we will be dirty.

I disagree. I've found a number of people (besides just me) that find  
extending miniunit to be MUCH MUCH easier than test/unit. I rely on  
basic idiomatic ruby to make miniunit much more approachable. See Phil  
Hagelberg's previous email as evidence.

Here is a real world (idiomatic) example of test/unit extension vs  
miniunit:

   def assert_sorted(actual, message=nil, &block)
     expected = actual.sort(&block)
     message ||= "Expected order:\n#{expected.inspect}\nbut got order: 
\n#{actual.inspect}\n"
     assert_block(message) { expected == actual }
   end

   vs:

   def assert_sorted(actual, message=nil, &block)
     expected = actual.sort(&block)
     assert_equal expected, actual, "Order is wrong:"
   end

And that is without using the ridiculous build_message/template. I  
should also add that despite miniunit not being extensible, when you  
define assert_sorted as above, you also extend mini/spec as well:

   my_array.must_be_sorted { ... }

> I want users to be simple rather than testing framework is
> simple.

I want both to be simple. These aren't mutually exclusive.

> (*) How do we add new command line option? How do we get
>    colorized output? How do we get diff between expected
>    and actual values? Need another filter command? Need to
>    overriding existing methods? It doesn't conflict with
>    other extension?

And you think that these are addressed better in test/unit? Tell me...  
where in the files below does colorized output go? How about  
commandline options? Filtering?

% find . -name \*.rb | xargs grep -l Filter
./unit/assertions.rb
./unit/error.rb
./unit/testcase.rb
./unit/util/backtracefilter.rb
% find lib/test -name \*.rb | xargs wc -l
       14 lib/test/unit/assertionfailederror.rb
      622 lib/test/unit/assertions.rb            # HERE?
      220 lib/test/unit/autorunner.rb
      107 lib/test/unit/collector/dir.rb
       34 lib/test/unit/collector/objectspace.rb
       43 lib/test/unit/collector.rb
       56 lib/test/unit/error.rb                 # HERE?
       51 lib/test/unit/failure.rb
      160 lib/test/unit/testcase.rb              # HERE?
       80 lib/test/unit/testresult.rb
       76 lib/test/unit/testsuite.rb
      127 lib/test/unit/ui/console/testrunner.rb
      268 lib/test/unit/ui/fox/testrunner.rb
      416 lib/test/unit/ui/gtk/testrunner.rb
      465 lib/test/unit/ui/gtk2/testrunner.rb
       68 lib/test/unit/ui/testrunnermediator.rb
       46 lib/test/unit/ui/testrunnerutilities.rb
      260 lib/test/unit/ui/tk/testrunner.rb
       40 lib/test/unit/util/backtracefilter.rb # HERE?
       90 lib/test/unit/util/observable.rb
       48 lib/test/unit/util/procwrapper.rb
      280 lib/test/unit.rb
     3571 total

% find lib/mini -name \*.rb | xargs wc -l
       31 lib/mini/mock.rb
       82 lib/mini/spec.rb
      436 lib/mini/test.rb                      # ALL HERE
      549 total

> === miniunit just provides a public Test::Unit API compatible layer
>
> There are some advanced users that uses Test::Unit's private
> API as I mentioned in the above section. They are more users
> that use tools developed by some advanced users.
>
> Some advanced users need to implement their extended
> features for miniunit and Test::Unit (for backward
> compatibility) because miniunit just provides a public
> Test::Unit API compatible layer. It means that miniunit will
> be simple but tools developed by some advanced users may be
> dirty.
>
> Some or many tests may not worked. It's too uneasy
> situation. We usually don't have tests for tests.

It took me less than an hour to make things work for rails apps...  
Mocha and other gems that rely on internal API can simply bypass by  
requiring test/unit as a gem. They do NOT have to have anything  
changed for them as they have test/unit available already.

> === miniunit just provides minimal features
>
> miniunit provides some advanced features: mock system and
> BDD style syntax. But they just provides limited
> functions. e.g. Mini::Mock can't handle multi expects for
> the same name.

I don't really see how mini/spec is limited. I talked to David  
Chelimsky at railsconf only feature it is really lacking from rspec is  
nested contexts.

> Most of users prefer to useful system rather than simple
> system but limited features because they want to write their
> tests simply like Ryan wants miniunit to be simple.

again... I'm not seeing this as a problem. miniunit is simple. it also  
supports nearly every project out there already. For those that  
require test/unit internals, things are actually cleaner now. They can  
either require test/unit as a gem, or miniunit... it'll still work on  
1.8 and 1.9.

> Most of users will use RSpec rather than Mini::Spec if they
> want to use BDD style syntax.

yup. how is this a problem? This is only 82 lines of overhead.

> Most of users will use Mocha or RSpec rather than Mini::Mock
> if they want to use a mock system.

yup. how is this a problem? This is only 31 lines of overhead.

I find it odd that you're calling entire featuresets that test/unit  
doesn't bother to address at all as "minimal features". miniunit  
provides a nice middle ground for those who might want to experiment  
with BDD vs TDD. It'll let them drive both styles side-by-side. It'll  
let them pick up mocking and if they need more, move to something  
else. I have converted an m/rspec style project to mini/spec with  
simple replacements in emacs. This is a feature, not a bug.

> Yes, I think that miniunit is very good solution for Ruby
> implementation developers because it has very small
> dependencies. It doesn't use standard libraries like
> OptionParser. But users doesn't require very small
> dependencies. They want useful tools rather than very small
> dependencies.

again... I don't see small as bad. In fact, I don't really see mini/ 
test as small, so much as I see it as clean. No more junk. No more  
cruft. No more obfuscation. I've been coding in ruby since 2000. I  
shouldn't have a problem adding stuff to test/unit but I do. Why?  
Because it is a mess. Why? It isn't because it provides that much more  
than mini/unit, that's for sure.

I could certainly make miniunit smaller and provide less features/ 
flexibility... but I went for what felt right: compatibility and  
usability.

> Here is a solution against the maintainer's problem proposed
> by me:
>
> ---
> = From my point of view
>
> == Solution
>
> I maintain Test::Unit.
>
> == Benefit
>
>  * Test::Unit's API isn't broken.
>    * Test::Unit extension libraries are kept working.
>  * miniunit doesn't need to provide Test::Unit compatible
>    API. miniunit will be more simple.

Yes it does. miniunit has already taken over for rubinius.

>  * Ryan doesn't need to be burdened.

I wasn't burdened. I was terrified. :P

> == Problem
>
>  * no dramatic speed up.
>  * no dramatic simplification.
>  * not trusted maintainer.

* a PITA to extend.

> Here is my opinion about Test::Unit:
>
> ---
> = From my point of view
>
> == What is the Test::Unit's problem?
>
> Extending Test::Unit causes ugly hacks.
>
>  * Test::Unit isn't extensible.
>
>  * Test::Unit has poor features under present circumstances.
>
> The details of them are mentioned in the above sections.
>
> == Solution
>
> Extending Test::Unit to make it extensible.
>
> Sample implementation:
>  http://rubyforge.org/projects/test-unit/
>
> == Benefit
>
>  * reduce ugly hacks.
>    * some advanced users will be able to write their
>      extensions more simply.
>  * provide useful (not limited) features.
>    * users will be able to write their test more simply.
>  * keep high backward compatibility rather than miniunit.
>
> == Problem
>
>  * will not be as simple as miniunit.
>  * will not be as fast as miniunit.

* modified/clean API will still break users (since they're relying on  
those hacks) and still have the same adoption/learning curve as  
miniunit... if not worse, since it will always be more code.