From: Yossef Mendelssohn <ymendel@...>
Date: 2009-03-31T03:17:56+09:00
Subject: Re: testy.rb - ruby testing that's mad at the world

Quite a bit of discussion since I looked at this last. It's almost as
if people care about testing.

On Mar 28, 11:27 pm, "ara.t.howard" <ara.t.how...@gmail.com> wrote:
> i'm open to suggestion on format though.  requirements are
>
>         . readable by humans
>          . easily parsed by computers
>
> basically that means some yaml format.  honestly open to suggestion  
> here...

Agreed on YAML. That's just a really simple way to go if you're going
to stick to those two (very sensible) requirements. Maybe JSON would
work as well. I think the trouble I had with the specific example is
that it's listed as a failure, yet shows 'a' matching. Maybe instead
of

    returning unexpected results:
      failure:
        expect:
          a: 42
          b: forty-two
        actual:
          a: 42
          b: 42.0

you could have something like

    returning unexpected results:
      status: failure
      vars:
        matched:
        - a
        unmatched:
          expect:
            b: forty-two
          actual:
            b: 42.0

It doesn't have to be exactly like that, of course, but I'd call
attention to specifically the status being something you can easily
check by using the 'status' key instead of looking for 'failure'. Also
that the variables are split into good and bad.

The status thing, admittedly, is more of a point if you're going to
have YAML reports of tests that passed as well. One of the things I
like about RSpec and bacon is the specdoc format, so if you have
sensibly-named contexts and examples, you can get a document
explaining the behavior of the code under test. In fact, I added this
output format to shoulda in the pre-git hullabaloo. I don't think it
ever made it in.

> i have major issues with points two and three wrst to most ruby  
> testing frameworks.  one of the main points of testy is to combine  
> examples with testing.  rspec and all the others do not serve as  
> examples unless you are a ruby master.  that is to say they introduce  
> too many additions to the code that's supposed to be an example to  
> really preserve it's 'exampleness'.  and of course the output is  
> utterly useless to normal humans.  if a framework provides 1000  
> asset_xxxxxxxx methods ad nausea then the point of the code - it's  
> level of example-good-ness - is lost to mere mortals

The examples I tend to give are just very high-level overviews, mostly
showing the API and drawing the reader into other documentation, the
specs, or finally the code if they really want to see everything my
lib/module/gem/whathaveyou can do. As such, they're usually not very
helpful as far as specs go.

I agree that having a glut of assertions/checks isn't useful, and is
fact hiding a very important fact when it comes to BDD (or really
testing in general): If something is painful to test, it should be
changed. What I love about BDD is having that something using my code
and defining its API as I go, so I don't get stuck in nasty test-land
with shitty implementation-specific tests, or at least not as easily.
So that pain, like any pain, is a signal that something is wrong, and
it should be fixed. It shouldn't be hidden behind a special assertion
or matcher or any other rug-sweeping activity unless there's good
reason, like the code that's hurting you to test isn't available to
you to fix.

> > I used RSpec for a long time, and still do with some projects. I've
> > switched bacon for my personal projects, and I love it. As for
> > mocking, which is necessary in some cases if you want to test without
> > turning into a serial killer, mocha with RSpec, facon with bacon.
>
> this will summarize where my thoughts are on that
>
> cfp:~/redfission > find vendor/gems/{faker,mocha,thoughtbot}* -type f|
> xargs -n1 cat|wc -l
>     24255
>
> cfp:~/redfission > find app -type f|xargs -n1 cat|wc -l
>      1828
>
> rspec and co might be fine but seriously, the above is insane right?

It's bordering on nuts. It'd be better to show LoC than just wc -l, of
course, and are all the thoughtbot gems for your tests? Also, someone
else mentioned flay/flog/reek output, which could be illuminating. And
someone also brought up the app framework. After all, what you're
doing here is comparing your app code to your test framework. Seems
like you should be comparing app to test, or app framework to test
framework, or app to test with their respective frameworks. (And don't
forget to include Test::Unit for the testing frameworks that are
simply a layer on top of that.)

And I'd like to once again bring up bacon. I say I live with a few
methods on Object and Kernel to be able to test without going crazy,
and I mean it. I don't care how much how much you say assert_equal
makes sense or result.check 'some name', :expect => x, :actual => y is
great, I'm never going to get confused about some_value.should == 5.
It's readable, and it's close to English. Where I think bacon wins out
huge over RSpec is that bacon keeps all those methods on the Should
object instead of making them global. It's a small change to syntax
(some_obj.should.respond_to(:meth) vs. some_obj.should respond_to
(:meth)), but it makes a world of difference as far as sensibility
goes.

Oh, and bacon is < 300 LoC, with facon at 365 (a line of code for
every day in the year!).

--
-yossef