From: Brian Candler Date: 2009-03-30T17:12:54+09:00 Subject: Re: testy.rb - ruby testing that's mad at the world Ara Howard wrote: >> assert_equal 42, ultimate.answer, "name" > > you can basically do that too, but i continually forget which is > expected and which is actual and, as you know, that's a slippery error > to track down at times. Perhaps - but it's one rule that only needs to be learned once. I notice that testy supports check , , too. Testy does (intentially) force you to name your tests, whereas Test::Unit will happily let you write check , I really don't like having to name each assertion, maybe because I'm lazy or maybe because it feels like DRY violation. I've already said what I want to compare, why say it again? > because littering the example code with esoteric testing framework > voodoo turns it into code in the testing language that does not > resemble how people might actually use the code I agree with this. This is why I absolutely prefer Test::Unit (and Shoulda on top of that) over Rspec. > i always end up writing > both samples and tests - one of the goals of testy is that, by having > a really simple interface and really simple human friendly output we > can just write examples that double as tests. Hmm, this is probably an argument *for* having a DSL for assertions - to make the assertions read as much like example code ("after running this example, you should see that A == B and C < D") Neither result.check "bar attribute", :expected => 123, :actual => res.bar nor assert_equal 123, res.bar, "bar attribute" reads particularly well here, I think. Ideally it should be as simple as possible to write these statements of expectation. How about some eval magic? expect[ "res.foo == 456", "res.bar == 123", "res.baz =~ /wibble/" ] Maybe need to pass a binding here, but you get the idea. (Before someone else points it out, this is clearly a case which LISP would be very well suited to handling - the same code to execute can also be displayed in the results) The problem here is reporting on expected versus actual, but perhaps you could split on space and report the value of the first item. expected: - res.foo == 456 - res.bar == 123 unexpected: - test: res.baz =~ /wibble/ term: res.baz value: "unexpected result" Going too far this way down this path ends up with rspec, I think. In fact, I don't really have a problem with writing res.foo.should == 456 The trouble is the hundreds of arcane variations on this. You solve this problem by only having a single test (Result#check), and indeed if rspec only had a single method (should_equal) that would be fairly clean too. However this is going to lead to awkwardness when you want to test for something other than equality: e.g. res = (foo =~ /error/) ? true : false result.check "foo should contain 'error'", :expected=>true, :actual=>res Apart from being hard to write and read, that also doesn't show you the actual value of 'foo' when the test fails. Is it worth passing the comparison method? result.check "foo should contain 'error'", foo, :=~, /error/ But again this is getting away from real ruby for the assertions, in which case it isn't much better than assert_match /error/, foo, "foo should contain 'error'" assert_match /error/, foo # lazy/DRY version > get a listing of which tests/examples i can run Yes, parseable results and test management are extremely beneficial. Those could be retro-fitted to Test::Unit though (or whatever its replacement in ruby 1.9 is called) Getting rid of the at_exit magic is also worth doing. > you can also do something like this (experimental) to just make a > simple example > > cfp:~/src/git/testy > cat a.rb > require 'testy' > > Testy.testing 'my lib' do > > test 'just an example of summing an array using inject' do > a = 1,2 > a.push 3 > sum = a.inject(0){|n,i| n += i} > end > > end Nice, could perhaps show the (expected) result inline too? test 'an example of summing and array using inject' do a = 1,2 a.push 3 sum = a.inject(0){|n,i| n += i} end.<< 6 A bit magical though. Also, we can only test the result of the entire block, whereas a more complex example will want to create multiple values and test them all. > so the goal is making it even easier to have a user play with your > tests/examples to see how they work, and even to allow simple examples > to be integrated with your test suite so you make sure you samples > still run without error too. of course you can do this with test/unit > or rspec but the output isnt' friendly in the least - not from the > perspective of a user trying to learn a library, nor is it useful to > computers because it cannot be parsed - basically it's just vomiting > stats and backtraces to the console that are hard for people to read > and hard for computers to read. surely i am not the only one that > sometimes resorts to factoring out a failing test in a separate > program because test/unit and rspec output is too messy to play nice > with instrumenting code? I agree. Part of the problem is that when one thing is wrong making 20 tests fail, all with their respective backtraces, it can be very hard to see the wood for the trees. What would be nice would be a folding-type display with perhaps one line for each failed assertion, and a [+] you can click on to get the detail for that particular one. > yeah that's on deck for sure. i *do* really like contexts with > shoulda. but still > > > cfp:/opt/local/lib/ruby/gems/1.8/gems/thoughtbot-shoulda-2.9.1 > find > lib/ -type f|xargs -n1 cat|wc -l > 3910 > > > if we accept the research and assume that bugs scale linerarly with > the # of lines of code this is not good for robustness. I disagree there - not with the research, but the implied conclusion that you should never use a large codebase. Shoulda works well, and I've not once found a bizarre behaviour in the testing framework itself that I've had to debug, so I trust it. (This is not true of other frameworks though. e.g. I spent a while tracking this one down: https://rspec.lighthouseapp.com/projects/5645-rspec/tickets/677-rspec-doesnt-play-well-with-delegateclass) > this is one > of my main gripes with current ruby testing - my current rails app has > about 1000 lines of code and 25,000 lines of testing framework! Yeah, but how many lines of Rails framework? :-) Cheers, Brian. -- Posted via http://www.ruby-forum.com/.