From: Austin Ziegler Date: 2006-09-15T00:18:31+09:00 Subject: Re: Metaruby, BFTS, Cardinal and Rubicon - State of play? On 9/14/06, M. Edward (Ed) Borasky wrote: > Austin Ziegler wrote: >>> I'm even happier that it's drawing some attention to the various >>> Ruby implementations and the growing toolkit around them. >> I'd be happier if Mr Tew didn't try to lend legitimacy to the alioth >> shootout. Microbenchmarks don't show anything useful, even if they're >> run correctly -- which the shootout has never been run correctly. It >> isn't even administered correctly. (I was similarly annoyed that Joel >> Spolsky used it in his latest slam on Ruby. Stupid, Joel, stupid.) > Well ... as a working performance engineer, I'm going to defend > microbenchmarks as virtually (no pun intended) the *only* way to > improve performance over all for the Ruby interpreter, coupled of > course with profiling said interpreter and careful design of the data > structures the interpreter must maintain during execution. Benchmarking for internal purposes is fine. What the shootout does is something different entirely. Have you ever really *looked* at the code they run for the various different versions? Some of it is so blatantly tweaked to run faster on the benchmark that it's not funny. (There's a Perl example I looked at a couple of years ago that *deliberately* had obfuscated code because the obfuscated code took advantage of internals that you're not supposed to use and ran faster than the other versions.) There's no excuse for that sort of thing showing up on a benchmarking site. It's no different than NVidia or ATI detecting a benchmark program and optimizing certain things for that program only. It gets worse, Ed: the administrators behind the shootout don't care. They never have. They continually promote their website, but when challenged on the methodology used or technical issues, they give the quote that television psychics use: for entertainment purposes only. They're dishonest and run a benchmark comparison site that is so flawed that you can't even remotely trust it. The saying goes "lies, damned lies, and statistics". Well, any published benchmark is even worse than statistics in that line. I'm *not* against the concept of benchmarking. I'm definitely against the concept of comparative benchmarking in the way that the shootout does it. I will often benchmark the code that I write to make sure that I'm writing efficient code. But I won't pretend that the results are useful for comparisons. They never are. -austin -- Austin Ziegler * halostatue@gmail.com * http://www.halostatue.ca/ * austin@halostatue.ca * http://www.halostatue.ca/feed/ * austin@zieglers.ca