From: Gregory Brown Date: 2009-06-16T04:39:17+09:00 Subject: [ANN] Starting work on Unity, a web-based RubySpec reporting system Unity is a web based reporting tool for searching and analyzing RubySpec across Ruby versions and implementations. It aims to make it easier to find details about the specs themselves, as well as their pass/fail/error stats. What follows is a description of the goals of this project, a call for community support, and a rough roadmap for what we hope to accomplish. Those who don't read long emails but still like this idea are welcome to go straight to pledgie and drop in a donation: http://pledgie.com/campaigns/4640 For those who need more convincing, read on... == Main Goals In an interview[0] for the Ruby Best Practices blog, Brian Ford discussed his ideal goals for the RubySpec project. With Unity, we hope to do our part to address most of them. Here are the goals Brian listed, followed by the ways we hope to support them in Unity: 1. Provide a reasonably precise definition of the Ruby programming language so that there can be a certain degree of uniformity among the implementations. Unity will help make it easy for you to know what RubySpec defines Ruby as. By just typing in the name of any Ruby method, Unity will provide you with the descriptions of the examples used in RubySpec for that method. It will also provide you with the test results across Ruby language and implementation versions, so you can see just how uniform things really are (or aren't). 2. Provide quality assurance and regression notification for MRI and all implementations. Unity will provide failure reports for every release of every implementation we support. We will also make it easy for users to find the right place to report unexpected failures. Further support for advanced reporting and alerts may be planned later. 3. Improving the quality of Ruby as a language by discovering dusty corners and inconsistencies. While we won't be able to directly address this issue (at least at first), having an easier way to look over how Ruby is defined behaviorally will surely make it easier for community members to come up with ideas on how to clean things up. 4. Experimentation with implementations and the Ruby language itself. Unity will make it relatively clear which versions of which implementations support a certain feature. This will help make Ruby users and library maintainers more likely to try new implementations or versions of the Ruby language, as it'll give them a clear sense of what to expect. While this can be accomplished with RubySpec alone, Unity will surely lower the barrier to entry and eliminate some variables. 5. A learning tool that provides Ruby programmers with concrete examples of code that demonstrates every single Ruby language feature. Unity will provide links or inline source listings for all examples within RubySpec. This should be as easy as clicking on an example description within a method's report listing. In summary, Unity does not so much have any *new* goals, but instead is an additional tool for advancing the goals of the RubySpec project. == Project Roadmap Madriska Media Group[1] is partially sponsoring this project, by covering some of my developmentment time ( a few hours a week, likely ), donating some of their own development time, and providing some necessary resources. However, I'd like to put this project on an accelerated schedule so that I can launch something by the end of the summer. The easiest way for me to do that is to ask for just a little bit of extra support from the community. If you think these goals are worthwhile, consider making a donation so that I can spend some additional dedicated time on this project. http://pledgie.com/campaigns/4640 I will be working on the project no matter what, but if I can guarantee an extra 10 hours a week of dev time, it's much more likely that I'll have something useful launched within the next couple months. What follows is a bit more detail of what specifically we've already done, and what we hope to do in the coming weeks. == What we've done so far A simple Rails 2.3, Ruby 1.9 based prototype that can model spec output data and report against it. For demo purposes, we have loaded data from MRI Ruby 1.8.6p369 and YARV Ruby 1.9.1p129 Screenshot: http://is.gd/12Cu2 Code: http://github.com/madriska/unity/tree/master > Features: * can look up spec descriptions and results by method descriptor (ala RI) e.g. Object#an_instance_method, Object.a_class_method * displays example descriptions along with the Ruby versions they run on. * displays pass-fail data for the implementations that it has data for * data is loaded into database via an importer which reads from real mspec output * internally able to model testing environments based on both ruby version and ruby implmentation version * currently reports results based on ruby version and implementation (without release version) > Limitations: * while full spec output is stored, including backtraces for errors, it is not currently reported on * does not support Class::method syntax for class methods (use Class.method instead) * does not support Nested::Modules yet * relies on patch to mspec at: http://github.com/madriska/mspec/commit/36b8bce62c272d546bcbd8ab9f136576ff4701e5 * very limited testing so far (Array specs on MRI 1.8.6 and YARV 1.9.1 only) * mspec loading can be brittle -- no recourse if spec execution hangs. == What needs to be done In addition to the limitations discussed above, we would like to do the following: Core functionality: - Patch back to mspec where needed - Improve reporting output to more clearly see descriptions organized by which Ruby versions they are relevant to - Add filters to prune out specific ruby versions / implementations from a given report - Provide per-class reports so that it is easy to navigate to a particular method, as well as see pass/fail/error data aggregated - Provide per-release reports of P/F/E data for all the Ruby implementations we support, faceted by class/module or method. - Expose a service interface for importing mspec data, to decouple web app from toolchain Additional Features We Want: - Provide a way to automate collection of report data across the Ruby implementations we support. (Communicate over service API) - Generate a 'state of Ruby' comprehensive report for a given implementation release and ruby version. - Provide some comparative analysis between Ruby versions - Link back (or list inline) spec source text for each example. == When Will This Be Ready? We can't make any promises, but our goal is to have a preliminary site launched by July 15th, 2009. Shortly after that, we'll post a revised roadmap that describes what we hope to call 1.0, which we hope to have ready by August 15th, 2009. While we won't be accepting patches or formal feature requests between now and July 15th, we welcome you to discuss this with us on our google group, and will definitely be interested in code contributions after mid July. http://groups.google.com/group/ruby-unity Anyway, we look forward to having something more to show you soon! - Greg & Brad [0] http://blog.rubybestpractices.com/posts/gregory/006-brian-ford-rubyspec.html [1] http://madriska.com