[ruby-core:17414] Re: Time for a release management committee?

M. Edward (Ed) Borasky wrote:
> Well ... I don't think it's so much a lack of manpower as a lack of 
> hardware for the insane amount of testing and SCM automation that 
> needs to be done. So ... suppose we limit this to everybody that can 
> run Rails. That way, we get more test suites. So that's MRI 
> 1.8.6/1.8.7, IronRuby, jRuby, and Rubinius, right? Or does KRI run 
> Rails now?
>
> What's the hardware/OS collection required? At least MacOS, Linux, 
> Windows, Solaris and one of the server-grade BSD variants. How many 
> different versions of GCC need to be tested? There's only one platform 
> that needs to be available for jRuby testing -- the latest JVM. Would 
> we need to test on anything besides x86, x86_64, and SPARC?
I think this is easier than you think.

For instance, I test AutomateIt releases against a single machine with 
10 VMs on it, ranging from XP to OS X, Ubuntu to OpenBSD.

Running the tests is pretty quick too. When I was testing proposed 
patches for the vulnerability earlier this week, I could compile Ruby, 
install the various gems, and run the test suites for RubySpec, Rails, 
and RSpec in under 3 minutes, and this will be easy to automate.

An inexpensive modern server can run dozens of virtual machines at the 
same time, each with a unique OS and environment. Jobs can be run 
on-demand using a UI that fires off ssh sessions to start tasks across 
the cluster, or the machines can execute jobs periodically or as commits 
come in. Building this won't be hard, but there are also commercial 
services and open source solutions to consider, such as Perl's 
distributed continuous integration system.

An automated testing service like this could provide a simple, yet 
effective way of catching bugs, compatibility flaws, and doing static 
analysis for vulnerabilities before releases are shipped.

Of course, for this to work, the tests need to be improved so they 
properly describe the behavior of a compliant Ruby implementation, 
provide adequate code coverage to test all paths through the code, and 
-- of course -- the developers have to pay attention to this system's 
reports.

-igal