[ruby-core:68250] Re: What to do with rubyspec errors (was: Re: Re: Future of test suites for Ruby)

Martin,

A good place to start is to determine which spec file is causing the
problem.  Mspec options can be given to test-rubyspec like this...

make test-rubyspec MSPECOPT=3D"--verbose"

This option will output the name of the spec file before it is ran.  This
should reveal which spec files are producing the F (failure), E (error) and
the eventual lock up.  Once a problematic file is identified then that
single spec file can be ran like this...

make test-rubyspec MSPECOPT=3D"core/marshal/dump_spec.rb"

For the specs that produce F and E output the details of those failures and
errors should be displayed.  For the spec that is locking up, this option
should give the details up until the lockup and should help find the
problem...

make test-rubyspec MSPECOPT=3D"--format spec core/marshal/dump_spec.rb"

Once the problems are identified, it may be more clear where the issues
should be submitted.

I hope this is helpful,

Anthony

On Mon Feb 23 2015 at 4:17:33 AM "Martin J. D=C3=BCrst" <[e-mail].a=
c.jp>
wrote:

> Hello Anthony,
>
> Maybe you can help me. I'm frequently compiling CRuby trunk on cygwin,
> and occasionally tried to run rubyspec (make test-rubyspec). However, I
> always get the same result: Some lines of textual output, then about 7
> lines of mostly dots with a few Fs and an occasional E or two, and then
> the thing hangs with a busy ruby executable, which I have to kill via
> the task manager (Ctrl-C doesn't help).
>
> I'd like to be able to submit bug reports, either for the spec or for
> the implementation, but I don't know what's wrong where I get Fs, and I
> don't know which test produced an endless loop. I didn't get any help
> from http://rubyspec.org/mspec/, but maybe that's the wrong place to look=
.
>
> Regards,   Martin.
>
> On 2015/02/17 22:16, Anthony Crumley wrote:
> > Beniot,
> >
> > I have been working on reconciling RubySpec with the 2.x MRI versions
> over
> > the last month.  https://github.com/anthonycrumley/rubyspec/
> commits/master
> >
> > My intention is to:
> > 1) Get RubySpec updated to run with all the 2.x versions of MRI.
> > 2) Get the nurse/rubyspec updates since the fork added.
> > 3) Get the updated RubySpecs into the MRI CI.
> > 4) Hopefully find that repo a home at either rubyspec/rubyspec or
> > ruby/rubyspec.
> >
> > I agree with you that the RubySpec tests are very valuable and would LO=
VE
> > to work with you on them.
> >
> > Anthony
> >
> > On Tue Feb 17 2015 at 6:59:48 AM Benoit Daloze <[e-mail]>
> wrote:
> >
> >> On 17 February 2015 at 13:32, Benoit Daloze <[e-mail]> wrote=
:
> >>
> >>> - The second step is really to choose a canonical RubySpec repository=
,
> to
> >>> avoid "death by too much forks".
> >>> This repository should only contain RubySpec tests for practical
> reasons.
> >>> We should allow many specs contributors to take part in merging chang=
es
> >>> and maintaining specs.
> >>> I think this was a fatal flaw of rubyspec/rubyspec in that too few
> people
> >>> had the large burden of merging and maintaining the specs.
> >>>
> >>> The main existing repository I see today is nurse/rubyspec.
> >>> I am thinking the process could be similar to handling pull requests =
on
> >>> ruby/ruby in that some contributors would provide feedback and merge
> them.
> >>> The CI is very useful in this regard to ensure MRI is not broken
> >>> inadvertently.
> >>>
> >>
> >> I think it would make sense in that case to move nurse/rubyspec to
> >> ruby/rubyspec for clarity.
> >>
> >
>