[ruby-core:39328] Re: RubySpec vs CRuby's test/...

On Mon, Sep 5, 2011 at 3:08 AM, NARUSE, Yui <[e-mail]> wrote:
> It means we must run test-all and RubySpec before each commit.

On my 5 year old machine, running all RubySpec takes 42 seconds.
That's less than `make test` and 14 times less than `make test-all`
which takes more than 10 minutes on my machine.

If someone decides he should `make test-all`, then yes, he should also
`make test-rubyspec`. This will take him 6% more time than just
running `test-all`. Is that a cost even worth mentioning?

>> Please realize that the reverse is true. I have been told repeatedly
>> to add tests to "test/". I prefer adding them to RubySpec and want to
>> continue to do so. I am against the notion that "You must add a test
>> to test-all when you change CRuby's behavior" [ruby-core:39169].
>
> You can add tests both RubySpec and test/.
> What you are doing is, you ignore our historical policy and breaks
> test-all's completeness.
>
>> I would like that we all agree on the following: to change CRuby, one
>> must add a test, either in RubySpec or in test/. If it is
>> implementation specific, it belongs in test/, otherwise it is
>> acceptable in either RubySpec or test/. I also believe it would be
>> preferable if it was added to RubySpec, but I will not fight over that
>> notion.
>
> I disagree.
> I want test-all to be complete.

Why do you want it to be complete?
Or why not just include `make RubySpec` in `make test-all`?

> Of course suggestion is welcome.
> What I only want to complain is that you stopped to add test before
> you get consensus.

Indeed, I've added tests to RubySpec instead of test/* as soon as it
was decided that CRuby would pass RubySpec. That was two years ago...

> Ah, I must say one big difference:
> CRuby test-all is regression test, so if a new failure or error
> occurs, someone must breaks CRuby.
> In other words, regression is just happened.
> # under this policy, a test which fails on CRuby won't be added
>
> RubySpec doesn't have such policy so even if failure happens, I can't
> know who adds the failure.

I'm not sure I follow. RubySpec that fail on trunk are guarded as such
by `ruby_bug`. That refers to issue #, so it should be easy to know
what it is about. If a different behavior is desired, this can
definitely be discussed.