From: Charles Oliver Nutter Date: 2011-09-09T11:03:11+09:00 Subject: [ruby-core:39402] Re: RubySpec vs CRuby's test/... On Thu, Sep 8, 2011 at 5:40 PM, SASADA Koichi wrote: > Thank you for summary of needs. ��Following is problems I feel (and > discussed on IRC). > > Problems: > > (1) Most of MRI developers (is it true?) can't change RubySpec. ��Also > most of MRI developers don't know RubySpec rules. I don't think there would be a problem getting MRI developers access to RubySpec for direct commits. I understand the issues with "rules" though. Personally I'd be happy to see *any* specs getting in, even if they require some cleanup after the fact. It would be better than nothing. > (2) We can add code modification and test cases in one commit. ��If we > can't do it, we need commit commit modifications and RubySpec isolation. This is an issue, but a small one. You point out later that you could just keep a copy of RubySpec in MRI's repository and migrate changes over. That's not a difficult task, if it's done periodically. This is how Rubinius uses RubySpec. For JRuby, we chose not to keep our own copy of RubySpec. Instead, our build will fetch a "known stable" revision of RubySpec as part of the build and run against that. We make all our RubySpec commits directly to RubySpec. > (3) It is difficult to judge test is an independent or not. > Additionally, we don't need to insert "guard" about version or > implementation on our "test/*". ��It seems tough work to insert correct > "guard". That's certainly true. I wouldn't expect that all tests would immediately start going to the "right place". I believe the frustration I and others feel mostly stems from the fact that there's almost no specs coming from Ruby core now, which means those of us *not* in ruby-core have to reverse-engineer behavior changes and write specs ourselves. It's a dismal situation, and any improvement would be welcome. > (4) Current RubySpec is not portable (especially Windows) > [ruby-core:39379]. This is just from lack of maintenance. As early as last year the whole suite ran green on Windows. We can do it again. > (5) As you know, trunk is not unstable. ��For example, some modifications > are revertd soon. ��Should we add such features as "Specifications"? ��(I > know that there is an opinion that the experimental code should have a > spec). If MRI maintained a copy of RubySpec in the MRI repository, then it would be easy to revert spec changes as well. Maybe this would be a good guide (edits welcome): * If it is a new experimental feature, tests/specs (if they exist) could certainly live only in MRI's repository. If that feature becomes "official" those tests should be migrated to RubySpec. * If an existing feature changes, the associated specs should change too. * Anything that's MRI-specific or which Matz deems "implementation specific" should only be tested by MRI's own suite. > (6) test/* contains many many corner cases depend on MRI implementation > to increase code coverage (Thanks Endo-san). ��It is independent as Ruby > language, but dependent on MRI. ��Where should we write such tests? Those all stay in MRI's suite. There will never be a clear line between what's implementation/MRI-specific and what is true specified behavior since there's no specification (or rather, the only specification that exists encompasses only a small subset of Ruby). Nobody expects ruby-core to be perfect here. But when a commit includes visible behavioral changes that other implementations will eventually want to duplicate, we *must* have a test/spec somewhere independent of MRI's tests. > I think this problem is "who pay efforts on it?". > > One solution is continuing current style (using test/*) and someone > migrate tests to RubySpec if needed. ��But we need "someone". In the past, there have been many contributors migrating tests (and reverse-engineering MRI to write new ones) by hand. That works, if we can keep up such efforts and keep finding new people as old ones leave. Migrating tests is a thankless job, so perhaps we need to thank people more often :) One way to start this process would be to start at "a" in the existing test/* tests and just start moving bits to RubySpec. I've done this for some MRI tests in the past and for many JRuby tests too. It's a slow process, but it feels great to delete the old test in favor of a new spec. > Other solution is MRI committer join to maintain RubySpec. ��MRI > developer (including me) should change development style and several > overhead to do (I think inserting correct "guard" is difficult). Guarding isn't too bad. The main guards that ruby-core would need to know are: ruby_version_is/is_not - for specifying a range of Ruby versions, as Strings, where the behavior is expected to exist platform_is/is_not - for specifying platform-specific behaviors (e.g. :windows). There are more complex forms in mspec, but we can help teach them to ruby-core as we get going. I think ruby-core would only be responsible for making sure specs pass on the Ruby versions that RubySpec covers: 1.8.7 and 1.9.2. 1.9.3 changes are probably not covered in specs currently. > Brian Ford told me that we can have our own RubySpec repository in MRI > repository and migrating updating specs from MRI's RubySpec repository > to the RubySpec repository is easy. ��It is also one solution. ��However, > we need "someone" who can migrate. ��It solves (1), (2) and maybe (3). For 1.9.2, I believe yugui took the lead on making sure specs were written for 1.9.2 behaviors (as much as possible...time was limited) and also ensuring that 1.9.2 passed all specs (with appropriate version guards, etc). Could it be release manager's responsibility to ensure specs get written or migrated, perhaps by asking the developers making changes to write/migrate those specs? Those of us outside ruby-core really, really want to help make this happen. Let me know what I personally can do to help. - Charlie