[ruby-core:114467] [Ruby master Bug#14064] test-all with and without -j - incorrect assertions and missing test methods

Issue #14064 has been updated by jeremyevans0 (Jeremy Evans).

Status changed from Open to Closed

It looks like a fix for this was committed in commit:dd3851d2786412de019350a11e749c56fa5a07cc

----------------------------------------
Bug #14064: test-all with and without -j - incorrect assertions and missing test methods
https://bugs.ruby-lang.org/issues/14064#change-104240

* Author: MSP-Greg (Greg L)
* Status: Closed
* Priority: Normal
* ruby -v: ruby 2.5.0dev (2017-10-29 trunk 60531) [x64-mingw32]
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN
----------------------------------------
I always run test-all with a `-v` parameter, and although both Travis & Appveyor scripts do not, many of the https://rubyci.org builds seem to.  With `-v`, one has a log of all test methods.

For quite a while, I ran ruby-loco (MinGW trunk build) with `test-all` having no `-j` parameter.  I've now changed to using `-j`, and I'm surprised at what I'm finding.

When run without a `-j` parameter, the total test count is correct, but the total assertions count is high.

When run with a `-j` parameter (even set to 1), the test count drops (along with tests not appearing in the log), and the total assertions count is correct.  Often, whole test files (classes) disappear from the log.

Re the test count variance, I wrote code to parse the logs (the total test count variance), which showed which test methods were present in non-parallel test logs that were missing in parallel test logs.

As to the assertions, I ran test-all on a few single files, and added the assertion count to each test method line, then compared with and without `-j`.  Finally, by checking individual methods for their respective number of assertions, it was clear that the numbers shown in a non-parallel run were high.  At first, it seemed that they were high by a count of three, but that only held for a few files.

First of all, I'm wondering if anyone else has found similar results.

Secondly, I've spent some time verifying this, but I can't seem to find where the code needs corrections to function properly.  I hoped it might be a timeout issue, but that doesn't seem to be the case.  Hence, suggestions as to where to start are welcome.

Finally, in Travis, the line for test-all is:

```
make -s $JOBS test-all -o exts TESTOPTS='-q --color=never --job-status=normal'
```

Using MinGW, adding $JOBS as a make parameter seems to have no effect.




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- [e-mail]
 To unsubscribe send an email to [e-mail]
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/