[ruby-core:88431] [Ruby trunk Feature#14187] `make test` and `make check` to run all test suites

Issue #14187 has been updated by naruse (Yui NARUSE).


ioquatix (Samuel Williams) wrote:
> I've been working on PRs for MRI, and I got bitten by `make test` not running all tests. I wish it was the simple default, `make test` should run all tests. If users want something else, make more specific targets, e.g. `make test-core` etc.

`test-all` includes many tests may fail.
For example recent tzdata update breaks tests: https://rubyci.org/logs/rubyci.s3.amazonaws.com/centos6/ruby-2.3/log/20180810T073929Z.fail.html.gz
Many users who tries to run tests cannot understand why this fails and resolve this.
Other tests related to IPv6, /etc/hosts, firewall, and so on also depend on the environment.

If you don't know `make test-all` yet, you are not CRuby developer yet.
And now you know `test-all` and other test targets, welcome to CRuby developer.

----------------------------------------
Feature #14187: `make test` and `make check` to run all test suites
https://bugs.ruby-lang.org/issues/14187#change-73485

* Author: mame (Yusuke Endoh)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: next minor
----------------------------------------
Currently, MRI has many test suites: "`bootstraptest/`", "`KNOWNBUGS.rb`", "`basictest/`", "`test/`", and "`spec/`".  And, the target name design of `make` is really complicated:

* `make test` runs the first three suites
* `make test-all` runs only `test/`
* `make check` runs all the suites except `spec/`
* `make exam` runs all the suites

I think when a casual user build and install ruby, s/he will expect `make test` or `make check` to run all recommended test suites because many other projects uses them as the semantics.

Thus, how about changing `make test` and `make check` to run all test suites?

I think this change will affect some ruby development tools including `rubyci` (`chkbuild`), `mswin-build`, and `ruby-loco`.  @hsbt, @usa, @MSP-Greg, what do you think?

```diff
diff --git a/common.mk b/common.mk
index 80001b481c..ba5fffdf4b 100644
--- a/common.mk
+++ b/common.mk
@@ -634,10 +634,6 @@ clean-spec: PHONY
 	-$(Q) $(RM) $(RUBYSPEC_CAPIEXT)/*.$(OBJEXT) $(RUBYSPEC_CAPIEXT)/*.$(DLEXT)
 	-$(Q) $(RMDIRS) $(RUBYSPEC_CAPIEXT) 2> $(NULL) || exit 0
 
-check: main test test-testframework test-almost
-	$(ECHO) check succeeded
-check-ruby: test test-ruby
-
 fake: $(CROSS_COMPILING)-fake
 yes-fake: $(arch)-fake.rb $(RBCONFIG) PHONY
 no-fake -fake: PHONY
@@ -660,6 +656,8 @@ no-btest-ruby: PHONY
 yes-btest-ruby: prog PHONY
 	$(Q)$(exec) $(RUNRUBY) "$(srcdir)/bootstraptest/runner.rb" --ruby="$(PROGRAM) -I$(srcdir)/lib $(RUN_OPTS)" -q $(OPTS) $(TESTOPTS)
 
+test-bootstrap: btest-ruby
+
 test-basic: $(TEST_RUNNABLE)-test-basic
 no-test-basic: PHONY
 yes-test-basic: prog PHONY
@@ -677,7 +675,6 @@ yes-test-testframework: prog PHONY
 no-test-testframework: PHONY
 
 test-sample: test-basic # backward compatibility for mswin-build
-test: btest-ruby test-knownbug test-basic
 
 # $ make test-all TESTOPTS="--help" displays more detail
 # for example, make test-all TESTOPTS="-j2 -v -n test-name -- test-file-name"
@@ -755,7 +752,7 @@ $(ENC_MK): $(srcdir)/enc/make_encmake.rb $(srcdir)/enc/Makefile.in $(srcdir)/enc
 .PHONY: clean clean-ext clean-local clean-enc clean-golf clean-rdoc clean-html clean-extout
 .PHONY: distclean distclean-ext distclean-local distclean-enc distclean-golf distclean-extout
 .PHONY: realclean realclean-ext realclean-local realclean-enc realclean-golf realclean-extout
-.PHONY: check test test-all btest btest-ruby test-basic test-knownbug
+.PHONY: exam check test test-all test-bootstrap btest btest-ruby test-basic test-knownbug
 .PHONY: run runruby parse benchmark benchmark-each tbench gdb gdb-ruby
 .PHONY: update-mspec update-rubyspec test-rubyspec test-spec
 .PHONY: touch-unicode-files
@@ -1322,9 +1319,12 @@ info-arch: PHONY
 change: PHONY
 	$(BASERUBY) -C "$(srcdir)" ./tool/change_maker.rb $(CHANGES) > change.log
 
-exam: check test-spec
+test: main test-bootstrap test-knownbug test-basic test-testframework test-almost test-spec
+	$(ECHO) check succeeded
+check: test
+exam: test
 
-love: sudo-precheck up all test install exam
+love: sudo-precheck up all test install
 	@echo love is all you need
 
 great: exam
@@ -1356,11 +1356,11 @@ help: PHONY
 	"  runruby:             runs test.rb by ruby you just built" \
 	"  gdb:                 runs test.rb by miniruby under gdb" \
 	"  gdb-ruby:            runs test.rb by ruby under gdb" \
-	"  check:               equals make test test-all" \
-	"  exam:                equals make check test-spec" \
-	"  test:                ruby core tests" \
-	"  test-all:            all ruby tests [TESTOPTS=-j4 TESTS=<test files>]" \
-	"  test-spec:           run the Ruby spec suite" \
+	"  test:                ruby all test suites" \
+	"  check:               equals make test" \
+	"  exam:                equals make test" \
+	"  test-all:            run `test/` suite [TESTOPTS=-j4 TESTS=<test files>]" \
+	"  test-spec:           run the Ruby spec suite (in `spec/`)" \
 	"  test-rubyspec:       same as test-spec" \
 	"  test-bundler:        run the Bundler spec" \
 	"  test-bundled-gems:   run the test suite of bundled gems" \
```



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>