[ruby-core:111008] [Ruby master Misc#19149] Minimal covered tests with the --enable-yjit case?

Issue #19149 has been updated by jaruga (Jun Aruga).


maximecb (Maxime Chevalier-Boisvert) wrote in #note-1:
> If you want to thoroughly test everything, you could run make check two times, once with YJIT, and once without.
> 
> If you just run:
> ```
> $ make check RUN_OPTS="--yjit"
> ```
> 
> This will run YJIT with its default options, which is what most people would use when running YJIT. If you specify `--yjit-call-threshold=1 --yjit-verify-ctx`, it will make YJIT compile every single method and run extra verifications. This is what we do on the CI to be extra thorough, but it will be slower.
> 
> If you just want to quickly verify that YJIT works on the host platform, you could just run `make btest` with `--yjit` for some quick minimal verification. We do already test YJIT pretty well on the GitHub CI and on `rubyci.org`.

Thank you for clarifying it. I understood it. I am still trying to find a good balance of running the YJIT tests between the running time and the coverage.

In the case of "MJIT", while the MJIT is supported as a default. And we don't run the tests with RUN_OPTS="--mjit" in the Feodra Ruby's RPM recipe file, while I notice there are tests with `RUN_OPTS="--mjit"` now at <https://github.com/ruby/ruby/blob/4ab89d57bbc569143e9833addb88b91db86ad057/.github/workflows/mjit.yml#L35>.

Because we can run the the following MJIT tests within the `make check` without `RUN_OPTS="--mjit"`. Here are the test results on the yesterday's latest master `66e5200ba435361624caa3e23db7962d906b70db`.

```
$ grep MJIT_SUPPORT rbconfig.rb 
  CONFIG["MJIT_SUPPORT"] = "yes"

$ make test-all TESTS=test/ruby/test_mjit.rb
revision.h updated
Run options: 
  --seed=16704
  "--ruby=./miniruby -I./lib -I. -I.ext/common  ./tool/runruby.rb --extout=.ext  -- --disable-gems"
  --excludes-dir=./test/excludes
  --name=!/memory_leak/

# Running tests:

Finished tests in 132.081114s, 0.7950 tests/s, 4.2020 assertions/s.
105 tests, 555 assertions, 0 failures, 0 errors, 4 skips

ruby -v: ruby 3.2.0dev (2022-11-24T06:13:00Z master 66e5200ba4) [x86_64-linux]
```

However, we cannot run the following YJIT tests without `RUN_OPTS="--yjit"`. The tests were skipped.

```
$ grep YJIT_SUPPORT rbconfig.rb 
  CONFIG["YJIT_SUPPORT"] = "dev"

$ make test-all TESTS=test/ruby/test_yjit.rb
revision.h updated
Run options: 
  --seed=38677
  "--ruby=./miniruby -I./lib -I. -I.ext/common  ./tool/runruby.rb --extout=.ext  -- --disable-gems"
  --excludes-dir=./test/excludes
  --name=!/memory_leak/

# Running tests:

Finished tests in 0.017483s, 0.0000 tests/s, 0.0000 assertions/s.
0 tests, 0 assertions, 0 failures, 0 errors, 0 skips

ruby -v: ruby 3.2.0dev (2022-11-24T06:13:00Z master 66e5200ba4) [x86_64-linux]
```

What does the difference of the ideas come from?


----------------------------------------
Misc #19149: Minimal covered tests with the --enable-yjit case?
https://bugs.ruby-lang.org/issues/19149#change-100258

* Author: jaruga (Jun Aruga)
* Status: Open
* Priority: Normal
----------------------------------------
In the [Fedora Ruby's RPM recipe file](https://src.fedoraproject.org/rpms/ruby/blob/rawhide/f/ruby.spec), we were running the commands below.

```
$ ./autogen.sh
$ ./configure ...
$ make
$ make check
```

## What is the minimal covered tests?

Now the we want to add the `--enable-yjit` option. Could you tell me what is the minimal covered test commands in the `--enable-yjit` case?

```
$ ./autogen.sh
$ ./configure --enable-yjit ...
$ make
$ ??
```

Then do we need to run the `make check` 2 times with both with yjit and without yjit as follows? The yjit command options `--yjit-call-threshold=1 --yjit-verify-ctx` comes from the <https://github.com/ruby/ruby/blob/d2fa67de81f66cb42cfeebc81a03c57a4621c09a/.github/workflows/yjit-ubuntu.yml#L59>.

```
$ make check
$ make check RUN_OPTS="--yjit-call-threshold=1 --yjit-verify-ctx"
```

Or is it good enough to run the `make check` and the specific tests with the yjit options as follows?

```
$ make check
$ make test-all RUN_OPTS="--yjit-call-threshold=1 --yjit-verify-ctx" TESTS="test/ruby/{test_yjit_exit_locations.rb,test_yjit.rb}"
```

Or is it good enough to run the `make check` with the YJIT options as follows?

```
$ make check RUN_OPTS="--yjit-call-threshold=1 --yjit-verify-ctx"
```

## YJIT command options

Could you explain why the command options `--yjit-call-threshold=1 --yjit-verify-ctx` above is better to test the YJIT cases rather than just `--yjit`?

## Ideal situation

I want to see the just running `make check` covers necessary cases in YJIT. Because it is convenience, and I think users tend to be satisfied with only running the `make check`. What do you think?

```
$ ./autogen.sh
$ ./configure --enable-yjit ...
$ make
$ make check
```

I tried to inject the YJIT command options in a test file for that. Perhaps it might be like this. But so far I am not succeeded.

```
diff --git a/test/lib/jit_support.rb b/test/lib/jit_support.rb
index 26f8542dc2..3fce402e32 100644
--- a/test/lib/jit_support.rb
+++ b/test/lib/jit_support.rb
@@ -69,8 +69,10 @@ def supported?
   end
 
   def yjit_supported?
+    return @yjit_supported if defined?(@yjit_supported)
     # e.g. x86_64-linux, x64-mswin64_140, x64-mingw32, x64-mingw-ucrt
-    RUBY_PLATFORM.match?(/^(x86_64|x64|arm64|aarch64)-/)
+    @yjit_supported = RbConfig::CONFIG["YJIT_SUPPORT"] != 'no' &&
+      RUBY_PLATFORM.match?(/^(x86_64|x64|arm64|aarch64)-/)
   end
 
   def remove_mjit_logs(stderr)
diff --git a/test/ruby/test_yjit.rb b/test/ruby/test_yjit.rb
index 9ab058d97b..10c8e3b891 100644
--- a/test/ruby/test_yjit.rb
+++ b/test/ruby/test_yjit.rb
@@ -8,7 +8,7 @@
 require 'tmpdir'
 require_relative '../lib/jit_support'
 
-return unless defined?(RubyVM::YJIT) && RubyVM::YJIT.enabled?
+return unless JITSupport.yjit_supported?
 
 # Tests for YJIT with assertions on compilation and side exits
 # insipired by the MJIT tests in test/ruby/test_mjit.rb
```




-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>