From: "byroot (Jean Boussier) via ruby-core" Date: 2024-02-14T20:17:03+00:00 Subject: [ruby-core:116762] [Ruby master Misc#20254] FYI: Add Launchable into Ruby CI Issue #20254 has been updated by byroot (Jean Boussier). So if I understand correctly, this isn't an alternative CI, just a side system that collect statistics and also better format test failures. I haven't looked at it in detail, but some form of flakiness analysis would indeed be very welcome. The one small bit of feedback I have about what I've seen, and that is mostly down to personal preference, is I tend to dislike bots that post GitHub comments, as I find them very noisy given my GitHub notification setup (email). So I wonder if it would be possible for it to register itself as a normal commit status instead? That said, GitHub Action already publish so many of these I understand one worry might be that it will be hard to find in the list? Also it would be nice to see a PR with it enabled on `ruby/ruby`. The result on `ruby/debug` look nice, but `ruby/ruby` CI output is way more complicated, so I'd be curious to see how well it adapts to that. ---------------------------------------- Misc #20254: FYI: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-106785 * Author: ono-max (Naoto Ono) * Status: Open * Priority: Normal * Assignee: ono-max (Naoto Ono) ---------------------------------------- I���m a software engineer who works at Launchable, Inc. Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. Screenshot 2024-01-29 at 19 23 08 ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. Test reports in Github comments ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. Flakey test page ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/ ______________________________________________ ruby-core mailing list -- ruby-core@ml.ruby-lang.org To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/