From: xtkoba+ruby@... Date: 2021-05-14T10:56:22+00:00 Subject: [ruby-core:103844] [Ruby master Bug#17864] [BUG] try to mark T_NONE object (in Ractors) Issue #17864 has been updated by xtkoba (Tee KOBAYASHI). Yeah, I believe this has already been fixed (or at least the probability of segfault has been made much lower). ---------------------------------------- Bug #17864: [BUG] try to mark T_NONE object (in Ractors) https://bugs.ruby-lang.org/issues/17864#change-91963 * Author: NuriYuri (Youri Nouri) * Status: Open * Priority: Normal * ruby -v: ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [i386-mingw32] * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN ---------------------------------------- Hello, I tried to make an experiment with Ractors, computing SHA1 using Ractors (I was hopping to divide the computation time because SHA1 hashing does not release GVL and I'm running a "GUI" that has time constraint). When I test with StringIO everything works fine, I can compute 400 StringIO that are like 100MB without issues. But when I use files (what I need to do in real environment) I face a internal bug (that doesn't not spawn in all attempt, it's very random). And here's all the information about the bug: ruby -v = `ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [i386-mingw32]` Error log: ``` 0x053fbf68 [0 M ] T_NONE D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105: [BUG] try to mark T_NONE object ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [i386-mingw32] -- Control frame information ----------------------------------------------- c:0003 p:---- s:0017 e:000016 CFUNC :read c:0002 p:0056 s:0012 e:000011 BLOCK D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105 [FINISH] c:0001 p:---- s:0003 e:000002 (none) [FINISH] -- Ruby level backtrace information ---------------------------------------- D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105:in `block in make_ractor' D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105:in `read' -- C level backtrace information ------------------------------------------- C:\WINDOWS\SYSTEM32\ntdll.dll(ZwWaitForSingleObject+0xc) [0x775729ac] C:\WINDOWS\System32\KERNELBASE.dll(WaitForSingleObject+0x12) [0x767d0452] D:\nuriy\persoWork\game_launcher\msvcrt-ruby300.dll(rb_vm_bugreport+0x669) [0x68134f29] D:\nuriy\persoWork\game_launcher\msvcrt-ruby300.dll(rb_source_location+0x53) [0x68118d83] C:\WINDOWS\System32\KERNELBASE.dll(CreateEventA+0x24) [0x767e1914] D:\nuriy\persoWork\game_launcher\msvcrt-ruby300.dll(rb_vm_proc_local_ep+0xd81) [0x680d8741] C:\WINDOWS\SYSTEM32\ntdll.dll(RtlGetAppContainerNamedObjectPath+0x11e) [0x77567a4e] -- Other runtime information ----------------------------------------------- * Loaded script: src/IODigester.ytest copy.rb * Loaded features: 0 enumerator.so 1 thread.rb 2 rational.so 3 complex.so 4 ruby2_keywords.rb 5 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/encdb.so 6 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/trans/transdb.so 7 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/rbconfig.rb 8 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/compatibility.rb 9 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/defaults.rb 10 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/deprecate.rb 11 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/errors.rb 12 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/exceptions.rb 13 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime/singleton.rb 14 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime.rb 15 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime/msys2_installation.rb 16 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/fiddle.so 17 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/closure.rb 18 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/function.rb 19 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/version.rb 20 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle.rb 21 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime/dll_directory.rb 22 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/value.rb 23 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/pack.rb 24 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/struct.rb 25 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/cparser.rb 26 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/import.rb 27 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/utf_16le.so 28 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/trans/utf_16_32.so 29 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/win32/registry.rb 30 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/trans/single_byte.so 31 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/defaults/operating_system.rb 32 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/basic_specification.rb 33 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/stub_specification.rb 34 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/text.rb 35 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/user_interaction.rb 36 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/specification_policy.rb 37 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/util/list.rb 38 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/platform.rb 39 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/version.rb 40 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/requirement.rb 41 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/specification.rb 42 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/util.rb 43 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/dependency.rb 44 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/core_ext/kernel_gem.rb 45 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/monitor.so 46 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/monitor.rb 47 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb 48 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/core_ext/kernel_warn.rb 49 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems.rb 50 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/path_support.rb 51 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/version.rb 52 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/core_ext/name_error.rb 53 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/levenshtein.rb 54 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/jaro_winkler.rb 55 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checker.rb 56 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/name_error_checkers/class_name_checker.rb 57 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/name_error_checkers/variable_name_checker.rb 58 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/name_error_checkers.rb 59 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/method_name_checker.rb 60 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/key_error_checker.rb 61 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/null_checker.rb 62 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/tree_spell_checker.rb 63 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/require_path_checker.rb 64 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/formatters/plain_formatter.rb 65 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean.rb 66 D:/nuriy/persoWork/game_launcher/src/IODigester.rb 67 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/digest.so 68 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/digest.rb 69 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/digest/sha1.so ``` Note: I noticed while writing the thread that my ruby version in this project was not updated, I tried with `ruby 3.0.1p64 (2021-04-05 revision 0fb782ee38) [i386-mingw32]` and I have nothing but Idk if that's luck or not. If someone can confirm that this bug is solved that would be nice because I really hope to use that in production even if Ractors are still experimental. **Additional info** Here's the class I wrote to handle the SHA1 computation over ractors: ```ruby # Class responsive of digesting data (hashing) over several CPU core using Ractors # # How to use ? # 1. First of all encapsulate all your data into IO object (File or StringIO) # 2. Then create a new IODigester object by passing the Digest class you need and all the IO to digest in a Hash # 3. Call the start function to let the system instanciate everything # * You can check the overall progress using #progess, it's a number between 0 & 1 # 4. If failed? respond true, you can grab the failure info using #failures (key = IO key, value = StandardError object) # # @example Demo # digester = IODigester.new(Digest::SHA1, { 'test' => io1, 'test2' => io2 }) # digester.start { |d| print format("\r%.2f%%", progress: d.progress * 100) } # digester.hashes # => { 'test' => '', 'test2' => '' } # # @note IODigester requires the IO to respond to size and give accurate value for size (in bytes)! class IODigester # Get the hashes # @return [Hash{ String => String }] attr_reader :hashes # Get the failures # @return [Hash{ String => StandardError }] attr_reader :failures # Create a new IODigester # @param klass [Class] digest class to use to digest data # @param ios [Hash{ String => IO }] io to digest (key can be symbol if you want) def initialize(klass, ios) @klass = klass @ios = ios.clone @progresses = @ios.keys.map { |k| [k, 0] }.to_h @hashes = {} @failures = {} end # Start the process # @return [self] # @yieldparam d [self] current digester def start(&block) raise 'Already started' if @ractors @ractors = @ios.map { |key, value| make_ractor(key, value) } process_ractors(block) return self end # Test if the process has failures # @return [Boolean] def failed? return !@failures.empty? end # Get the progress # @return [Float] def progress @progresses.reduce(0.0) { |prev, curr| prev + curr.last } / @progresses.size end private # Function that processes all the ractors (in a Thread in order to not lock the GVL) # @param block [Proc] def process_ractors(block) until @ractors.empty? begin ractor, object = Ractor.select(*@ractors) parse_ractor_response(ractor, object) block&.call(self) rescue Ractor::RemoteError => e @failures[e.ractor.name] = e.cause @ractors.delete(e.ractor) end end end # Function that analyzes the payload of the received object from Ractor.select # @param ractor [Ractor] ractor object that we're talking about # @param object [Array] response from the ractor def parse_ractor_response(ractor, object) message, body = object case message when :progress @progresses[ractor.name] = body when :result @progresses[ractor.name] = 1 @hashes[ractor.name] = body @ractors.delete(ractor) end end # Function that creates the ractor # @param name [String] name of the io (hash key) # @param io [IO] io to process # @return [Ractor] def make_ractor(name, io) ractor = Ractor.new(@klass, io, name: name) do |digest_klass, io_to_process| close_incoming # @type [Integer] io_size = io_to_process.size io_read = 0 # @type [Digest::Base] digest = digest_klass.new while io_size > io_read Ractor.yield [:progress, io_read.to_f / io_size] # @type [String] data = io_to_process.read(20_971_520) io_read += data.bytesize digest << data end Ractor.yield [:result, digest.hexdigest] end return ractor end end ``` Here's the script I use to try to reproduce the bug: ```ruby require_relative 'IODigester' require 'digest' ios = ( Dir['D:/nuriy/Work/pokemonsdk/Release/Data/*'] + Dir['D:/nuriy/Work/pokemonsdk/Release/pokemonsdk/**/*'] ).reject { |f| File.directory?(f) }.map { |f| [f, File.new(f, 'rb')] }.to_h digester = IODigester.new(Digest::SHA1, ios) digester.start { |d| print format("\r%.2f%%", progress: d.progress * 100) } p digester.hashes ``` (I'm not closing the files because I don't actually care in that test since the bug appears in "start") Here's the files in case you need it: https://www.mediafire.com/file/wt0eqzthxpb1nc8/Release.7z/file (it's over 2MB) -- https://bugs.ruby-lang.org/ Unsubscribe: