[ruby-core:71236] Re: [Ruby trunk - Feature #11625] Unlock GVL for SHA1 calculations

What's about other hashsum algos? MD5, SHA2, etc
Issue #11625 has been updated by Aaron Patterson.

File sha1gvl.diff added

Hi Eric,

Thanks for the feedback.  I've updated the patch will your suggestions.
Thank you!

sha1.h exports SHA1_Transform, so I was worried about touching that one.

----------------------------------------
Feature #11625: Unlock GVL for SHA1 calculations
https://bugs.ruby-lang.org/issues/11625#change-54608

* Author: Aaron Patterson
* Status: Open
* Priority: Normal
* Assignee:
----------------------------------------
I'm trying to calculate many sha1 checksums, but the current sha1
implementation doesn't unlock the GVL, so I can't do it in parallel.  I've
attached a patch that unlocks the GVL when calculating sha1sums so that I
can do them in parallel.

The good point about this patch is that I can calculate sha1's in
parallel.  Here is the test code I'm using:

~~~
require 'digest/sha1'
require 'thread'

Thread.abort_on_exception =3D true

THREADS =3D (ENV['THREADS'] || 1).to_i

store =3D 'x' * (ENV['SIZE'] || 1024).to_i

queue =3D Queue.new

600000.times do
  queue << store
end

THREADS.times { queue << nil }

ts =3D THREADS.times.map {
  Thread.new {
    while work =3D queue.pop
      Digest::SHA1.hexdigest(work)
    end
  }
}

ts.each(&:join)
~~~

Here is what the output looks like after I've applied the patch:

~~~
[aaron@TC ruby (trunk)]$ THREADS=3D1 SIZE=3D4096 time ./ruby test.rb
       22.62 real        21.78 user         0.66 sys
[aaron@TC ruby (trunk)]$ THREADS=3D4 SIZE=3D4096 time ./ruby test.rb
       15.87 real        34.53 user         8.27 sys
[aaron@TC ruby (trunk)]$
~~~

The digests that I'm calculating are for fairly large strings, so this
patch works well for me.  The downsides are that it seems slightly slower
(though I'm not sure that it's significant) with a single thread:

Test code:

~~~
require 'benchmark/ips'
require 'digest/sha1'

Benchmark.ips do |x|
  x.report('sha1') { Digest::SHA1.hexdigest('x' * 4096) }
end
~~~

Before my patch (higher numbers are better):

~~~
[aaron@TC ruby (trunk)]$ ./ruby shaips.rb
Calculating -------------------------------------
                sha1     2.604k i/100ms
-------------------------------------------------
                sha1     27.441k (=C2=B1 3.9%) i/s -    138.012k
~~~

After my patch:

~~~
[aaron@TC ruby (trunk)]$ ./ruby shaips.rb
Calculating -------------------------------------
                sha1     2.419k i/100ms
-------------------------------------------------
                sha1     25.848k (=C2=B1 2.8%) i/s -    130.626k
~~~

Other downside is that I changed the `update` method to dup strings so that
the GVL can be safely released.

This patch pays off for me because of the size of the strings I'm working
with, but I'm not sure if it's fine for the general case.

---Files--------------------------------
sha1gvl.diff (3.16 KB)
sha1gvl.diff (3.79 KB)


--
https://bugs.ruby-lang.org/