[ruby-core:71230] [Ruby trunk - Feature #11625] Unlock GVL for SHA1 calculations

Issue #11625 has been updated by Aaron Patterson.

File sha1gvl.diff added

Hi Eric,

Thanks for the feedback.  I've updated the patch will your suggestions.  Th=
ank you!

sha1.h exports SHA1_Transform, so I was worried about touching that one.

----------------------------------------
Feature #11625: Unlock GVL for SHA1 calculations
https://bugs.ruby-lang.org/issues/11625#change-54608

* Author: Aaron Patterson
* Status: Open
* Priority: Normal
* Assignee:=20
----------------------------------------
I'm trying to calculate many sha1 checksums, but the current sha1 implement=
ation doesn't unlock the GVL, so I can't do it in parallel.  I've attached =
a patch that unlocks the GVL when calculating sha1sums so that I can do the=
m in parallel.

The good point about this patch is that I can calculate sha1's in parallel.=
  Here is the test code I'm using:

~~~
require 'digest/sha1'
require 'thread'

Thread.abort_on_exception =3D true

THREADS =3D (ENV['THREADS'] || 1).to_i

store =3D 'x' * (ENV['SIZE'] || 1024).to_i

queue =3D Queue.new

600000.times do
  queue << store
end

THREADS.times { queue << nil }

ts =3D THREADS.times.map {
  Thread.new {
    while work =3D queue.pop
      Digest::SHA1.hexdigest(work)
    end
  }
}

ts.each(&:join)
~~~

Here is what the output looks like after I've applied the patch:

~~~
[aaron@TC ruby (trunk)]$ THREADS=3D1 SIZE=3D4096 time ./ruby test.rb=20
       22.62 real        21.78 user         0.66 sys
[aaron@TC ruby (trunk)]$ THREADS=3D4 SIZE=3D4096 time ./ruby test.rb=20
       15.87 real        34.53 user         8.27 sys
[aaron@TC ruby (trunk)]$=20
~~~

The digests that I'm calculating are for fairly large strings, so this patc=
h works well for me.  The downsides are that it seems slightly slower (thou=
gh I'm not sure that it's significant) with a single thread:

Test code:

~~~
require 'benchmark/ips'
require 'digest/sha1'

Benchmark.ips do |x|
  x.report('sha1') { Digest::SHA1.hexdigest('x' * 4096) }
end
~~~

Before my patch (higher numbers are better):

~~~
[aaron@TC ruby (trunk)]$ ./ruby shaips.rb=20
Calculating -------------------------------------
                sha1     2.604k i/100ms
-------------------------------------------------
                sha1     27.441k (=C2=B1 3.9%) i/s -    138.012k
~~~

After my patch:

~~~
[aaron@TC ruby (trunk)]$ ./ruby shaips.rb=20
Calculating -------------------------------------
                sha1     2.419k i/100ms
-------------------------------------------------
                sha1     25.848k (=C2=B1 2.8%) i/s -    130.626k
~~~

Other downside is that I changed the `update` method to dup strings so that=
 the GVL can be safely released.

This patch pays off for me because of the size of the strings I'm working w=
ith, but I'm not sure if it's fine for the general case.

---Files--------------------------------
sha1gvl.diff (3.16 KB)
sha1gvl.diff (3.79 KB)


--=20
https://bugs.ruby-lang.org/