From: Xiangrong Fang <xrfang@...>
Date: 2003-03-16T14:33:55+09:00
Subject: Hash load and GC

Hi Guys,

I am writing a program that need to load a big English dictionary from
disk. The format of the dictionary is one word per line. I used the
following routine to test:

require 'yabm'
a={}
dict1="en.dic"
GC.disable
bm=Yabm.new
File.open(dict1) do |f|
  f.binmode
  str=f.sysread(File.size(dict1))
  str.scan(/[^\n]+/) do |w|
 	a[w]=1
  end
end
bm.log("sysread")
a.clear
bm.log("clear hash")
File.open(dict1) do |f|
	while line=f.gets
		a[line]=1
	end
end
bm.log("f.gets")
bm.report
GC.enable

The result:

Without GC.disable/GC.enable

Time used: 8.071 sec.
---------------------
sysread: 2.643 sec. (32.75%)
clear hash: 0.101 sec. (1.25%)
f.gets: 5.327 sec. (66.00%)

With GC.disable/GC.enable

Time used: 7.561 sec.
---------------------
sysread: 2.424 sec. (32.06%)
clear hash: 0.080 sec. (1.06%)
f.gets: 5.057 sec. (66.88%)

Now I have 2 questions:

1. Which is the best way to load a big hash from disk? I have searched
the talk archive, and I find that sysread seems the fastest. I also
tried to use Marshal.load, it seems slower. BTW, I'm using 1.6.7-mswin32

2. Disable GC, as discussed before in the list, seems useful in
improving performance, however, if I do so, will Ruby recover memory
after I call GC.enable? In general, is it safe to use GC.disable in
program to speed up?

Thanks!

-- 
Shannon Fang <xrfang@hotmail.com>