From: Kent Sibilev <ksibilev@...>
Date: 2004-10-14T08:06:57+09:00
Subject: Re: GC and low file performance when large array is allocated

If you run Unix, maybe you should consider using mmap module?

Cheers,
Kent.
On Oct 13, 2004, at 5:44 PM, Geert Fannes wrote:

> Hello,
>
> I noticed that ruby's disc performance drops drastically when a large
> array is allocated. I think it has to do with garbage collection since
> the performance increases again by disabling the garbage collection. I
> created a small test program to illustrate the problem:
>
> #
> #begin of program
> #
> allocateBefore=true
> useFileLoop=true
> disableGC=false
>
> GC.disable if disableGC
>
> #create a file containing 100000 lines of 'test'
> File.open('testfile','w'){|fo| 100000.times{fo.puts 'test'}}
>
> largeArray=Array.new(20000000) if allocateBefore
>
> if useFileLoop
>   File.open('testfile') do |fi|
>     fi.each{|line|}
>   end
> else
>   1000000.times{|i|}
> end
>
> largeArray=Array.new(20000000) if !allocateBefore
> #
> #end of program
> #
>
> On my home pc, the above program takes 3.225 sec. If I allocate the
> large array AFTER the fi.each-loop by setting allocateBefore=false, it
> takes only 0.467 sec. The same good performance occurs when I disable
> the garbage collection by setting disableGC=true. Unfortunately,
> disabling GC is not an option in my real application since my file is
> a lot larger and all my memory gets consumed very fast.
>
> If I play with the allocateBefore and disableGC when the
> 1000000.times-loop is enabled (by setting useFileLoop=false), I don't
> get this difference anymore.
>
> Any idea what is going on here? How can I achieve a good file
> performance with large arrays in memory?
>
> Greets,
> Geert Fannes.
>