From: Tom Trebisky <tom@...>
Date: 2004-05-28T09:12:04+09:00
Subject: Nasty string growth performance


Hello all,

I am running:

    ruby 1.8.0 (2003-08-04) [i386-linux-gnu]

I am just writing to ask about and describe a really terrible
performance issue.  I ran across this writing some code to
process mail folders.  I wanted a method to gather up the
entire body of a mail message into one object, and initially
I chose to use a string as follows:

	    line = nil
	    buf = ""
            while line = @file.gets
                if line =~ /^From /
                    @pushback = line
                    break
                end
                buf += line
            end

Processing a folder with 2355 mail messages with this took
about 386 seconds on my 1Ghz pentium machine.

Then, I decided to use an array of lines rather than a really
big string, as follows:

	    line = nil
	    buf = Array.new
            while line = @file.gets
                if line =~ /^From /
                    @pushback = line
                    break
                end
                buf.push line
            end
                                                                                                                                                             
This code processes the same folder in 9.4 seconds.
(about 40 times faster than the above).

We first noticed this problem when another (and entirely
different) ruby application began consuming huge amounts
of CPU resources, and before we were able to really
understand what was going on ....
  I hang my head in shame when I admit it ....
    we threw our hands up and recoded it in Perl.

	    Tom

-- 
Tom Trebisky
MMT Observatory
University of Arizona -- Tucson
tom@mmto.org