From: Robert Klemme <shortcutter@...>
Date: 2006-05-16T23:49:14+09:00
Subject: Re: Nonlinear scaling - could symbols help solve this?

2006/5/16, Jeremy Hanford <hanford_j@aircom.co.za>:

> The code below does this in a somewhat clumsy way using Ruby. I find that when running this on 10 000 lines -> execution time is 4.5 seconds, however if I increase to 300 000 lines (30 times the data) -> execution time is 14 min 21 sec (168 times the execution time). Is this nonlinear scaling due to allocating memory to hold the data? Could I improve this using symbols? Any other general remarks would be appreciated.

You can verify that yourself.  Use your operating system's tools to
monitor memory usage and page faults of the Ruby process and you'll
soon know whether it's slow due to paging or not.

Symbols can help with repeated content but you could as well feed
strings you want to store through a hash like this in order to prevent
duplicates:

strings = Hash.new {|h,k| k=k.freeze; h[k]=k}

str = strings[str]

Kind regards

robert