From: Austin Ziegler <halostatue@...>
Date: 2004-07-02T13:31:14+09:00
Subject: Re: Why I don't use Ruby.

On Fri, 2 Jul 2004 11:37:48 +0900, TLOlczyk <olczyk2002@yahoo.com> wrote:
> You've written the program and tested it on a small data set.
> Now you want to test it on a large data set. It fails miserably
> and you try to debug.
> Phase one of the program gets a list of all the files in the directory
> ( recursively ).
> Phase two is to scan the root files for hrefs. Then scan the files in
> the hrefs. Then ... untill there are no more files to scan.
> Phase three is to pick some file you didn't reach and create a link to
> it.
> 
> So you work on Phase one and get it to run.
> Then you work on Phase two. But it takes Phase one
> ten minutes to run. So each time you test a fix in phase two
> it takes at least ten minutes to run.
> 
> So you save an image at the end of phase one. Each time
> you test phase two, you load the image from phase one
> and just test phase two. Save ten minutes on each test run.

There is no reason that you can't do this with Ruby. If I have code
that works like:

  file_list = find_files
  missing_files = file_list - find_hrefs(file_list)
  make_hrefs(missing_files)

Then I can simply do:

  file_list = find_files
  File.open("file_list.m", "wb") { |f| f.write Marshal.dump(file_list) }

Then, I can make the main loop do:

  file_list = Marshal.load(File.read("file_list.m"))
  missing_files = file_list - find_hrefs(find_files)
  make_hrefs(missing_files)

With a bit of work, you could integrate this into the main program; it
might even be desirable if find_hrefs takes even longer than
find_files; that way, you could choose to load a predetermined
file_list (making it possible to "version" such lists) or use the
stored file_list as a comparison point. Or, even, do something like:

  if File.exists?("file_list.m")
    file_list = Marshal.load(File.read("file_list.m"))
    File.rm("missing_files.rm")
  else
    file_list = find_files
  end

  begin
    if File.exists?("missing_files.m")
      missing_files = Marshal.load(File.read("missing_files.m")
      File.rm("missing_files.rm")
    else
      missing_files = file_list - find_hrefs(find_files)
    end
  rescue
    File.open("file_list.m", "wb") { |f| f.write Marshal.dump(file_list) }
    raise
  end

  begin
    make_hrefs(missing_files)
  rescue
    File.open("missing_files.m", "wb") { |f| f.write Marshal.dump(file_list) }
    raise
  end

Now, you have something repeatable that withstands all sorts of errors
without having to repeat portions that you don't need to do.

-austin 
--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca