From: "Ara.T.Howard" <Ara.T.Howard@...>
Date: 2005-08-13T09:16:33+09:00
Subject: Re: Performance Ruby

On Sat, 13 Aug 2005, Austin Ziegler wrote:

> Real world performance numbers are more useful, and when Ara Howard tells me
> that Ruby performs fast enough for his image crunching, or Rich Kilmer tells
> me that Ruby made Cougaar possible and fast, then I believe them. These
> people (and others) are pushing Ruby further than anyone else, including
> people who use Ruby on Rails. And yet, I also believe DHH when he says that
> Rails scalability isn't a big issue, in that it scales the same way that web
> applications have scaled for a while, and the same way that Kuro5hin and
> Slashdot scaled -- more machines with load balancing and separating the
> database server from the application server.

<snip>

here's some food for thought:

the other day my co-worker was out in dc.  one of our clients wanted to gain a
better understanding of one of our products.  the product in question is a
'fires' product that we make using nighttime lights of the world, some temp
info, etc. to show where nighttime fires are.  the fires are placed into a
geographic grid - an image where each pixel maps to a lat/lon.  the images are
quite simple and all meta info is in a .hdr file (this is envi's file format).
in any case placing fires in a grid leaves some empty grid cells since the
grid is of coarser resolution than the data.  we fill this grid in using
nearest-neighbor.  the client wanted to no which pixels were 'nubs' or centers
- which ones were actual pixels and not a product of filling.  it turns out we
track this info, along with 15 bits of other stuff, in a 'flag' image - it's
simply an image of pixels where each pixel is a bitmask marking certain
attributes of each pixel as true or false such as 'cloudy' and the like.

so i needed to make a 'fire center' image - it would be just like a fire image
but contain only 'nubs' or centers.  this is an easy thing to do since all i
had to do was copy the fires image and set everything to background that was
NOT a nub and was not no data (missing pixels).  here's a sample run:

   jib:~/eg/ruby/fires2firecenters/fires2firecenters-0.0.0 > time fires2firecenters F1420050720.0.fires

   real    0m9.931s
   user    0m2.240s
   sys     0m4.580s

o.k. ten seconds.  but what does that mean?

   jib:~/eg/ruby/fires2firecenters/fires2firecenters-0.0.0 > ls -ltarh
   F1420050720.0.fires F1420050720.0.flag F1420050720.0.fires.centers
   -rw-rw-r--    1 ahoward  ahoward      102M Aug 12 17:42 F1420050720.0.fires
   -rw-rw-r--    1 ahoward  ahoward      204M Aug 12 17:43 F1420050720.0.flag
   -rw-rw-r--    1 ahoward  ahoward      102M Aug 12 17:44 F1420050720.0.fires.centers

so you can see both input images (fires, flag) are 100/200 mb respectively and
the output is 100 mb.  so that's 300mb of io - at least.  i didn't even bother
to profile the code because i was super busy that day and it was fast enough.
the really good part, however, is the code:

     #!/usr/bin/env ruby
     require 'narray'
     require 'fileutils'
     #
     # parse command line - setup
     #
       fires_grid, flag_grid, fire_centers_grid = ARGV

       unless fires_grid
         STDERR.puts "#{ File::basename $0 } fires_grid [flag_grid] [fire_centers_grid]"
         exit 1
       end

       flag_grid ||= fires_grid.gsub( %r/fires$/, 'flag' )
       fire_centers_grid ||= fires_grid.gsub( %r/fires$/, 'fires.centers' )

       center_mask = 2 ** (center_bit = 4)
       no_data_mask = 2 ** (no_data_bit = 15)
     #
     # load files
     #
       fires = NArray::to_na( IO::read( fires_grid ), NArray::BYTE )
       flags = NArray::to_na( IO::read( flag_grid ), NArray::SINT )
     #
     # zero out anything that's neither center nor no_data
     #
       fire_centers = fires
       is_data = ( flags & no_data_mask ).not
       is_not_center = ( flags & center_mask ).not
       fire_centers[ is_not_center.and( is_data ) ] = 0
     #
     # write centers output
     #
       open(fire_centers_grid, 'wb'){|f| f.write fire_centers.to_s}
     #
     # copy .fires hdr to .fires.centers.hdr
     #
       FileUtils::cp "#{ fires_grid }.hdr", "#{ fire_centers_grid }.hdr"


it's a hack, it uses extensions, and it took me 5 minutes to write and ran
perfectly the first time.  i love it.

cheers.

-a
-- 
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death 
| Like a lamp standing in a strong breeze.  --Nagarjuna 
===============================================================================