From: Robert Klemme <bob.news@...>
Date: 2005-09-24T16:51:42+09:00
Subject: Re: Optimizing a single slow method

------=_NextPart_000_006C_01C5C0ED.4504C5D0
Content-Type: text/plain;
	format=flowed;
	charset="iso-8859-1";
	reply-type=response
Content-Transfer-Encoding: 7bit

Robert Klemme <bob.news@gmx.net> wrote:
> Now it would be interesting to see which difference in the code
> caused the performance difference.  My guess is it's any or several
> of these:
> - I freeze hash keys.  This saves a dup.freeze on the keys inserted
> into hashes (it's an internal implementation speciality of Hash to
> avoid accidental aliasing effects through key strings changed after
> the insert)
> - I didn't use split thus avoiding unnecessary object creations in
> case a record is not needed.
>
> - I probably made the regexp more selective and thus more efficient.
>
> <snip/>

I forgot missing date conversion in this list.  One solution for date 
calculations then would be to do the calculations for date range with dates 
and then convert results to strings for comparison.  However, I found a 
conversion method that's quite efficient (see attached) although it slows 
down the post processing (sorting) a bit. :-)

There's also a more efficient access to individual values in there...

Btw, I left the interest and volume out to keep time measurements 
comparable.

Kind regards

    robert


------=_NextPart_000_006C_01C5C0ED.4504C5D0
Content-Type: application/octet-stream;
	name="parse_contracts.rb"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
	filename="parse_contracts.rb"

#!/usr/bin/ruby=0A=
=0A=
# a single line:=0A=
# A62005U,050909,0.7726,0.7758,0.7703,0.7737,12366,0=0A=
=0A=
require 'date'=0A=
=0A=
class IllegalStateException < StandardError=0A=
end=0A=
=0A=
# -------------------------------=0A=
=0A=
class Contract=0A=
  Tick =3D Struct.new :date, :open, :high, :low, :close=0A=
=0A=
  class Tick=0A=
    include Comparable=0A=
=0A=
    def <=3D> (tick) self.date <=3D> tick.date end=0A=
  end=0A=
=0A=
  attr_reader :contract_id=0A=
=0A=
  def initialize(contract_id)=0A=
    @contract_id =3D contract_id=0A=
    @ticks =3D []=0A=
  end=0A=
=0A=
  def add_tick(date, open, high, low, close)=0A=
    @ticks << Tick.new( date, open, high, low, close )=0A=
  end=0A=
=0A=
  def sort_ticks() @ticks.sort! end=0A=
=0A=
  # get all ticks for the given field=0A=
  def ticks(field) FieldEnum.new(@ticks, field) end=0A=
=0A=
private=0A=
  # helper class, ideally this would be handled by RCR 262=0A=
  # http://www.rcrchive.net/rcr/show/262=0A=
  class FieldEnum=0A=
    include Enumerable=0A=
=0A=
    def initialize(enums, field)=0A=
      @enums =3D enums=0A=
      @field =3D field=0A=
    end=0A=
=0A=
    def each=0A=
      @enums.each {|e| yield e[@field]}=0A=
      self=0A=
    end=0A=
  end=0A=
end=0A=
=0A=
# -------------------------------=0A=
=0A=
class ContractSet=0A=
  def initialize=0A=
    # efficient date conversion, saves also memory=0A=
    # because dates (immutable!) are aliased=0A=
    @dates =3D Hash.new do |h,ds|=0A=
      yr =3D ds[0,2].to_i=0A=
      h[ds.freeze] =3D Date.new( ( yr < 40 ? 2000 : 1900 ) + yr, =
ds[2,2].to_i, ds[4,2].to_i )=0A=
    end=0A=
  end=0A=
=0A=
  def parse_reference(io)=0A=
    @contracts =3D {}=0A=
=0A=
    io.each_line do |line|=0A=
      if /^(\w\d+\w),/ =3D~ line=0A=
        cid =3D $1.freeze=0A=
        @contracts[cid] ||=3D Contract.new cid=0A=
      end=0A=
    end=0A=
  end=0A=
=0A=
  def parse_ticks(io)=0A=
    raise IllegalStateException, "need to read reference first" unless =
@contracts=0A=
=0A=
    io.each_line do |line|=0A=
      if %r{^=0A=
             (\w\d+\w),         # contractid=0A=
             (\d{6}),           # date=0A=
             (\d+(?:\.\d+)?),   # open=0A=
             (\d+(?:\.\d+)?),   # high=0A=
             (\d+(?:\.\d+)?),   # low=0A=
             (\d+(?:\.\d+)?),   # close=0A=
           }x =3D~ line=0A=
        cid =3D $1.freeze=0A=
=0A=
        contract =3D @contracts[cid] and=0A=
          contract.add_tick @dates[$2], $3.to_f, $4.to_f, $5.to_f, =
$6.to_f=0A=
      end=0A=
    end=0A=
  end=0A=
=0A=
  def parse_ticks_2(io)=0A=
    create =3D @contracts.nil?=0A=
    @contracts =3D {} if create=0A=
=0A=
    io.each_line do |line|=0A=
      if %r{^=0A=
             (\w\d+\w),         # contractid=0A=
             (\d{6}),           # date=0A=
             (\d+(?:\.\d+)?),   # open=0A=
             (\d+(?:\.\d+)?),   # high=0A=
             (\d+(?:\.\d+)?),   # low=0A=
             (\d+(?:\.\d+)?),   # close=0A=
           }x =3D~ line=0A=
        cid =3D $1.freeze=0A=
=0A=
        contract =3D (@contracts[cid] || ( create && (@contracts[cid] =
=3D Contract.new cid) ) ) and=0A=
          contract.add_tick @dates[$2], $3.to_f, $4.to_f, $5.to_f, =
$6.to_f=0A=
      end=0A=
    end=0A=
  end=0A=
=0A=
  def parse_ticks_3(io)=0A=
    if @contracts=0A=
      cg =3D @contracts=0A=
    else=0A=
      @contracts =3D {}=0A=
      cg =3D lambda {|c_id| @contracts[c_id] ||=3D Contract.new c_id}=0A=
    end=0A=
=0A=
    io.each_line do |line|=0A=
      if %r{^=0A=
             (\w\d+\w),         # contractid=0A=
             (\d{6}),           # date=0A=
             (\d+(?:\.\d+)?),   # open=0A=
             (\d+(?:\.\d+)?),   # high=0A=
             (\d+(?:\.\d+)?),   # low=0A=
             (\d+(?:\.\d+)?),   # close=0A=
           }x =3D~ line=0A=
        cid =3D $1.freeze=0A=
=0A=
        contract =3D cg[cid] and=0A=
          contract.add_tick @dates[$2], $3.to_f, $4.to_f, $5.to_f, =
$6.to_f=0A=
      end=0A=
    end=0A=
  end=0A=
=0A=
  def post_process=0A=
    @contracts.each_value do |contract|=0A=
      contract.sort_ticks=0A=
    end=0A=
  end=0A=
end=0A=
=0A=
class Ticker=0A=
  def initialize()=0A=
    @t =3D Time.now.to_f=0A=
    @count =3D 0.0=0A=
  end=0A=
=0A=
  def tick(msg)=0A=
    diff =3D Time.now.to_f - @t=0A=
    @count +=3D diff=0A=
    print diff, msg=0A=
    @t =3D Time.now.to_f=0A=
  end=0A=
=0A=
  def total=0A=
    print @count, "total"=0A=
  end=0A=
=0A=
  def print(diff, msg)=0A=
    printf "%8.3fs %s\n", diff, msg=0A=
  end=0A=
end=0A=
=0A=
# =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=0A=
# MAIN=0A=
# =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=0A=
=0A=
t =3D Ticker.new=0A=
cs =3D ContractSet.new=0A=
=0A=
# File.open(ARGV.shift, IO::RDONLY) {|io| cs.parse_reference io}=0A=
# t.tick "read ref"=0A=
=0A=
ARGV.each do |f|=0A=
  File.open(f, IO::RDONLY) {|io| cs.parse_ticks_3 io}=0A=
  t.tick "read file #{f}"=0A=
end=0A=
=0A=
cs.post_process=0A=
=0A=
t.tick "finished post processing"=0A=
t.total=0A=

------=_NextPart_000_006C_01C5C0ED.4504C5D0--