From: Tristin Davis <tristin.colby@...>
Date: 2008-03-09T06:31:20+09:00
Subject: Re: IO#Foreach -- Max line length

Thanks for the ideas Adam. I thought someone might be able to use it so 
I figured i'd post it. It processed about 675,000 1100+ byte records in 
an hour.  Not fantastic performance, but it works.  If someone can tell 
me how to improve the performance then have at it.  :)


module Util

  def too_large?(buffer,max=10)
    return true if buffer.length >= max
    false
  end
end

include Util

file = ARGV.shift #"C:/Documents and Settings/trdavi/Desktop/a1-1k.aa"
buf=''
record = 1
frequency = 100

f = File.open(file,'r')

 while c=f.getc
    buf << c

    if too_large?(buf,max=102400)
        p "record #{record} is too long, skipping to end"
        while(x=f.getc)
          if x.chr == $/
            buf=''
            record += 1
            p "At record #{record}" if( (record % frequency ) == 0 )
            break
          end
        end
    end

    if c.chr == $/
       record += 1
       print "At record #{record}" if( (record % frequency ) == 0 )
       buf = ''
    end
end

#If we still have something in the buffer, then it is probably the last 
record.
unless buf.empty?
  #record += 1
  p "Last record is:" + buf
end

f.close
p record



Adam Shelly wrote:
> On 3/6/08, Pe�a, Botp <botp@delmonte-phil.com> wrote:
>> On Behalf Of Tristin Davis:
>> # But by the time you actually get count, isn't the line
>> # already read in
>> # memory.  So if the line is 7 gigabytes, it'll probably crash
>> # the system.
>>
>> read will accept arg on how many bytes to read.
>>
>> so how about,
>>
> ...
>> irb(main):043:0> File.open "test.rb" do |f| while x=f.read(2); p x; end; end
> 
> That solution essentially ignores linebreaks.
> If you want to read up to a linebreak or N characters, whichever comes
> first, you could one of these:
> 
> ------
> class IO
>   #read by characters
>   def for_eachA(linelen)
>     c=0
>     while (c)
>       buf=''
>       linelen.times {
>         break unless c=getc
>         buf<<c
>         break if c.chr== $/
>       }
>       yield buf
>     end
>   end
> 
>   #read by lines
>   def for_eachB(linelen)
>     re = Regexp.new(".*?#{Regexp.escape($/)}")
>     buf=''
>     while (line = read(linelen-buf.length))
>       buf = (buf+line).gsub(re){|l| yield l;''}
>       if buf.length == linelen
>         yield buf
>         buf=''
>       end
>     end
>     yield buf
>   end
> end
> 
> File.open("foreach.rb") do |f|
> f.for_eachA(10){|l| p l}
> end
> 
> File.open("foreach.rb") do |f|
> f.for_eachB(10){|l| p l}
> end
> ------
> 
> I'd guess the second version would be faster, but I didn't time it.
> 
> -Adam

-- 
Posted via http://www.ruby-forum.com/.