From: Tristin Davis <tristin.colby@...>
Date: 2008-03-09T19:45:47+09:00
Subject: Re: IO#Foreach -- Max line length

That's what the 2nd if statement is; for catching the delimiter if the 
buffer isn't too large.  I can't use gets b/c I may expend all the 
memory before the actual line is read.  I'm reading variable length 
records, but some of them are bad data and exceed a max length of 100k. 
That's what the script is scanning for. :)

7stud -- wrote:
> Tristin Davis wrote:
>> Thanks for the ideas Adam. I thought someone might be able to use it so 
>> I figured i'd post it. It processed about 675,000 1100+ byte records in 
>> an hour.  Not fantastic performance, but it works.  If someone can tell 
>> me how to improve the performance then have at it.  :)
>> 
>> 
>> module Util
>> 
>>   def too_large?(buffer,max=10)
>>     return true if buffer.length >= max
>>     false
>>   end
>> end
>> 
>> include Util
>> 
>> file = ARGV.shift #"C:/Documents and Settings/trdavi/Desktop/a1-1k.aa"
>> buf=''
>> record = 1
>> frequency = 100
>> 
>> f = File.open(file,'r')
>> 
>>  while c=f.getc
>         if buf.length < max   #(but what if you find a '\n' before max?)
>           buf << c
>         else
>            buf = ''
>            f.gets
>         end

-- 
Posted via http://www.ruby-forum.com/.