From: Rob Muhlestein <rmuhlestein@...>
Date: 2006-12-29T06:10:05+09:00
Subject: Re: unsafe readline(), anything better?

On Thu, 28 Dec 2006 12:10:04 -0800, William James wrote:

> Rob Muhlestein wrote:
>> Looking for a combination of readpartial() and readline() in order to
>> safely read a line of maxlen. IO.readline() appears to suffer from
>> lacking this maximum allowing DOS against servers reading HTTP headers,
>> for example, just by using sock.readline() alone, which is what
>> Net::HTTPResponse and others do:
>>
>>    1974     class << HTTPResponse
>>    1975       def read_new(sock)   #:nodoc: internal use only 1976
>>    httpv, code, msg = read_status_line(sock) 1977         res =
>>    response_class(code).new(httpv, code, msg) 1978
>>    each_response_header(sock) do |k,v| 1979           res.add_field k, v
>>    1980         end
>>    1981         res
>>    1982       end
>>    1983
>>    1984       private
>>    1985
>>    1986       def read_status_line(sock) 1987         str =
>>    sock.readline 1988         m =
>>    /\AHTTP(?:\/(\d+\.\d+))?\s+(\d\d\d)\s*(.*)\z/in.match(str) o        r
>>    1989           raise HTTPBadResponse, "wrong status line:
>>    #{str.dump}" 1990         m.captures
>>    1991       end
>>
>> I know this is fundamentally a problem with the popular readline C libs
>> out there. Wish they had an nreadline like the sprintf and snprintf
>> additional function. Sure I could readchar() or readbytes() watching
>> each read for a newline, but that is just unfun. Have I overlooked
>> something obvious in my search? Hoping to not have to write my own
>> safe/buffered IO layer like I've had to do with other langs.
>>
>> If there is enough interest, maybe I'll hack a readmaxline() method into
>> an IO patch to submit. Actually, on second thought, how about adding a
>> second parameter:
>>
>>     ios.readline(sep_string=$/,maxlen=nil)
>>
>> The tough question would then be whether to raise an LineTooLong
>> exception or just return what could be read of the line up to that
>> point.
>>
>> Thanks,
>>
>> Rob
> 
> class File
>   def safe_readline(sep_string=$/,maxlen=nil)
>     buf_size = 1024
>     line = ""
>     while !self.eof?
>       s = read( [ buf_size, maxlen - line.size ].min ) line << s
>       if i = line.index( sep_string )
>         line = line[0,i+sep_string.size]
>         return [ line, true ]
>       end
>       return [ line[0,maxlen], false ]  if maxlen &&
>         line.size >= maxlen
>     end
>     [ line, true ]
>   end
> end
> 
> open('junk'){|h| p h.safe_readline("\n",9) }

Thanks for the attempt, this is very close to the reader-flavor of
classes I've had to write in other langs. But it doesn't play nice
dealing with non-seekable streams like sockets, which can cause some
nasty IO socket blocking (or the nonblocking socket read run around).
For example:

  Header1: something
  Header2: else, followed by blank line
  Content-Length: 55 (or whatever)

  Here is the data portion that could be binary or text.

If I called h.safe_readline("\r\n",1024) and was working with an IO
socket there wouldn't be anything left by the time I want to read
the data. That is, if I ever had a chance to try since the read would
block my proggy from doing anything.

One solution is to create a Reader class and maintain a buffer which
keeps the extra overrun available. I was hoping at the binary level
during the byte read that readline does that it could keep a counter of
the number of bytes read as it reads them and throw or whatever when
maxlen exceeded. Doing a readbyte from most languages at the script/lang
level is usually way to costly, it would probably be too costly to weigh
down read() with a count and check for every byte, but perhaps not in
another safe_readline() native extension.

I do love that Ruby let's me add to IO itself, which handles adding
@prev_read buffer to store the overrun in for next read and is what
I'll do and post for review.

Thanks again,
-- 
Rob Muhlestein
http://rob.muhlestein.net