From: Johannes Friestad <johannes.friestad@...>
Date: 2005-12-19T06:16:44+09:00
Subject: Re: Removing parts from a file

On 12/18/05, Thomas Dutch <rubyforum@ikwisthet.net> wrote:
> Is it possible to remove one or more lines from a file, without reading
> the whole file and writing it away again? This because I'll have to do
> this with files of 1 gigabyte and larger... Is there a high performance
> solution for this?
>

Sorry, have to agree with the other respondents.
So you are looking for a high-performance write-through as the next best thing.

For higher performance on large files, you'll want to read into a
buffer. Something like this:

BUFFER_SIZE=10000
# infile: input file name.
# outfile: target file name
# omit_start, omit_end: skips part between start and end; start/end given as
# file position byte count.
def copy_file_except(infile, outfile, omit_start, omit_end)
  out=File.open(outfile, 'w')
  begin
  File.open(infile, 'r') {|input|
    buffer=" "*BUFFER_SIZE # create a read buffer
    fpos=0 # current file position for end of buffer
    while input.read(BUFFER_SIZE, buffer)
      bstart=fpos # file position of buffer start
      bend=fpos+buffer.length # file position of buffer end
      fpos+=buffer.length
      if bend<omit_start or bstart>omit_end # entire buffer outside 'omit' range
        out.write(buffer)
        next
      elsif bstart>=omit_start and bend<=omit_end # entire buffer
inside 'omit' range
        next # skip all of it
      end
      # first part of buffer outside omit range
      out.write(buffer[0, omit_start-bstart]) if bstart<omit_start
      # last part of buffer outside omit range
      out.write(buffer[(omit_end-bstart..-1)]) if bend>omit_end
    end
  }
  ensure
    out.close
  end
end

I don't have any gigabyte files lying around, but a 25MB file took
~17s with File.each_line and ~2s using a buffer as above.

I also think I would try experimenting with the 'IO.sysread/syswrite' methods.

I'm fairly new to Ruby as well (couple of weeks), so there may be
simpler and/or better performing solutions available.