From: Robert Klemme <lists@...>
Date: 2012-03-30T17:14:14+09:00
Subject: Re: copy file into new without dups, eol problem

Jeremy Bopp wrote in post #1054030:
> On 03/29/2012 11:11 AM, Robert Klemme wrote:
>> the original with a small insertion:
>>
>>     in.each_line do |line|
>>       line.chomp!
>>       out.puts line unless line == last
>>       last = line
>>     end
>>   end
>> end
>
> The limitation here is that duplicate lines must occur in consecutive
> runs.  If they are interleaved with different lines, this filter won't
> work.  For the general case, loading all the lines and running #uniq
> over the array is more likely to work.  I admit that it's not very
> efficient for large files though.

The example had consecutive duplicate lines.  But we do not have a clear
specification of the problem, namely the input and desired output.  If
the identical behavior of the readlines-uniq approach is needed I would
do this:

require 'set'

File.open "oldf.txt" do |io_in|
  File.open "newf.txt", "w" do |out|
    seen = Set.new

    io_in.each_line do |line|
      line.chomp!
      out.puts line if seen.add? line
    end
  end
end

Kind regards

robert

-- 
Posted via http://www.ruby-forum.com/.