From: Brandon Weaver Date: 2012-11-13T07:24:46+09:00 Subject: Re: Efficient way for comparing records between 2 large files (16 million records) --047d7b33d4d4baf77e04ce53c49a Content-Type: text/plain; charset=ISO-8859-1 Generate an MD5 or stronger hash on the file. If anything changes, the hash changes. I say 'or stronger' because MD5 has been shown to have (extremely rare) overlaps. On Mon, Nov 12, 2012 at 4:21 PM, Ruby Student wrote: > Team, > > I have two large files, about 16 million records each. > The files are sorted. > The first 13 characters are used as a key. > We get an updated file every week. > We also keep the previous week file. > > I need to compare the keys from the new file to the keys on the file from > last week. If the rest of the records are the same, then I do nothing. If > the keys matches but the rest of the records are different, I then have an > update and I will output that record to a new file. > I was wondering if there is an efficient way to do this in Ruby. Either > any built-in method or an efficient algorithm which I can implement. > > Thank you > > > -- > Ruby Student > --047d7b33d4d4baf77e04ce53c49a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Generate an MD5 or stronger hash on the file. If anything changes, the hash= changes. I say 'or stronger' because MD5 has been shown to have (e= xtremely rare) overlaps.

On Mon, Nov 12, = 2012 at 4:21 PM, Ruby Student <ruby.student@gmail.com> = wrote:

Team,

I have two large files, about 1= 6 million records each.
The files are sorted.
The first 13 characters= are used as a key.
We get an updated file every week.
We also keep the previous week file.<= br>
I need to compare the keys from the new file to the keys on the file from l= ast week. If the rest of the records are the same, then I do nothing. If th= e keys matches but the rest of the records are different, I then have an up= date and I will output that record to a new file.
I was wondering if there is an efficient way to do this in Ruby. Either any= built-in method or an efficient algorithm which I can implement.

Th= ank you

--
Ruby Student

--047d7b33d4d4baf77e04ce53c49a--