From: Ruby Student Date: 2012-11-16T22:35:09+09:00 Subject: Re: Efficient way for comparing records between 2 large files (16 million records) --f46d043c7c6afc946904ce9cd521 Content-Type: text/plain; charset=ISO-8859-7 Content-Transfer-Encoding: quoted-printable Robert, You stated: "IIRC I even posted a solution outline." I went to your blog (http://blog.rubybestpractices.com/) but could not find the item you mentioned. Would you mind to post the link to it? Thank you On Wed, Nov 14, 2012 at 10:53 AM, Panagiotis Atmatzidis wrote: > Hello, > > On 12 =CD=EF=E5 2012, at 23:21 , Ruby Student wr= ote: > > > Team, > > > > I have two large files, about 16 million records each. > > The files are sorted. > > The first 13 characters are used as a key. > > We get an updated file every week. > > We also keep the previous week file. > > > > I need to compare the keys from the new file to the keys on the file > from last week. If the rest of the records are the same, then I do nothin= g. > If the keys matches but the rest of the records are different, I then hav= e > an update and I will output that record to a new file. > > I was wondering if there is an efficient way to do this in Ruby. Either > any built-in method or an efficient algorithm which I can implement. > > Not sure if you dealing with hashes or arrays but take a look at the '-' > method for arrays, it's blazing fast for comparing even large arrays. > > > > > Thank you > > > > > > -- > > Ruby Student > > > Panagiotis (atmosx) Atmatzidis > > email: atma@convalesco.org > URL: http://www.convalesco.org > GnuPG ID: 0xE736C6A0 > gpg --keyserver x-hkp://pgp.mit.edu --recv-keys 0xE736C6A0 > -- > The wise man said: "Never argue with an idiot. They bring you down to > their level and beat you with experience." > > > > --=20 Ruby Student --f46d043c7c6afc946904ce9cd521 Content-Type: text/html; charset=ISO-8859-7 Content-Transfer-Encoding: quoted-printable Robert,

You stated: "IIRC I even posted a solution outline.&quo= t;
I went to your blog (= http://blog.rubybestpractices.com/) but could not find the item you men= tioned. Would you mind to post the link to it?

Thank you

On Wed, Nov 14, 2012 at 10:= 53 AM, Panagiotis Atmatzidis <atma@convalesco.org> wrote:<= br>
Hello,

On 12 =CD=EF=E5 2012, at 23:21 , Ruby Student <ruby.student@gmail.com> wrote:

> Team,
>
> I have two large files, about 16 million records each.
> The files are sorted.
> The first 13 characters are used as a key.
> We get an updated file every week.
> We also keep the previous week file.
>
> I need to compare the keys from the new file to the keys on the file f= rom last week. If the rest of the records are the same, then I do nothing. = If the keys matches but the rest of the records are different, I then have = an update and I will output that record to a new file.
> I was wondering if there is an efficient way to do this in Ruby. Eithe= r any built-in method or an efficient algorithm which I can implement.

Not sure if you dealing with hashes or arrays but take a look a= t the '-' method for arrays, it's blazing fast for comparing ev= en large arrays.

>
> Thank you
>
>
> --
> Ruby Student


Panagiotis (atmosx) Atmatzidis

email: =A0atma@convalesco.org URL: =A0 =A0http://= www.convalesco.org
GnuPG ID: 0xE736C6A0
gpg --keyserver x-hkp://pg= p.mit.edu --recv-keys 0xE736C6A0
--
The wise man said: "Never argue with an idiot. They bring you down to = their level and beat you with experience."






--
Ruby Student
--f46d043c7c6afc946904ce9cd521--