From: Ruby Student <ruby.student@...>
Date: 2012-10-23T03:21:23+09:00
Subject: Looking for suggestions processing and comparing 2 very large files

--f46d043be1c695b1cb04cca9ebb4
Content-Type: text/plain; charset=ISO-8859-1

Team,

Every week I get a large file, over 50 millions records with record length
> 150 chars. These files can actually be over 15GB.
I need to take the new file and compare it against the one from the
previous week.
Reading the files into two arrays would make the process a bit easier, but
the files are too large and when I try using arrays, the process crashes
with out of storage messages.

I am looking for suggestions on how to efficiently perform the following
process:


   1. Compare each record from the file from this week against last week
   file
   2. If every record are the same, do nothing or just indicate so: *SAM*
   3. If there is any duplicate records on the new file, output the record
   to a file of dups
   4. If there are any new record, (records found on the new file, but not
   on last week file) output: *INS* followed by the record
   5. If there is a record which is found on last week file (old file) but
   not on this week file, output: *DEL* followed by the record
   6. If there is a record with the same key (the first 13 chars) on both
   files, but the rest of the record is different, output: *UPD* followed
   by the record

Hey, I can do all of the above doing reading each record from both files
and do different type of comparison/match, but I was wondering if there is
an efficient way to do this. I was looking for suggestions.

Thank you

-- 
Ruby Student

--f46d043be1c695b1cb04cca9ebb4
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Team,<div><br></div><div>Every week I get a large file, over 50 millions re=
cords with record length &gt; 150 chars. These files can actually be over 1=
5GB.</div><div>I need to take the new file and compare it against the one f=
rom the previous week.</div>
<div>Reading the files into two arrays would make the process a bit easier,=
 but the files are too large and when I try using arrays, the process crash=
es with out of storage messages.</div><div><br></div><div>I am looking for =
suggestions on how to efficiently perform the following process:</div>
<div><br></div><div><ol><li>Compare each record from the file from this wee=
k against last week file</li><li>If every record are the same, do nothing o=
r just indicate so: <b>SAM</b></li><li>If there is any duplicate records on=
 the new file, output the record to a file of dups</li>
<li>If there are any new record, (records found on the new file, but not on=
 last week file) output: <b>INS</b> followed by the record</li><li>If there=
 is a record which is found on last week file (old file) but not on this we=
ek file, output: <b>DEL</b> followed by the record</li>
<li>If there is a record with the same key (the first 13 chars) on both fil=
es, but the rest of the record is different, output: <b>UPD</b>=A0followed =
by the record</li></ol></div><div>Hey, I can do all of the above doing read=
ing each record from both files and do different type of comparison/match, =
but I was wondering if there is an efficient way to do this. I was looking =
for suggestions.=A0</div>
<div><br></div><div>Thank you</div><div><div><br></div>-- <br>Ruby Student<=
br>
</div>

--f46d043be1c695b1cb04cca9ebb4--