From: "JeremyWoertink@..." <JeremyWoertink@...>
Date: 2007-11-03T04:31:09+09:00
Subject: Re: Database speed issues

haha, you said "call the % method on them." and then used the "&"
method.
That may work, but how long would it take to build an array from 500GB
of text files? Then every time that no duplicate exists, that file is
added to the array?

but wait... THAT'S AWESOME... hmmmm so I could probably do something
like
IO.readlines("somefile) & IO.readlines("anotherfile) and it would
return the lines that are the same.


The "anotherfile" would have to be an iteration through a complete
network drive of folders, so I could find each folder and then go
through each file inside that folder doing the same thing.

ughh. I hate dealing with these stupid mass quantities of records.
Thanks for the advice :)


~Jeremy


On Nov 2, 12:14 pm, Daniel Waite <rabbitb...@gmail.com> wrote:
> Jeremy Woertink wrote:
> > So, I have a bit of a design problem. I have an application that work,
> > but not as well as I would like it to. My problem is that I had to
> > write an application that checks millions of records against hundreds
> > of millions of records to see if there are any duplicates. The only
> > way I could think to do this is pretty much select from the different
> > tables where the specific column matches my search string. This is
> > really slow! Obviously if it returns something, then  there is a dup,
> > if not, then YAY!
>
> > I know there has to be a better way then just doing a SQL select
> > statement. Any ideas?
>
> I dunno how fast it is compared to your select statement, but if you can
> build  arrays out of the data you want to compare you call the % method
> on them. Example...
>
> irb>  [ 5, 10 ] & [ 10, 15 ]
> => [10]
>
> It works with more complex arrays, too...
>
> irb>  [ [ 5, 10 ], [ 15, 20 ] ] & [ [ 10, 15 ], [ 5, 10] ]
> => [[5, 10]]
>
> Hope that helps...
> --
> Posted viahttp://www.ruby-forum.com/.