From: Robert Klemme <bob.news@...>
Date: 2005-09-07T16:41:29+09:00
Subject: Re: Sorted arrays

ruby@danb64.com wrote:
> Thanks for the help.  I switched it to use a Hash, and that performed
> MUCH better.  The data is not coming from any SQL server--it is just
> a flat file.  The Ruby script processes it into a format so that it
> can be BCP'd into MS SQL Server.  And when the table that it gets
> BCP'd into has a unique constraint on the MD5 hash field--hence the
> need for me to eliminate the duplicate values beforehand.  Granted, I
> could have changed the process of importing the data so that this
> unique constraint was not necessary, but it seemed better to me to
> eliminate the duplicates using Ruby in this case.

If you're on MS SQL Server there's another option: you can use an index
with IGNORE DUPLICATES: that way you don't have to preprocess the file at
all:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/tsqlref/ts_create_64l4.asp

Kind regards

    robert