From: Chuck Remes <cremes.devlist@...>
Date: 2009-07-18T14:27:13+09:00
Subject: Re: [Q] removing array duplicates where a subset is unique


On Jul 17, 2009, at 7:55 PM, David A. Black wrote:

> Hi --
>
> On Sat, 18 Jul 2009, Chuck Remes wrote:
>> David and Joel,
>>
>> you both provided the same solution. I will test this to see what  
>> kind of performance I get. It will be hell on memory, but I assumed  
>> any solution likely would be. (And Joel, I have presorted the array  
>> prior to removing the dupes so I have already taken care of the  
>> ordering issue.)
>
> I believe the version you had originally, where you do a mapping of
> the whole array, will typically use much more memory than the hash
> version. Let's say your original array has 1000 inner arrays, with 10
> that are considered unique. The mapping will be a new array, also of
> 1000 elements. The hash will have 10 key/value pairs -- thus much
> smaller.

Oh yes, my version had terrible execution performance and memory  
performance. I was trying to figure out how to use a hash but did not  
make the leap to the ||= construction on my own.  I knew I was missing  
something obvious... all of your rapid responses proved it.

FYI, the dedup code you provided performs quite admirably. I'll take a  
look at its memory footprint when I get in the office Monday and  
report back.

cr