From: Tom Copeland <tom@...>
Date: 2006-04-29T01:02:01+09:00
Subject: Re: Ruby Drops

> Just had a thought...  Something similar _has_ been done before, but  
> targeted at a different application.  Take a  look at services like  
> MOSS <http://www.cs.berkeley.edu/~aiken/moss.html>, which was  
> designed for plagiarism detection -- e.g. finding similar 
> code across  
> projects.  There is a nice paper linked that talks about how they go  
> about doing it.

That is an interesting paper, thanks for the link!  He lists a couple of
requirements:

1) whitespace independence - CPD has this for several languages (C, C++,
Java) since for those languages uses JavaCC-generated parsers that
discard whitespace.  For other languages (Ruby, PHP) it also discards
whitespace but does it a bit more clunkily, at a higher level in the
framework.

2) noise suppression - yup, this is important since you don't want to
catch little matches like "x.each do |y|".  CPD allows you to set the
minimum match size; I usually start at about 100 and work my way lower
from there.

3) position independence - since you don't want moving things around to
affect the analysis.  CPD mostly has this, I think :-)

Of course, the hard problem is fixing the duplicates once you find them;
that can be a delicate job.  

[SHAMELESS PLUG] If you're interested in reading more about CPD, I've
got a chapter on it in my book:

http://pmdapplied.com/

Yours,

Tom