From: Paul Brannan <pbrannan@...>
Date: 2003-08-21T04:32:42+09:00
Subject: Re: Punctuation as noise

On Thu, Aug 21, 2003 at 02:49:19AM +0900, Hal E. Fulton wrote:
> I'd be curious to see people's results on a large
> corpus of code (Ruby, Perl, etc.).

On 27K lines of integration test code (Ruby), I get:
  Signal/noise: 458705/119801 =  3.8

On 10K lines of C++ extension code that the above tests test:
  Signal/noise: 163676/40803 =  4.0

On 1.3K lines of unit test code (C++), I get:
  Signal/noise: 28549/8335 =  3.4

On 544 lines of unit test code (Perl), I get:
  Signal/noise: 9365/4350 =  2.2

On 4K lines of vim scripts that I regularly use, I get:
  Signal/noise: 500/73 =  6.8

On the 63K lines of Ruby code that come with 1.8:
  Signal/noise: 912431/218858 =  4.2

On the 60K lines of C extension code that comes with 1.8:
  Signal/noise: 823062/251038 =  3.3

On the 71K lines of C code that make up the Ruby 1.8 interpreter:
  Signal/noise: 904042/313624 =  2.9

> I have noticed an odd effect already, though. The 
> symbol/alpha ratio is fairly low (1-2) for smaller 
> programs and larger (4-6) for larger programs. I've
> tried it on sources ranging from 10 lines to 2000
> lines.

This is probably the result I would expect.  If I'm writing a throwaway
script, I don't care what it looks like, so long as it works.

BTW, I played golf with your script and got:

  noise = 0
  alpha = 0
  punc  = '\',./`=[]\\;~!@#$%^&*()_+{}|:"<>?-'
  white = " \t\r\n"
  while buf = $stdin.read(512) do
    punc_count  = buf.count(punc)
    white_count = buf.count(white)
    noise += punc_count
    alpha += buf.size - white_count - punc_count
  end

  ratio = alpha/noise.to_f
  puts "Signal/noise: #{alpha}/#{noise} = #{'%4.1f' % ratio}"

It runs significantly faster this way.

Paul