From: Simon Strandgaard <0bz63fz3m1qt3001@...>
Date: 2003-08-21T06:31:37+09:00
Subject: Re: Punctuation as noise

On Thu, 21 Aug 2003 05:20:16 +0900, Hal E. Fulton wrote:
> 
> Anyway, I'd be curious as to other people's results on this little
> experiment.

I have just made a small script which can measure signal2noise
alphanum / (total-blank).  I am also interested in hearing what your
signal2noise ratio are ?

Try the script at the bottom of this posting.

--
Simon Strandgaard


noise.rb => 75% 

> ruby noise.rb noise.rb
alphas = 571
blanks = 156
others = 189
alpha/(alpha+others) = 0.751315789473684
>

readme's => 85%

> ruby noise.rb REA*
alphas = 1864
blanks = 540
others = 319
alpha/(alpha+others) = 0.853870819972515
> ls -la REA*
-rw-r--r--  1 neoneye  neoneye  1634 10 Aug 23:51 README
-rw-r--r--  1 neoneye  neoneye   259  3 Jul 07:46 README.ADMIN
-rw-r--r--  1 neoneye  neoneye   830 19 Aug 00:54 README.CVS
>


signal2noise in my AEditor project source => 85%

> wc -l *.rb | grep total
    3663 total
> ruby noise.rb test.non_test/*.rb
alphas = 54765
blanks = 16721
others = 9551
alpha/(alpha+others) = 0.851498849430935
>


signal2noise in my AEditor project unittests => 80%

> wc -l *.rb | grep total
    4998 total
> ruby noise.rb test.test/*.rb
alphas = 99522
blanks = 21985
others = 24864
alpha/(alpha+others) = 0.800106121267667
>

signal2noise in a c++ project, .cpp => 79%  .h => 86%

> ruby noise.rb *.cpp
alphas = 51331
blanks = 13813
others = 13171
alpha/(alpha+others) = 0.795804781247093
> ruby noise.rb *.h
alphas = 31507
blanks = 7416
others = 4806
alpha/(alpha+others) = 0.86765070360477
>


The noise.rb scrip is here:

> expand -t2 noise.rb
def calc_history(res, file)
  data = nil
  File.open(file, "r"){|f| data = f.read}
  ary = data.unpack('C*')
  ary.each{|i| res[i]+=1}
  res
end

def count_freq(history, *matchs)
  count = [0] * matchs.size
  count_bad = 0
  history.each_index{|ascii|
    ok = false
    matchs.each_index{|i|
      if matchs[i].index(ascii) != nil
        count[i] += history[ascii]
        ok = true
      end
    }
    count_bad += history[ascii] unless ok
  }
  count << count_bad
  count
end

# scan files
result = [0]*256
ARGV.each{|file| calc_history(result, file) }

# count frequency
match_alphanum = ('a'[0]..'z'[0]).to_a + ('A'[0]..'Z'[0]).to_a + ('0'[0]..'9'[0]).to_a
match_blanks = [32, 13, 10, 9]
alphas, blanks, others = count_freq(result, match_alphanum, match_blanks)

# output result
totals = alphas + others
ratio = alphas.to_f / totals.to_f
puts <<MSG
alphas = #{alphas}
blanks = #{blanks}
others = #{others}
alpha/(alpha+others) = #{ratio}
MSG
>