From: Walter Szewelanczyk <walter@...>
Date: 2004-10-28T03:27:33+09:00
Subject: Re: Rounding error, (100.0 * 9.95).to_i == 994

Dan Janowski wrote:

> I can understand floating point errors, but this one seems too small for 
> that.
> Especially given this:
> 
> irb(main):023:0> v=100.0*9.95
> => 995
> irb(main):024:0> v.to_i
> => 994
> 
> 
> irb(main):016:0> (100.0*9.95).to_i
> => 994
> irb(main):017:0> (100*9.95).to_i
> => 994
> irb(main):018:0> (100*9.95+0.0000000000001).to_i
> => 995
> irb(main):019:0> RUBY_VERSION
> => "1.8.1"
> irb(main):020:0> RUBY_PLATFORM
> => "sparc-solaris2.8"
> 
> 
> irb(main):020:0> (100.0*9.95).to_i
> => 994
> irb(main):021:0> (100*9.95).to_i
> => 994
> irb(main):022:0> (100*9.95+0.0000000000001).to_i
> => 995
> irb(main):023:0> RUBY_VERSION
> => "1.6.8"
> irb(main):024:0> RUBY_PLATFORM
> => "powerpc-darwin7.0"
> 
> Dan
> 
> 

 >ri Float#to_i
------------------------------------------------------------- Float#to_i
      flt.to_i       => integer
      flt.to_int     => integer
      flt.truncate   => integer
------------------------------------------------------------------------
      Returns _flt_ truncated to an +Integer+.

you may want to try

 >ri Float#round
------------------------------------------------------------ Float#round
      flt.round   => integer
------------------------------------------------------------------------
      Rounds _flt_ to the nearest integer. Equivalent to:

         def round

           return floor(self+0.5) if self > 0.0

           return ceil(self-0.5)  if self < 0.0

           return 0.0

         end



         1.5.round      #=> 2

         (-1.5).round   #=> -2


Hope that helps,

Walt

-- 
Walter Szewelanczyk
IS Director
M.W. Sewall & CO.        email : walter@mwsewall.com
259 Front St.            Phone : (207) 442-7994 x 128
Bath, ME 04530           Fax   : (207) 443-6284