From: "葉志立 YIP Chi Lap  [Beta] " <beetah@...>
Date: 2008-01-09T04:35:54+09:00
Subject: Precision of Float

Hi,

     I understand that we cannot expect a high level of precision
when dealing with floats, but this is really weird to me: Float#to_s
and Float#to_i seem to be inconsistent, and I'm getting only one
decimal digit of precision.  Is it the problem of my instance of Ruby
or something more serious?  The run session shows it all:

[powermeow:~/tryout] beta% irb
 >> (8.2*100).to_s
=> "820.0"
 >> (8.2*100).to_i
=> 819
 >> (8.2*100).to_int
=> 819
 >> 0.upto(100) { |x| y=x/10.0; printf("%d\t", (y*100).to_i) }
0	10	20	30	40	50	60	70	80	90	100	110	120	130	140	150	160	170	180	190	 
200	210	220	229	240	250	260	270	280	290	300	310	320	330	340	350	360	 
370	380	390	400	409	420	430	440	450	459	470	480	490	500	509	520	530	 
540	550	560	570	580	590	600	610	620	630	640	650	660	670	680	690	700	 
710	720	730	740	750	760	770	780	790	800	810	819	830	840	850	860	869	 
880	890	900	910	919	930	940	950	960	969	980	990	1000	=> 0
 >> ^D
[powermeow:~/tryout] beta% uname -a
Darwin powermeow.local 9.1.0 Darwin Kernel Version 9.1.0: Wed Oct 31  
17:48:21 PDT 2007; root:xnu-1228.0.2~1/RELEASE_PPC Power Macintosh
[powermeow:~/tryout] beta% ruby --version
ruby 1.8.6 (2007-09-24 patchlevel 111) [universal-darwin9.0]
[powermeow:~/tryout] beta% irb
 >> (8.2).class
=> Float
 >> (8.2*100).class
=> Float
 >> require 'bigdecimal'
=> true
 >> a = BigDecimal.new("8.2")
=> #<BigDecimal:363300,'0.82E1',8(8)>
 >> a.to_f
=> 8.2
 >> a*100
=> #<BigDecimal:36185c,'0.82E3',4(20)>
 >> (a*100).to_f
=> 820.0
 >> (a*100).to_i
=> 820
 >> ^D
[powermeow:~/tryout] beta% rails --version
Rails 2.0.2


Regards,
   Beta.