From: brian <brian@...>
Date: 2005-06-09T11:21:09+09:00
Subject: Re: Newb math question

Thanks, I'll give it a read.  Thanks to Bill too.  I have to say this is 
one of the friendlier communities I've come across.  The BigDecimal is 
working out.  Newb's not getting burned with RTFM's, tends to keep them 
coming back for more.  Thanks again!


Gary Wright wrote:

>
> On Jun 8, 2005, at 8:40 PM, brian wrote:
>
>> Can someone tell me why this code:
>>    puts (9.0-8.9).to_s
>> results in:
>>    0.0999999999999996
>>
>> I'm expecting 0.01 as the result.  Do I have to specify the  
>> precision?  Even with the full precision available on the platform  
>> (Windows XP), the correct answer is still 0.01, not what's  printed.  
>> What's the deal?  I like this language so far, but this  is confusing...
>
>
> This is a floating point math issue not a ruby language issue.
> Some base 10 floating point numbers can not be represented exactly
> when converted to binary (and vice versa). This often leads to small
> differences between the arithmetic results expected by you (in base 10
> format) and calculated by the computer (in binary).
>
> The solution is to look for "close answers" or switch to a different
> encoding (i.e. not binary floating point) for your math.  In many
> cases just using integers with an appropriate scaling factor solves
> the problem.
>
> If anyone is interested in all the gory details of this topic here is
> a good read: "What Every Computer Scientist Should Know About  
> Floating-Point
> Arithmetic".
>
> http://docs.sun.com/source/806-3568/ncg_goldberg.html
>