From: Jacob Fugal <lukfugl@...>
Date: 2005-06-10T01:27:32+09:00
Subject: Re: Newb math question

On 6/8/05, brian <brian@le-roy.org> wrote:
> Can someone tell me why this code:
>     puts (9.0-8.9).to_s
> results in:
>     0.0999999999999996
> 
> I'm expecting 0.01 as the result.  Do I have to specify the precision?
> Even with the full precision available on the platform (Windows XP), the
> correct answer is still 0.01, not what's printed.  What's the deal?  I
> like this language so far, but this is confusing...
> 
> Brian

Along with all the other good responses, I thought I'd just take the
moment to (once again) state that this behaviour isn't unique to ruby:

    ruby:   "%.16f" % (9.0-8.9)             ==> 0.0999999999999996
    perl:   sprintf("%.16f", 9.0-8.9)       ==> 0.0999999999999996
    python: "%.16f" % (9.0-8.9)             ==> 0.0999999999999996
    php:    sprintf("%.16f", 9.0-8.9)       ==> 0.0999999999999996
    C:      sprintf(dest, "%.16f", 9.0-8.9) ==> 0.0999999999999996

In all of the cases, specifying a lower precision allows the
float->string conversion to round to the "expected" result:

    ruby:   "%.10f" % (9.0-8.9)             ==> 0.1000000000
    perl:   sprintf("%.10f", 9.0-8.9)       ==> 0.1000000000
    python: "%.10f" % (9.0-8.9)             ==> 0.1000000000
    php:    sprintf("%.10f", 9.0-8.9)       ==> 0.1000000000
    C:      sprintf(dest, "%.10f", 9.0-8.9) ==> 0.1000000000

(The source for round_error_c_2 is identical to round_error_c but with
only 10 instead of 16 digits of precision in the printf.)

The major difference is the default precision used by languages in the
float->string conversion. From experimentation (not actually looking
at sources), I came to the following conclusions on the default
precision:

    * ruby and perl both use up to 15 significant digits
    * python uses up to 12 significant digits
    * php uses up to 14 significant digits
    * C uses 6 places after the decimal, always
      (with %f instead of %.10f in the printf)

For all of ruby, perl, python and php I say "up to X significant
digits" because significant 0's at the end are truncated (e.g. (1.0 -
0.5).to_s returns '0.5', not '0.500000000000000') and significant
digit rules are obeyed (significant digits before the decimal point
reduce the number of digits after the decimal point, leading zeros
after the decimal point are not counted).

Jacob Fugal