From: Marc-Andre Lafortune <ruby-core@...>
Date: 2011-09-16T15:29:00+09:00
Subject: [ruby-core:39575] [Ruby 1.9 - Bug #5273] Float#round returns the wrong floats for higher precision


Issue #5273 has been updated by Marc-Andre Lafortune.


Yui NARUSE wrote:
> I can't your English of this line, I think you are saying you can't understand [ruby-core:39443].

No, I understood what you are saying. I meant the tour argument is not valid justification for the current behavior.

You say that 3.0e-31.round(31) should be 3.0/10**31 == 3.0000000000000003e-31.

I say it should be 3.0e-31.

If we follow your argument, then it would be ok if `Float("3.0e-31") == 3.0000000000000003e-31` but that is clearly not so.

I say that `any_float.round(prec) == Float("#{some_int}.0e-#{prec}")`, by definition (for positive prec).

Do you have any reason why you think that 3.0000000000000003e-31 is preferable to 3.0e-31? Do you truly think it is the expected result and the most useful result?

Note that in addition to commonsense and mathematics, javascript also agrees with me:

    3.0e-31.toPrecision(1) == 3.0000000000000003e-31.toPrecision(1) == "3e-31"



----------------------------------------
Bug #5273: Float#round returns the wrong floats for higher precision
http://redmine.ruby-lang.org/issues/5273

Author: Marc-Andre Lafortune
Status: Assigned
Priority: Normal
Assignee: Marc-Andre Lafortune
Category: core
Target version: 1.9.4
ruby -v: r33186


Float#round returns the wrong floats for higher precision.

Rounding can be off:
    2.5e-22.round(22) # => 2.0e-22

It can also return values that have extra decimals:
    2.5e-31.round(31) # => 3.0000000000000003e-31
    2.5e-36.round(36) # => 2.9999999999999998e-36

Both errors can occur at the same time too:
2.5e-39.round(39) # => 2.0000000000000002e-39

I believe these errors occur only for ndigits >= 22. For ndigits > 22, 10**(-ndigits) is not an exact Float (as 5 ** 23 won't fit in the 53 bits of a double's mantissa). For ndigits < 22, there should be enough bits left to avoid rounding errors.

For ndigits >= 22, the algorithm to use should be similar to the one used to convert doubles to string.

Hopefully, this is the last bug for #round with a given precision.


-- 
http://redmine.ruby-lang.org