[ruby-core:86324] [Ruby trunk Bug#5273][Closed] Float#round returns the wrong floats for higher precision

Issue #5273 has been updated by mame (Yusuke Endoh).

Status changed from Open to Closed
Assignee set to marcandre (Marc-Andre Lafortune)

I think there are two issues in this one ticket: `3.0e-31.round(31) #=> 3.0000000000000003e-31` and `2.5e-22.round(22) #=> 2.0e-22`.

I created another ticket (#14635) about the former issue with some investigation.

I think that we cannot fix the latter issue for the reason that Naruse-san said.  `2.5e-22` represents the range of `0.249999999999999953374475e-21` and `0.250000000000000000394250e-21`, so the implementation cannot determine which `0.2e-38` and `0.3e-38` are preferable.

So unfortunately, `2.5e-39.round(39)` would be `2.0e-39`.  Anyway, `2.0000000000000002e-39` is a wrong result, I think.

I'm closing this ticket because there are no other issue in this ticket.  Marc-Andre, let me know if I am wrong.

----------------------------------------
Bug #5273: Float#round returns the wrong floats for higher precision
https://bugs.ruby-lang.org/issues/5273#change-71245

* Author: marcandre (Marc-Andre Lafortune)
* Status: Closed
* Priority: Normal
* Assignee: marcandre (Marc-Andre Lafortune)
* Target version: 
* ruby -v: r33186
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN
----------------------------------------
Float#round returns the wrong floats for higher precision.

Rounding can be off:
    2.5e-22.round(22) # => 2.0e-22

It can also return values that have extra decimals:
    2.5e-31.round(31) # => 3.0000000000000003e-31
    2.5e-36.round(36) # => 2.9999999999999998e-36

Both errors can occur at the same time too:
2.5e-39.round(39) # => 2.0000000000000002e-39

I believe these errors occur only for ndigits >= 22. For ndigits > 22, 10**(-ndigits) is not an exact Float (as 5 ** 23 won't fit in the 53 bits of a double's mantissa). For ndigits < 22, there should be enough bits left to avoid rounding errors.

For ndigits >= 22, the algorithm to use should be similar to the one used to convert doubles to string.

Hopefully, this is the last bug for #round with a given precision.



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>