From: David Vallner <david@...>
Date: 2006-10-01T03:21:23+09:00
Subject: Re: Integer division with / - request explanation of behavior

--------------enig36EC0771B2612EAB387A253F
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Jeremy Tregunna wrote:
>> Actually, integers are more precise than IEEE floating points
>> since 2.2 - 1.2 !=3D 1.0 :) s/precise/accurate !
>=20
> Not on all platforms. For instance, in Ocaml, it's true:
>=20
> # 2. -. 1. =3D 1.;;
> - : bool =3D true
>=20

Your example works because integers are rational IEEE floating point
numbers. Problems arise when you use a number that's a rational decimal
number, but an irrational IEEE binary floating point one. Those aren't
precise no matter what storage size you use.

# 2.2 -. 1.2;;
- : float =3D 1.0000000000000002
# (2.2 -. 1.2) !=3D 1.0;;
- : bool =3D true

For this example, Ruby's rounding earlier loses the imprecision.

Ripping off a tutorial article on IEEE floats[1] produces an example
that manifests in Ruby (also in OCaml):

irb(main):001:0> a =3D 0.0
=3D> 0.0
irb(main):002:0> 10000.times {
irb(main):003:1*   a +=3D 0.0001
irb(main):004:1> }
=3D> 10000
irb(main):005:0> a
=3D> 0.999999999999906
irb(main):006:0>

David Vallner

[1] http://support.microsoft.com/default.aspx?scid=3Dkb;EN-US;q42980


--------------enig36EC0771B2612EAB387A253F
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)

iD8DBQFFHrWly6MhrS8astoRAitAAJwJSZxvTwqKGraKwoxLVnwxIgzbWQCffXCj
rizPem9shLANFmM4J9oOVjg=
=PIfZ
-----END PGP SIGNATURE-----

--------------enig36EC0771B2612EAB387A253F--