From: Mark Hubbart <discordantus@...>
Date: 2005-04-01T05:44:46+09:00
Subject: Re: Integer/Float oddity

On Fri, 1 Apr 2005 00:59:45 +0900, michael.b.masi@gmail.com
<michael.b.masi@gmail.com> wrote:
> i'm assuming that since this behaviour is exhibited in ruby and the
> rest of the above "other languages" and NOT xbasic that it is really
> due to an idiosyncracy of xbasic that the "right" answer comes out.
> 
> so the question becomes, what is different about xbasic's internal
> numeric representation that allows it to give a correct answer to this
> seemingly simple number manipulation?
> 
> horati0

While the behavior of xbasic in this circumstance would seem to be
"correct", and is probably useful in simple calculations, it would
make it unsuitable for serious calculations. The "correct" answer is
actually not. This is due to the way floating point operations are
calculated by computers.

Without going into too much depth, lets just say that floats are
stored as fractions in the computer's memory. Specifically, as
fractions with denominators that are powers of 2; (x/2^n). Just as
some fractions don't translate precisely to decimal numbers (1/2
becomes 0.5, but 1/7 becomes 0.142857142857142...), many decimal
numbers don't translate precisely to binary numbers (0.5 becomes
0.1b2, but 0.3 becomes 0.00100110011b2...). This causes rounding
errors.

To avoid this, use rational numbers, which are precise fractions:
require 'mathn'
    ==> true
num = 115/100
    ==>23/20
num -= 1 
    ==>3/20
num * 100 
    ==>15

cheers,
Mark