From: Gary Wright <gwtmp01@...>
Date: 2007-10-23T12:14:45+09:00
Subject: Re: computers + floating-point = a chore


On Oct 22, 2007, at 10:30 PM, Yossef Mendelssohn wrote:
> Yes, I know the troubles with floating point. I think what I found
> most interesting was a number that claimed to be 19.76 when it
> actually wasn't, simply because it's more convenient (?) to display it
> as such.

Well, the culprit is actually Float#to_s, which has it's own idea of
how to convert floating point numbers to text.  In particular it  
defaults
to 15 digits of precession and also works hard to strip trailing zero's:

$ irb
irb(main):001:0> a = 19.76
=> 19.76
irb(main):002:0> a.to_s
=> "19.76"
irb(main):003:0> sprintf "%#.15g", a
=> "19.7600000000000"
irb(main):004:0> sprintf "%#.32g", a
=> "19.760000000000001563194018672220"
irb(main):005:0>


You can see from this session that 15 digits of precision is not
enough to 'show' that a isn't really 19.76.

What was surprising to me is that Float#to_s doesn't just use a standard
sprintf format string:

static VALUE
flo_to_s(flt)
     VALUE flt;
{
     char buf[32];
     double value = RFLOAT(flt)->value;
     char *p, *e;

     if (isinf(value))
         return rb_str_new2(value < 0 ? "-Infinity" : "Infinity");
     else if(isnan(value))
         return rb_str_new2("NaN");

     sprintf(buf, "%#.15g", value); /* ensure to print decimal point */
     if (!(e = strchr(buf, 'e'))) {
         e = buf + strlen(buf);
     }
     if (!ISDIGIT(e[-1])) { /* reformat if ended with decimal point  
(ex 111111111111111.) */
         sprintf(buf, "%#.14e", value);
         if (!(e = strchr(buf, 'e'))) {
             e = buf + strlen(buf);
         }
     }
     p = e;
     while (p[-1]=='0' && ISDIGIT(p[-2]))
         p--;
     memmove(p, e, strlen(e)+1);
     return rb_str_new2(buf);
}


Gary Wright