From: Robert Klemme <shortcutter@...>
Date: 2006-05-11T01:55:57+09:00
Subject: Re: Numeric precision problem

2006/5/10, doug meharry <doug@dmeharry.com>:
> Greetings all.
>
> I'm having a little problem with numeric precision and being a NOOB to
> Ruby, I'm not quite sure where I am going wrong.
>
> When I perform the following calculation:
>
> n = ((9703.0 * 16346204482307500.0) + 1).to_f
> d = (65537.0).to_f
>
> res = (n / d).to_f
>
> Ruby is returning a value of 249,419,480,327.0
>
> However, the more precise number I am expecting is 249,419,480,328.784
>
> How do I do this to get the more precise value?

You can use rational:

>> require 'rational'
=> true
>> n = (9703 * 16346204482307500) + 1
=> 158607222091829672501
>> d = 65537
=> 65537
>> res = (n / d)
=> 2420117217630188
>> res.to_f
=> 2.42011721763019e+15

Floating point arithmetic always has some numeric issues.

robert
--
Have a look: http://www.flickr.com/photos/fussel-foto/