From: doug meharry <doug@...>
Date: 2006-05-11T02:09:49+09:00
Subject: Re: Numeric precision problem

doug meharry wrote:
> Greetings all.
> 
> I'm having a little problem with numeric precision and being a NOOB to 
> Ruby, I'm not quite sure where I am going wrong.
> 
> When I perform the following calculation:
> 
> n = ((9703.0 * 16346204482307500.0) + 1).to_f
> d = (65537.0).to_f
> 
> res = (n / d).to_f
> 
> Ruby is returning a value of 249,419,480,327.0
> 
> However, the more precise number I am expecting is 249,419,480,328.784
> 
> How do I do this to get the more precise value?
> 
> Thanks.
> 
> Doug

Oops!  Never mind.  I some of the numbers in my preceeding message were 
slightly off.  This produced the expected results.

n = ((9704.0 * 1684654692600.0) + 1).to_f
d = (65537.0).to_f

res = (n / d).to_f

Ruby is returning a value of 249,445,185,727.0 which is what I also 
think it should be.





-- 
Posted via http://www.ruby-forum.com/.