From: Logan Capaldo <logancapaldo@...>
Date: 2006-01-31T07:05:29+09:00
Subject: Re: Is this a floating point precision problem?


On Jan 30, 2006, at 4:28 PM, Eero Saynatkari wrote:

> On 2006.01.31 05:09, Todd S. wrote:
>> I'm using the Matrix module (matrix.rb)  to help place a vertex into
>> local coordinates and then back into world coordinates.  Under  
>> certain
>> situations the transformation to and from local space corrupts the
>> data...
>>
>> This matrix...
>>     -0.0108226964530337  -0.224934679233174 -0.974313737622412
>>       0.0468247818757306    0.973187905351297 -0.225194894880513
>>     -0.998844486916645      0.0480592442327619 0.0
>>
>> will invert to this...
>>   64.0                               0.0
>> -0.998844486916645
>>     0.0                               1.0
>> 0.0480592442327619
>>   -0.974313737622412 -0.225194894880513 -2.40312135813741e-18
>>
>>
>> starting with vertex: 0.015525, -0.28474399, 0.53221899
>> multiplyng by the inverse matrix and then back again by the matrix
>> itself will not yield the same number as it should but instead
>> returns... 0.00555723611485535, -0.24161810434515, -0.4739174731596
>>
>> So am I seeing strange behavior due to the extreamly small (e-18)  
>> number
>> in the matrix?
>
> Without pretending to have any idea about mathematical operations,
> it is quite likely that floating-pointedness is causing the problem.
> You cannot really even count on 1.8 and 2.1 - 0.3 being the same.
>
>> If so, how can I set the precision so that I don't see this error?
>
> BigDecimal in the stdlib is, as far as I know, lossless but of
> course somewhat slower if you are doing anything very intense.
>
>
>
> E
>

Alternatively, if you know how precise you need it to be, and its not  
too precise, and speed is important you can use fixed-point math  
instead of floating point.