From: "Todd S." <tgate@...>
Date: 2006-01-31T05:09:33+09:00
Subject: Is this a floating point precision problem?

I'm using the Matrix module (matrix.rb)  to help place a vertex into 
local coordinates and then back into world coordinates.  Under certain 
situations the transformation to and from local space corrupts the 
data...

This matrix...
    -0.0108226964530337  -0.224934679233174 -0.974313737622412
      0.0468247818757306    0.973187905351297 -0.225194894880513
    -0.998844486916645      0.0480592442327619 0.0

will invert to this...
  64.0                               0.0 
-0.998844486916645
    0.0                               1.0 
0.0480592442327619
  -0.974313737622412 -0.225194894880513 -2.40312135813741e-18


starting with vertex: 0.015525, -0.28474399, 0.53221899
multiplyng by the inverse matrix and then back again by the matrix 
itself will not yield the same number as it should but instead 
returns... 0.00555723611485535, -0.24161810434515, -0.4739174731596

So am I seeing strange behavior due to the extreamly small (e-18) number 
in the matrix?

If so, how can I set the precision so that I don't see this error?

-- 
Posted via http://www.ruby-forum.com/.