Re: Floating point calculus error...

[Erik Trulsson <ertr1013@xxxxxxxxxxxxx>]

On Sun, Aug 25, 2002 at 05:22:20PM +0200, Patrick Aussems wrote:
> Hi all...
> 
> I was doing some shell scripts that were supposed to add CPU usages from
> the ps output... But I got puzzled when I got an answer with a 17th
> decimal while adding numbers with only 1 decimal... Strange isn't it?

Not strange at all.  That sort of things is to be expected when using
floating point.

> 
> This behaviour can be observed by doing the following command:
> 
> zyk@Amok:/home/zyk> echo $[0.4+0.4]
> 
> The value returned is: 0.80000000000000004 which is obviously not the
> correct answer... Okay, floating points numbers are inaccurate, but
> still...

Since neither 0.4 or 0.8 cannot be represented exactly in a binary
floating point format one shouldn't expect the ouput to be quite
correct.  
The internal form that "0.4" gets converted to is actually some other
number which has almost but not exactly the value 0.4 When you then add
two such numbers you will get a number that is almost but not quite
0.8.

This is similar to the behaviour of most pocket calculators where
trying to calculate 3*(1/3) usually results in 0.9999999999 instead of
the mathematically correct 1.


> 
> Patrick
> 
> PS: I don't have the time to read the mailing lists, so I don't know if
> this problem has already been reported...

It is not really a problem. It is just the way floating point is.

> 
> PS2: Please CC me on replies, I'm not subscribed to this mailing list...
> 
> 

-- 
<Insert your favourite quote here.>
Erik Trulsson
ertr1013@xxxxxxxxxxxxx