Re: Floating point calculus error...

[David Kågedal <davidk@xxxxxxxxxxxxxx>]

Vincent Lefevre <vincent@xxxxxxxxxx> writes:

> On Sun, Aug 25, 2002 at 17:22:20 +0200, Patrick Aussems wrote:
>> I was doing some shell scripts that were supposed to add CPU usages from
>> the ps output... But I got puzzled when I got an answer with a 17th
>> decimal while adding numbers with only 1 decimal... Strange isn't it?
>
> The numbers are internally represented in binary and 0.4 isn't exactly
> representable in binary... But here, this is a problem with zsh, which
> gives an additional decimal digit. You can see with:
>
> $ echo $[0.4]
> 0.40000000000000002
>
> I think that it should give the minimum number of decimals in such a way
> that when converting back to binary, one should get the same result.

But it does.

$ echo $[.40000000000000002]
0.40000000000000002

The point is, as you say, that 0.4 can't be represented as an exact
binary floating-point number, no matter how large the representation
is.  Just like 1/3 can't be represented as a decimal floating-point
number.

-- 
David Kågedal