Zsh Mailing List Archive
Messages sorted by: Reverse Date, Date, Thread, Author

Re: Floating point calculus error...

On Sun, Aug 25, 2002 at 17:22:20 +0200, Patrick Aussems wrote:
> I was doing some shell scripts that were supposed to add CPU usages from
> the ps output... But I got puzzled when I got an answer with a 17th
> decimal while adding numbers with only 1 decimal... Strange isn't it?

The numbers are internally represented in binary and 0.4 isn't exactly
representable in binary... But here, this is a problem with zsh, which
gives an additional decimal digit. You can see with:

$ echo $[0.4]
0.40000000000000002

I think that it should give the minimum number of decimals in such a way
that when converting back to binary, one should get the same result.

-- 
Vincent Lefèvre <vincent@xxxxxxxxxx> - Web: <http://www.vinc17.org/> - 100%
validated (X)HTML - Acorn Risc PC, Yellow Pig 17, Championnat International
des Jeux Mathématiques et Logiques, TETRHEX, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
Messages sorted by: Reverse Date, Date, Thread, Author