Re: zsh converts a floating-point number to string with too much precision

[Vincent Lefevre <vincent@xxxxxxxxxx>]

On 2019-12-21 08:47:36 +0000, Stephane Chazelas wrote:
> OK, I think I see what you mean.
> 
> So on a system (with a compiler) where C doubles are implemented
> as IEEE 754 double precision, both 1.1 and 1.1000000000000001
> are represented as the same binary double (whose exact value is
> 1.100000000000000088817841970012523233890533447265625).
> 
> So you're saying echo $((1.1000000000000001)) and echo $((1.1))
> should output 1.1, because even though 1.1000000000000001 is
> closer to that value than 1.1000000000000000, zsh should pick
> the latter because people prefer to see shorter number
> representations and in that case it doesn't matter which one we
> pick as both lead to the same double.

I now remember that our Handbook of Floating-Point Arithmetic covers
this issue (Section 4.9.2.1 "Output conversion: from radix 2 to
radix 10" in the 2nd edition). But it just gives references

--------------------------------------------------------------------
"[...] Steele and White designed an algorithm for that. Their
algorithm was later improved by Burger and Dybvig [86], and by
Gay [211]. Gay's code is available for anyone to use, and is
very robust.(17) Faster yet more complex algorithms have been
introduced by Loitsch [394] and Andrysco et al. [16]. [...]"

(17) At the time of writing this book, it can be obtained at
http://www.netlib.org/fp/ (file dtoa.c).
--------------------------------------------------------------------

and Burger and Dybvig's high-level algorithm.

-- 
Vincent Lefèvre <vincent@xxxxxxxxxx> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)