Zsh Mailing List Archive
Messages sorted by: Reverse Date, Date, Thread, Author

Re: arithmetic anomaly



On 2021-03-25 3:54 p.m., Roman Perepelitsa wrote:
On Thu, Mar 25, 2021 at 11:44 PM Ray Andrews <rayandrews@xxxxxxxxxxx <mailto:rayandrews@xxxxxxxxxxx>> wrote:

    On 2021-03-25 1:40 p.m., Lawrence Velázquez wrote:
    >
    > The bad news is that *all* your figures -- your numerators, Brett's
    > numerators, and the denominators -- are actually incorrect for
    > level=15 and higher.
    Sheesh, what gives?  Seems no accuracy beyond 15 digits?


Indeed. 64-bit floating point numbers usually have 15.95 decimal digits of precision. Depending on the compiler and how you write your code, you might get higher precision but in practice many projects turn this off because having borderline non-deterministic results of computation sucks.

That's for sure.  I'd rather the result was truncated.  Why show garbage?  Nuts, why compute garbage?






Messages sorted by: Reverse Date, Date, Thread, Author