[Fixed] [regression] values in spreadsheet have too many digits

[uwestoehr]

there is a regression compared to FC 0.18.4 in the spreadsheet WB:


FC 0.18.4:

value in spreadsheet - FC 0.18

FreeCAD_2fmQxyRCAt.png (14 KiB) Viewed 1330 times

FC 0.19:

value in spreadsheet - FC 0.19

FreeCAD_UUpneNchtS.png (18.07 KiB) Viewed 1330 times

In both cases I use 3 digits in the FC settings. I remember we had this issue already and it was meanwhile fixed.

However, the many digits are annoying because editing values in spreadsheets is slowed down.

OS: Windows 10 (10.0)
Word size of OS: 64-bit
Word size of FreeCAD: 64-bit
Version: 0.19.19972 (Git)
Build type: Release
Branch: master
Hash: 3461a5349bbde87dbf1914a8bd1b4165399e036d
Python version: 3.6.8
Qt version: 5.12.1
Coin version: 4.0.0a
OCC version: 7.3.0
Locale: German/Germany (de_DE)

[chrisb]

IIRC the outcome of the discussion was not to restrict the input of spreadsheet values to the general number of decimals. Anyone having the topic bookmarked?

[chrisb]

https://forum.freecadweb.org/viewtopic.php?f=3&t=34564

[uwestoehr]

IIRC the outcome of the discussion was not to restrict the input of spreadsheet values to the general number of decimals.

Yes, and Werner did this with git commit 58aa9b183. Thus I wonder why we have the issue again that was already solved.
Werner, do you have an idea?

.

[wmayer]

The commit to blame is git commit a1417c5ffa52. In version 0.18 we had this function to display a number:

Code: Select all

std::string NumberExpression::toString() const
{
    std::stringstream s;
    s << std::setprecision(std::numeric_limits<double>::digits10 + 1) << quantity.getValue();

    /* Trim of any extra spaces */
    //while (s.size() > 0 && s[s.size() - 1] == ' ')
//        s.erase(s.size() - 1);

   return s.str();
}

After some iterations of code refactoring we now have this function

Code: Select all

void NumberExpression::_toString(std::ostream &ss, bool,int) const
{
    ss << std::setprecision(std::numeric_limits<double>::digits10 + 2) << getValue();

    /* Trim of any extra spaces */
    //while (s.size() > 0 && s[s.size() - 1] == ' ')
//        s.erase(s.size() - 1);
}

The difference is made with digits10 + 2 vs. digits10 + 1.

The commit log message offers no explanation why 2 over 1 should be preferred now and in all other places where std::numeric_limits<double>::digits10 is used we add 1 and nowhere else 2.

So, I will replace the 2 with 1 again...

[wmayer]

git commit 338bf2ca3

[uwestoehr]

git commit 338bf2ca3

many thanks!

[chennes]

ETA: Nevermind, I think I understand now!

@wmayer, can you help me understand why we are using digits10+1 everywhere, instead of max_digits10? I see the references to both in your comment on this commit.

[wmayer]

According to https://en.cppreference.com/w/cpp/types ... x_digits10 the right way would be max_digits10 (= 17 for double) instead of digits10 + 1 (= 16 for double) when serializing/deserializing a double in text form.

If we want to change this then we must carefully test it (and write some unit tests) to make sure we don't cause a regression again. At least when changing it we must change it everywhere and my above commit was exactly about this where we had an inconsistency because of the digits10 + 2.

[chennes]

Thanks -- went back and re-read that URL you put in your code comments from Boost (https://www.boost.org/doc/libs/1_63_0/l ... tants.html) and it helped me a lot.

I guess we really have two independent problems in FreeCAD. For pure serialization to a file we definitely want max_digits10 so we are guaranteed not to lose precision. For use in the UI, though, that's problematic because a lot of the time the last digit displayed will be garbage. Not 100% of the time, but some of the time, just because of the way binary->decimal works. So by splitting the difference in the UI (that is, using either digits10+1 or max_digits10-1), we provide access to nearly all possible floating point values, without displaying the potentially garbage digit. I think that seems like a reasonable compromise, if we don't want to get complicated and figure out for any given number how many digits to write.