Sketcher formula displays incorrect values

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Post Reply
anthonyg
Posts: 51
Joined: Mon Oct 25, 2021 10:30 pm

Sketcher formula displays incorrect values

Post by anthonyg »

OS: Debian GNU/Linux 11 (bullseye) (XFCE/lightdm-xsession)
Word size of OS: 64-bit
Word size of FreeCAD: 64-bit
Version: 0.19.
Build type: Release
Python version: 3.9.7
Qt version: 5.15.2
Coin version: 4.0.0
OCC version: 7.5.1
Locale: English/United States (en_US)

On Sketcher, when I enter a division, it can display a wrong value, see image bellow where it displays 4.83 instead of 4.825:
sketcher_value.png
sketcher_value.png (23.53 KiB) Viewed 1425 times
Also, after entering a value and editing the formula it can display a wrong value:
sketcher_value_2.png
sketcher_value_2.png (19.7 KiB) Viewed 1425 times
The value I entered was 8.79mm / 2, and now it's showing 8.789999999999999mm / 2.
Last edited by anthonyg on Thu Oct 28, 2021 8:32 am, edited 1 time in total.
User avatar
Shalmeneser
Veteran
Posts: 9474
Joined: Wed Dec 23, 2020 12:04 am
Location: Fr

Re: Sketcher formula displays incorrect values

Post by Shalmeneser »

Please follow the banner in Help forum : version, ...
openBrain
Veteran
Posts: 9034
Joined: Fri Nov 09, 2018 5:38 pm
Contact:

Re: Sketcher formula displays incorrect values

Post by openBrain »

How many decimals did you set in the preferences?
anthonyg
Posts: 51
Joined: Mon Oct 25, 2021 10:30 pm

Re: Sketcher formula displays incorrect values

Post by anthonyg »

openBrain wrote: Wed Oct 27, 2021 8:37 pm How many decimals did you set in the preferences?
I have these settings:
units.png
units.png (26.93 KiB) Viewed 1337 times
openBrain
Veteran
Posts: 9034
Joined: Fri Nov 09, 2018 5:38 pm
Contact:

Re: Sketcher formula displays incorrect values

Post by openBrain »

anthonyg wrote: Thu Oct 28, 2021 8:28 am I have these settings:
units.png
So how surprising it is that is displays 4.83 instead of 4.825 ??? :)
anthonyg
Posts: 51
Joined: Mon Oct 25, 2021 10:30 pm

Re: Sketcher formula displays incorrect values

Post by anthonyg »

So how surprising it is that is displays 4.83 instead of 4.825 ??? :)
OK, I though there would be something wrong because on the drawing itself you can see it displays 4.825.
openBrain
Veteran
Posts: 9034
Joined: Fri Nov 09, 2018 5:38 pm
Contact:

Re: Sketcher formula displays incorrect values

Post by openBrain »

anthonyg wrote: Thu Oct 28, 2021 8:36 am OK, I though there would be something wrong because on the drawing itself you can see it displays 4.825.
Yep. There are some inconsistencies/weirdness in FreeCAD where user setting is sparsely applied.
Especially for Sketcher, it has a hardcoded precision that value is "6 significant digits". We discussed that already but no decision was made about a change : https://forum.freecadweb.org/viewtopic.php?f=3&t=61740

Regarding your other issue, it is due to some classical floating-point number representation. It's a bit weird but not so easy to tackle. ;)
User avatar
jonasb
Posts: 162
Joined: Tue Dec 22, 2020 7:57 pm

Re: Sketcher formula displays incorrect values

Post by jonasb »

Regarding your other issue, it is due to some classical floating-point number representation. It's a bit weird but not so easy to tackle. ;)
Python has some nice and detailed explanation: https://docs.python.org/3/tutorial/floa ... tion-error
Post Reply