Obsoleting material handling bash scripts with python

A forum to discuss the implementation of a good Materials system in FreeCAD
User avatar
PrzemoF
Posts: 2660
Joined: Fri Jul 25, 2014 4:52 pm
Contact:

Re: Obsoleting material handling bash scripts with python

Postby PrzemoF » Sat Feb 23, 2019 1:28 pm

Looks like a file can be opened with encoding specified:

Code: Select all

with open(oldfile, 'r', encoding='utf-8') as infile:
I'm not sure yet how it will affect yaml
User avatar
PrzemoF
Posts: 2660
Joined: Fri Jul 25, 2014 4:52 pm
Contact:

Re: Obsoleting material handling bash scripts with python

Postby PrzemoF » Mon Feb 25, 2019 7:11 pm

Edited!
I think we should aim for getting the material cards in yaml together with python3. Making it compatible with python2 is a pain and it would be used for a short time only. I think we have a problem with the existing content - I'm getting things like:

Code: Select all

- Meta:
    AuthorAndLicense: !!python/str '(c) 2019 Uwe Stöhr (CC-BY 3.0)'
    CardName: PA6-Generic
- General:
    Name: !!python/unicode 'Polyamide 6'
Random thoughts
- dump the comments in yaml file completely,
- use Meta section to store info like "created by",
- add API version for future use? API could be simply FreeCAD version
User avatar
PrzemoF
Posts: 2660
Joined: Fri Jul 25, 2014 4:52 pm
Contact:

Re: Obsoleting material handling bash scripts with python

Postby PrzemoF » Mon Feb 25, 2019 8:50 pm

That's a good starting point for me. Then your commit. This is in the master branch:

Code: Select all

$ file *.FCMat
ABS.FCMat:                   ASCII text
ABS-Generic.FCMat:           ASCII text
Acrylic-Glass-Generic.FCMat: UTF-8 Unicode text
CalculiX-Steel.FCMat:        ASCII text
Concrete-Generic.FCMat:      ASCII text
Glass-Generic.FCMat:         UTF-8 Unicode text
None.FCMat:                  ASCII text
PA6-Generic.FCMat:           UTF-8 Unicode text
PET-Generic.FCMat:           UTF-8 Unicode text
PLA.FCMat:                   ASCII text
PLA-Generic.FCMat:           UTF-8 Unicode text
PP-Generic.FCMat:            UTF-8 Unicode text
PTFE-Generic.FCMat:          UTF-8 Unicode text
PVC-Generic.FCMat:           UTF-8 Unicode text
Steel-Generic.FCMat:         ASCII text
TEMPLATE.FCMat:              ASCII text
Wood-Generic.FCMat:          ASCII text
Edit: Looks like ASCII text is a UTF-8 subset, so it' not the source of the problem. I'm trying to figure out why some strings are encoded as python/str and some as python/unicode in the same file.
User avatar
bernd
Posts: 8751
Joined: Sun Sep 08, 2013 8:07 pm
Location: Zürich, Switzerland

Re: Obsoleting material handling bash scripts with python

Postby bernd » Fri Mar 01, 2019 7:47 pm

strange because the cards are all written by your bash scripts on my linux debian machine. I wonder why they have different encoding ...

I am not sure about when Python 2 support will be dropped in FreeCAD I do not thing in 0.19 already. But I would like to switch in 0.19 to this yaml cards.
Jee-Bee
Posts: 1980
Joined: Tue Jun 16, 2015 10:32 am
Location: Netherlands

Re: Obsoleting material handling bash scripts with python

Postby Jee-Bee » Fri Mar 01, 2019 9:15 pm

If python 2 will be supported while python don't support it ( https://pythonclock.org) would be strange...
User avatar
PrzemoF
Posts: 2660
Joined: Fri Jul 25, 2014 4:52 pm
Contact:

Re: Obsoleting material handling bash scripts with python

Postby PrzemoF » Fri Mar 01, 2019 9:56 pm

bernd wrote:
Fri Mar 01, 2019 7:47 pm
strange because the cards are all written by your bash scripts on my linux debian machine. I wonder why they have different encoding ...

I am not sure about when Python 2 support will be dropped in FreeCAD I do not thing in 0.19 already. But I would like to switch in 0.19 to this yaml cards.
The encoding is OK. I'm digging into ascii/utf-8/python2-3 and yaml. If file contains only ascii characters it's a valid utf-8 file as well. I tried quite a few combinations, yet python2/yaml exports strings like "szkło" (unicode/utf-8) as !!python/str and I have no idea why. But I learned a lot :D
User avatar
PrzemoF
Posts: 2660
Joined: Fri Jul 25, 2014 4:52 pm
Contact:

Re: Obsoleting material handling bash scripts with python

Postby PrzemoF » Sat Mar 02, 2019 8:50 pm

The problem is with existing cards or the way the existing cards are read. Test from python 2.7.15:

Code: Select all

>>> y = yaml.load("""\n    - szkło\n    - test\n    """) ; print(yaml.dump(y))
["szk\u0142o", test]

>>> with open('y', 'w') as f:
...     yaml.dump(y, f, default_flow_style=False, allow_unicode=True)
and the result file looks perfect:

Code: Select all

$ cat y
- szkło
- test

Code: Select all

$ file y
y: UTF-8 Unicode text
Same code works with python 3.7.2 with identical (at least visually) results. Same result from FreeCAD python console. Also reading the file using yaml.load works fine on 2.7.15. I also did a test with glass card from the FreeCAD python console and it's all good.

We might need to edit existing cards and it might all work fine. Unless FreeCAD is making something strange "behind the scene". I'm talking about things like:

Code: Select all

pythonopen = open
or

Code: Select all

unicode = str
I should know that soon.
User avatar
PrzemoF
Posts: 2660
Joined: Fri Jul 25, 2014 4:52 pm
Contact:

Re: Obsoleting material handling bash scripts with python

Postby PrzemoF » Sat Mar 02, 2019 10:00 pm

This code works fine from FreeCAD python console:

Code: Select all

>>> with open('x-fc.yml') as f:
...         x = yaml.load(f)
... 
>>> with open('x-fc.yml', 'w') as f:
...         yaml.dump(x, f, default_flow_style=False, allow_unicode=True)
x-fc.yml is utf-8 file without any !!python/srt or !!python/unicode tags.
User avatar
PrzemoF
Posts: 2660
Joined: Fri Jul 25, 2014 4:52 pm
Contact:

Re: Obsoleting material handling bash scripts with python

Postby PrzemoF » Sat Mar 02, 2019 10:42 pm

Side note: we currently redefine python "open" and other build-in functions. Should we instead change it to fc_open across FreeCAD code? Renaming build-in functions is very confusing thing. I'm aware I'm asking for a huge thing, but unless there is a good reason to do it (C++/python hooks?) I'd avoid it even if it means changing it in the whole FreeCAD code.
User avatar
bernd
Posts: 8751
Joined: Sun Sep 08, 2013 8:07 pm
Location: Zürich, Switzerland

Re: Obsoleting material handling bash scripts with python

Postby bernd » Mon Mar 04, 2019 8:50 am

PrzemoF wrote:
Sat Mar 02, 2019 10:42 pm
Side note: we currently redefine python "open" and other build-in functions. Should we instead change it to fc_open across FreeCAD code? Renaming build-in functions is very confusing thing. I'm aware I'm asking for a huge thing, but unless there is a good reason to do it (C++/python hooks?) I'd avoid it even if it means changing it in the whole FreeCAD code.
To start a disscusion, a topic in developer part of the forum makes sence IMHO.
cron