Subfolder for Python modules

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Post by looo »

simonvanderveldt wrote: Did you pick these two commits? Or the older ones:
- https://github.com/simonvanderveldt/Fre ... 777c61f9d7
- https://github.com/simonvanderveldt/Fre ... a807728c43

And do you have any code in the different init files?
exactly. This two files + the line I have mentioned above.
the init files have only print statements to see if the files were executed.

I printed the __path__ variables and these had only one path, something like this:
freecad.__path__ -> [.../freecad]
freecad.modules.__path__ -> [.../freecad.modules]

So I think the exdent_path function does not find the freecad.modules in the site-packages.
Have you tried to place a package in dist or site-packages? Maybe you can show me the needed directory-structure.
simonvanderveldt
Posts: 62
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Post by simonvanderveldt »

looo wrote:
simonvanderveldt wrote: Did you pick these two commits? Or the older ones:
- https://github.com/simonvanderveldt/Fre ... 777c61f9d7
- https://github.com/simonvanderveldt/Fre ... a807728c43

And do you have any code in the different init files?
exactly. This two files + the line I have mentioned above.
the init files have only print statements to see if the files were executed.

I printed the __path__ variables and these had only one path, something like this:
freecad.__path__ -> [.../freecad]
freecad.modules.__path__ -> [.../freecad.modules]

So I think the exdent_path function does not find the freecad.modules in the site-packages.
Have you tried to place a package in dist or site-packages? Maybe you can show me the needed directory-structure.
OK, thanks for the info. I'll give it a try myself. Might very well be that the current FreeCAD code doesn't actually expose packages from your system site-packages, I ran into the same issue with the PYTHONPATH variable which is hardcoded to "" :?

I'll post an update here once I've tried it.
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Post by looo »

Code: Select all

>>> pkgutil.extend_path([], "freecad.modules")
['/home/lo/anaconda/envs/fc_test/Mod/freecad/modules']
>>> pkgutil.extend_path([], "freecad")
['/home/lo/anaconda/envs/fc_test/Mod/freecad', '/home/lo/anaconda/envs/fc_test/lib/python3.5/site-packages/freecad']
Ipython gives the same result. So I think the pkgutil doesn't support submodule names.
If this is the case my suggestion is to not use the modules dir, simple directly import from freecad.

ps.:
Maybe I was wrong. Placing the extend_path in both directories (freecad, freecad.modules) work for extending the __path__ of freecad.modules.
So in this situation we can extend both directories extern. The question is, do we really need the "modules" dir. I would vote against because it makes the import statements longer....

https://github.com/looooo/FreeCAD/commi ... 992c2c8ccf

ps2:
Maybe we can go with this structure?

Code: Select all

from freecad import core # core=[app, gui, part, part_design, fem,...]
from freecad.modules import 3rdParty # 3rdParty=[assembly2, drawing_dimension, ...]
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Post by looo »

wow, just tested this with a 3rdParty-module. It gets pip-installable with 5 lines: https://github.com/looooo/freecad_frame ... e/setup.py

This new structure will not only solve the name-clash problem. With this it's also possible to set the absolute-paths of icons. So no name clashes also there. And I have to admit, lower case imports make really sense :D
simonvanderveldt
Posts: 62
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Post by simonvanderveldt »

looo wrote:ps.:
Maybe I was wrong. Placing the extend_path in both directories (freecad, freecad.modules) work for extending the __path__ of freecad.modules.
So in this situation we can extend both directories extern. The question is, do we really need the "modules" dir. I would vote against because it makes the import statements longer....

https://github.com/looooo/FreeCAD/commi ... 992c2c8ccf
Since we want this to be compatible with both Python 2 and 3 I think the simplest fix is to add a __init__.py to both the freecad and the freecad.modules package with the pkgutil.extend_path call in it, just like you did.
Once the migration to Python3 is done, these __init__.py files can be removed.

I've added this to the 2nd commit https://github.com/simonvanderveldt/Fre ... e6dcdfdf31
looo wrote:I have cherry picked your two commits regarding the namespace-support. There was also one line from the arch-commit necessary to make FreeCAD start: https://github.com/simonvanderveldt/Fre ... sts.txt#L8
Thanks, good find! I've fixed this as well, part of the same commit.
looo wrote:wow, just tested this with a 3rdParty-module. It gets pip-installable with 5 lines: https://github.com/looooo/freecad_frame ... e/setup.py

This new structure will not only solve the name-clash problem. With this it's also possible to set the absolute-paths of icons. So no name clashes also there. And I have to admit, lower case imports make really sense :D
So you did get it working?
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Post by looo »

simonvanderveldt wrote: Since we want this to be compatible with both Python 2 and 3 I think the simplest fix is to add a __init__.py to both the freecad and the freecad.modules package with the pkgutil.extend_path call in it, just like you did.
Once the migration to Python3 is done, these __init__.py files can be removed.
I don't think this will work without the pkgutil.extend_path in python3. At least in my case this was necessary.
With python3 it's not necessary to place a __init__.py in the extern freecad.modules (eg in site-package/freecad/modules/test/__init__.py). Maybe that is needed with python2.
So you did get it working?
with py3 it works. I will test with python2 to understand the differences...
simonvanderveldt
Posts: 62
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Post by simonvanderveldt »

looo wrote:I don't think this will work without the pkgutil.extend_path in python3. At least in my case this was necessary.
With python3 it's not necessary to place a __init__.py in the extern freecad.modules (eg in site-package/freecad/modules/test/__init__.py). Maybe that is needed with python2.
With Python3 there explicitly shoulnd't be an __init__.py file for a package to be automatically picked up as a namespace package, see https://www.python.org/dev/peps/pep-0420/#specification for more details.

For Python 2 __init__.py is needed to "create"/mark a Python package.
looo wrote:with py3 it works. I will test with python2 to understand the differences...
Awesome! I've just the changes with Python 2 and it's still working, so should be fine now :)
Last edited by simonvanderveldt on Fri Mar 31, 2017 7:37 am, edited 1 time in total.
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Post by looo »

this diff was necessary to make my module python2 compatible. Maybe the __init__.py in the icons dir isn't necessary, but this way it was easier to get the icons installed with dist-utils.
Now I understand the problem with python2: These additional __init__.py files are never used. They seem to be only necessary for the pkgutil to detect the paths...

I think it's time to prepare for a PR. Please remove the function I added to allow packages to remove the path, as this is not necessary anymore. Maybe a first PR should only include the change in FreeCADInit.py and FreeCADGuiInit.py. So everyone can experiment with the new imports.
ian.rees
Posts: 696
Joined: Sun Jun 15, 2014 3:28 am
Contact:

Re: Subfolder for Python modules

Post by ian.rees »

Nice work guys! -Ian-
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Post by looo »

distutils/setuptools can not install two extensions to the same place. Tried this with this template https://github.com/looooo/TemplateExtension . Once calling pip with this extension my previously installed package is gone.

ps.: my fault. I used the same name in the setup step. So pip removed the package... So this should be no problem.

ps2: @microelly2 there was a topic about modules standardization some time ago. I think it was about __meta__.py or something similar. Is there any documentation about meta-data use in 3rd-party modules?
Post Reply