Subfolder for Python modules

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
triplus
Veteran
Posts: 9471
Joined: Mon Dec 12, 2011 4:45 pm

Re: Subfolder for Python modules

Post by triplus »

looo wrote:

Code: Select all

FreeCAD._useNewStyleImport()
Could work. Or maybe:

Code: Select all

FreeCAD._newModuleImportStyle()
But note that 10 years from now it won't be new anymore. :D That is if current import style support will still be there and this won't be removed altogether. Therefore maybe something like this would work better instead:

Code: Select all

FreeCAD._removeFromSysPath()
FreeCAD._removeModuleFromSysPath()
But beyond giving this suggestions it's your call.
triplus
Veteran
Posts: 9471
Joined: Mon Dec 12, 2011 4:45 pm

Re: Subfolder for Python modules

Post by triplus »

Some more thoughts i had. Note that for downstream modules situation isn't perfect. You need to check if new import style is available or not. If it is not available take care of sys path yourself. After you need to take into consideration this fact in your module. When doing the imports. And further if you write documentation on how to use the module from Python you need to provide it in a form it will work for both styles. And you must continue to prefix everything.

P.S. Full migration of the existing modules/macros with some relevance therefore realistically could take a few years.
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Post by looo »

P.S. Full migration of the existing modules/macros with some relevance therefore realistically could take a few years.
Until then 100 other things will breake, so at some point I think we are allowed to introduce proper importing.
After you need to take into consideration this fact in your module. When doing the imports. And further if you write documentation on how to use the module from Python you need to provide it in a form it will work for both styles. And you must continue to prefix everything.
This is not true. If a module has not used by another module it can savly move towards the new style. As shown this is possible for 0.17 or older. The only thing not available in 0.16 is the function I have added in the commit and the sys.path which points to the ~/.FreeCAD/Mod. But you can always do this by your own. So I see no problems downstream.
triplus
Veteran
Posts: 9471
Joined: Mon Dec 12, 2011 4:45 pm

Re: Subfolder for Python modules

Post by triplus »

Yes developer can do everything and that much is true. But i doubt all downstream module developers supporting FreeCAD 0.16 and up will go through the hassle of supporting the new import style (FreeCAD 0.17+) before dropping FreeCAD 0.16 support completely. That was the point i was trying to make. That we shouldn't blame them for doing that. At least not until around the time FreeCAD 0.18/0.19 is out (given the fact new import style support is added to FreeCAD 0.17).
simonvanderveldt
Posts: 62
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Post by simonvanderveldt »

@looo Thanks for pointing me here from your PR :) https://github.com/FreeCAD/FreeCAD/pull/602
Which exact part of the stuff discussed in this topic is the PR addressing? Might be good to get a clear list of steps to take defined that can be checked off.

Also, this might be relevant? https://www.python.org/dev/peps/pep-0420/
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Post by looo »

The PR has all the non breaking stuff towards support of proper python modules. The PR already has a list with the things changed.

What is missing is:
- port modules to the new style imports
- print warning for modules which use old style
- disable old style.

But as we have discussed in this topic, this will not happen in near future...
simonvanderveldt
Posts: 62
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Post by simonvanderveldt »

Isn't it a lot easier to just not hard append modules to the Python path like this and instead make the FreeCAD package a namespace package and install it together with it's modules to a normal location.
And then just do a pkgutil.iter_modules("FreeCAD.Modules") to iterate over the existing modules and init them?
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Post by looo »

Isn't it a lot easier to just not hard append modules to the Python path like this and instead make the FreeCAD package a namespace package
Isn't that exactly what we are trying to do? It's all about removing the mod-directories from the sys.path to get rid of the naming issues. But as we can't do this right now, we have to slowly get to this goal. The first step is to give new modules the possibility to use proper imports. (PR)
I do not really understand what a namespace package is, but using FreeCAD.__path__ gives us the possibility to have everything available under the FreeCAD name-space. And this is quite simple.
and install it together with it's modules to a normal location.
can you explain this? What is meant by a normal location?
And then just do a pkgutil.iter_modules("FreeCAD.Modules") to iterate over the existing modules and init them?
This should be equivalent to adding the locations of FreeCAD modules to FreeCAD.__path__. I don't see why this should be simpler.
simonvanderveldt
Posts: 62
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Post by simonvanderveldt »

looo wrote:Isn't that exactly what we are trying to do? It's all about removing the mod-directories from the sys.path to get rid of the naming issues. But as we can't do this right now, we have to slowly get to this goal. The first step is to give new modules the possibility to use proper imports. (PR)
Yeah, this is what I meant with my question. If I understand it correctly right now the modules are first added to the sys.path and then a new function is added to let a module remove itself from sys.path again if it wants to. Is that correct?

Why isn't it possible to simply move a FreeCAD module (or all FreeCAD modules for that matter) to freecad.modules, iterate over the modules in this path using for package in pkgutil.walk_packages(freecad.modules.__path__) and for each of them run their Init.py?

This ties in to the point about __path__ as well:
looo wrote:I do not really understand what a namespace package is, but using FreeCAD.__path__ gives us the possibility to have everything available under the FreeCAD name-space. And this is quite simple.
and install it together with it's modules to a normal location.
can you explain this? What is meant by a normal location?
And then just do a pkgutil.iter_modules("FreeCAD.Modules") to iterate over the existing modules and init them?
This should be equivalent to adding the locations of FreeCAD modules to FreeCAD.__path__. I don't see why this should be simpler.
The PyMOTW explains it best imho https://pymotw.com/2/pkgutil/, sample code lives here if you want to try it out https://bitbucket.org/dhellmann/pymotw/ ... ?at=master

Basically a namespace package is the normal Pythonic way to have a package span multiple directories on the actual filesystem.

By adding this:

Code: Select all

import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
to either freecad or freecad.modules's __init__.py it's possible to let regular Python modules irrespective of where they are stored on the actual filesystem add functionality (ie FreeCAD modules in this case) to the freecad or freecad.modules package. This means a FreeCAD module could be installed using pip and just live in site-packages like any normal Python package.
It also means if you want to try out some module locally you can just download it, set the PYTHONPATH=/dir/to/local/freecad/module env var and as long as it's got a freecad.modules.<something> package it'll be picked up.
Note that Python 3.3+ will automatically create namespace packages when no __init__.py is present making this even easier.

I hope I managed to explain my questions. It seems to me there are standard Pythonic ways to solve these issues which can easily be leveraged.
I'm probably missing some context though, so if I missed something please correct me :)
Last edited by simonvanderveldt on Sat Mar 18, 2017 10:09 pm, edited 1 time in total.
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Post by looo »

thanks for the explanation. I have to test this a bit, but actually this would solve quite some stuff.
Note that Python 3.3+ will automatically create namespace packages when no __init.py is present making this even easier.
can you give an example for this. What does it mean for the current structure of freecad? Does it mean, we do not have to use a __init__.py?But where should we place the "__path__ = pkgutil.extend_path(__path__, __name__)" then?
Last edited by looo on Sat Mar 18, 2017 12:04 pm, edited 1 time in total.
Post Reply