Subfolder for Python modules

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
triplus
Posts: 4836
Joined: Mon Dec 12, 2011 4:45 pm

Re: Subfolder for Python modules

Postby triplus » Thu Mar 09, 2017 10:36 pm

looo wrote:

Code: Select all

FreeCAD._useNewStyleImport()


Could work. Or maybe:

Code: Select all

FreeCAD._newModuleImportStyle()


But note that 10 years from now it won't be new anymore. :D That is if current import style support will still be there and this won't be removed altogether. Therefore maybe something like this would work better instead:

Code: Select all

FreeCAD._removeFromSysPath()
FreeCAD._removeModuleFromSysPath()


But beyond giving this suggestions it's your call.
triplus
Posts: 4836
Joined: Mon Dec 12, 2011 4:45 pm

Re: Subfolder for Python modules

Postby triplus » Thu Mar 09, 2017 10:52 pm

Some more thoughts i had. Note that for downstream modules situation isn't perfect. You need to check if new import style is available or not. If it is not available take care of sys path yourself. After you need to take into consideration this fact in your module. When doing the imports. And further if you write documentation on how to use the module from Python you need to provide it in a form it will work for both styles. And you must continue to prefix everything.

P.S. Full migration of the existing modules/macros with some relevance therefore realistically could take a few years.
looo
Posts: 678
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Postby looo » Thu Mar 09, 2017 11:33 pm

P.S. Full migration of the existing modules/macros with some relevance therefore realistically could take a few years.

Until then 100 other things will breake, so at some point I think we are allowed to introduce proper importing.

After you need to take into consideration this fact in your module. When doing the imports. And further if you write documentation on how to use the module from Python you need to provide it in a form it will work for both styles. And you must continue to prefix everything.

This is not true. If a module has not used by another module it can savly move towards the new style. As shown this is possible for 0.17 or older. The only thing not available in 0.16 is the function I have added in the commit and the sys.path which points to the ~/.FreeCAD/Mod. But you can always do this by your own. So I see no problems downstream.
triplus
Posts: 4836
Joined: Mon Dec 12, 2011 4:45 pm

Re: Subfolder for Python modules

Postby triplus » Thu Mar 09, 2017 11:45 pm

Yes developer can do everything and that much is true. But i doubt all downstream module developers supporting FreeCAD 0.16 and up will go through the hassle of supporting the new import style (FreeCAD 0.17+) before dropping FreeCAD 0.16 support completely. That was the point i was trying to make. That we shouldn't blame them for doing that. At least not until around the time FreeCAD 0.18/0.19 is out (given the fact new import style support is added to FreeCAD 0.17).
simonvanderveldt
Posts: 21
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Postby simonvanderveldt » Thu Mar 16, 2017 2:16 pm

@looo Thanks for pointing me here from your PR :) https://github.com/FreeCAD/FreeCAD/pull/602
Which exact part of the stuff discussed in this topic is the PR addressing? Might be good to get a clear list of steps to take defined that can be checked off.

Also, this might be relevant? https://www.python.org/dev/peps/pep-0420/
looo
Posts: 678
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Postby looo » Thu Mar 16, 2017 3:46 pm

The PR has all the non breaking stuff towards support of proper python modules. The PR already has a list with the things changed.

What is missing is:
- port modules to the new style imports
- print warning for modules which use old style
- disable old style.

But as we have discussed in this topic, this will not happen in near future...
simonvanderveldt
Posts: 21
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Postby simonvanderveldt » Thu Mar 16, 2017 5:00 pm

Isn't it a lot easier to just not hard append modules to the Python path like this and instead make the FreeCAD package a namespace package and install it together with it's modules to a normal location.
And then just do a pkgutil.iter_modules("FreeCAD.Modules") to iterate over the existing modules and init them?
looo
Posts: 678
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Postby looo » Thu Mar 16, 2017 5:37 pm

Isn't it a lot easier to just not hard append modules to the Python path like this and instead make the FreeCAD package a namespace package


Isn't that exactly what we are trying to do? It's all about removing the mod-directories from the sys.path to get rid of the naming issues. But as we can't do this right now, we have to slowly get to this goal. The first step is to give new modules the possibility to use proper imports. (PR)
I do not really understand what a namespace package is, but using FreeCAD.__path__ gives us the possibility to have everything available under the FreeCAD name-space. And this is quite simple.

and install it together with it's modules to a normal location.


can you explain this? What is meant by a normal location?

And then just do a pkgutil.iter_modules("FreeCAD.Modules") to iterate over the existing modules and init them?


This should be equivalent to adding the locations of FreeCAD modules to FreeCAD.__path__. I don't see why this should be simpler.
simonvanderveldt
Posts: 21
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Postby simonvanderveldt » Fri Mar 17, 2017 2:45 pm

looo wrote:Isn't that exactly what we are trying to do? It's all about removing the mod-directories from the sys.path to get rid of the naming issues. But as we can't do this right now, we have to slowly get to this goal. The first step is to give new modules the possibility to use proper imports. (PR)

Yeah, this is what I meant with my question. If I understand it correctly right now the modules are first added to the sys.path and then a new function is added to let a module remove itself from sys.path again if it wants to. Is that correct?

Why isn't it possible to simply move a FreeCAD module (or all FreeCAD modules for that matter) to freecad.modules, iterate over the modules in this path using for package in pkgutil.walk_packages(freecad.modules.__path__) and for each of them run their Init.py?

This ties in to the point about __path__ as well:
looo wrote:I do not really understand what a namespace package is, but using FreeCAD.__path__ gives us the possibility to have everything available under the FreeCAD name-space. And this is quite simple.

and install it together with it's modules to a normal location.


can you explain this? What is meant by a normal location?

And then just do a pkgutil.iter_modules("FreeCAD.Modules") to iterate over the existing modules and init them?


This should be equivalent to adding the locations of FreeCAD modules to FreeCAD.__path__. I don't see why this should be simpler.

The PyMOTW explains it best imho https://pymotw.com/2/pkgutil/, sample code lives here if you want to try it out https://bitbucket.org/dhellmann/pymotw/ ... ?at=master

Basically a namespace package is the normal Pythonic way to have a package span multiple directories on the actual filesystem.

By adding this:

Code: Select all

import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)

to either freecad or freecad.modules's __init__.py it's possible to let regular Python modules irrespective of where they are stored on the actual filesystem add functionality (ie FreeCAD modules in this case) to the freecad or freecad.modules package. This means a FreeCAD module could be installed using pip and just live in site-packages like any normal Python package.
It also means if you want to try out some module locally you can just download it, set the PYTHONPATH=/dir/to/local/freecad/module env var and as long as it's got a freecad.modules.<something> package it'll be picked up.
Note that Python 3.3+ will automatically create namespace packages when no __init__.py is present making this even easier.

I hope I managed to explain my questions. It seems to me there are standard Pythonic ways to solve these issues which can easily be leveraged.
I'm probably missing some context though, so if I missed something please correct me :)
Last edited by simonvanderveldt on Sat Mar 18, 2017 10:09 pm, edited 1 time in total.
looo
Posts: 678
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Postby looo » Sat Mar 18, 2017 11:30 am

thanks for the explanation. I have to test this a bit, but actually this would solve quite some stuff.
Note that Python 3.3+ will automatically create namespace packages when no __init.py is present making this even easier.


can you give an example for this. What does it mean for the current structure of freecad? Does it mean, we do not have to use a __init__.py?But where should we place the "__path__ = pkgutil.extend_path(__path__, __name__)" then?
Last edited by looo on Sat Mar 18, 2017 12:04 pm, edited 1 time in total.