Subfolder for Python modules

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
looo
Posts: 788
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Postby looo » Sat Mar 18, 2017 11:57 am

Seems I haven't understand this fully:

I am testing with this structure:

Code: Select all

~/Schreibtisch/test-package:
    |-__init__.py
    |-freecad
        |-__init__.py

now doing this:

Code: Select all

>>> import sys, pkgutil
>>> sys.path.append("~/Schreibtisch")
>>> pkgutil.extend_path([], "freecad")
[]


doesn't find the module extension...

regarding https://pymotw.com/2/pkgutil/:
extend_path() scans sys.path for directories that include a subdirectory named for the package given as the second argument.


it shouldn't be empy.

but:

Code: Select all

>>> pkgutil.extend_path([], "numpy")
['/usr/lib/python2.7/dist-packages/numpy']
simonvanderveldt
Posts: 61
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Postby simonvanderveldt » Sat Mar 18, 2017 9:31 pm

looo wrote:thanks for the explanation. I have to test this a bit, but actually this would solve quite some stuff.
Note that Python 3.3+ will automatically create namespace packages when no __init.py is present making this even easier.


can you give an example for this. What does it mean for the current structure of freecad? Does it mean, we do not have to use a __init__.py?But where should we place the "__path__ = pkgutil.extend_path(__path__, __name__)" then?

Indeed it means that it isn't necessary to add a __init__.py file to make use of namespace packages when using Python 3.3+.
There might be other reasons to have one though, in that case the pkgutil.extend_path() call would still need to be added.

looo wrote:Seems I haven't understand this fully:

I am testing with this structure:

Code: Select all

~/Schreibtisch/test-package:
    |-__init__.py
    |-freecad
        |-__init__.py

now doing this:

Code: Select all

>>> import sys, pkgutil
>>> sys.path.append("~/Schreibtisch")
>>> pkgutil.extend_path([], "freecad")
[]


doesn't find the module extension...

A quick guess would be you'd need to use sys.path.append("~/Schreibtisch/test-package") or you'd have to use test-package.freecad as package name

I've created a sample so it's a bit easier to see. See https://github.com/simonvanderveldt/fre ... es-example
Note that it's written for Python 3.3 or higher.

It was also useful for me because I've never used namespace packages in combination with subpackages.
Luckily pkutil.walk_packages() as well as pkgutil.iter_modules() returns a boolean if a submodule is a package thus allowing us to easily get all subpackages :)
This does mean it's necessary to add a __init__.py to the actual modules otherwise they're not being picked up when importing freecad.modules.
simonvanderveldt
Posts: 61
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Postby simonvanderveldt » Sun Mar 19, 2017 5:34 pm

I'm having a go at implementing what I described above in the current FreeCAD codebase, started with the non-gui modules just to give it a go and it seems to be working pretty well.

Code: Select all

FreeCAD 0.17, Libs: 0.17R10506 (Git)
© Juergen Riegel, Werner Mayer, Yorik van Havre 2001-2017
  #####                 ####  ###   #### 
  #                    #      # #   #   #
  #     ##  #### ####  #     #   #  #   #
  ####  # # #  # #  #  #     #####  #   #
  #     #   #### ####  #    #     # #   #
  #     #   #    #     #    #     # #   #  ##  ##  ##
  #     #   #### ####   ### #     # ####   ##  ##  ##

Submodule freecad.modules.arch, is a package: True
Submodule freecad.modules.startpage, is a package: True
During initialization the error No module named init occurred in freecad.modules.startpage
Submodule freecad.modules.test, is a package: True
Submodule freecad.modules.tux, is a package: True
During initialization the error No module named init occurred in freecad.modules.tux
[FreeCAD Console mode <Use Ctrl-D (i.e. EOF) to exit.>]
>>> FreeCAD.getExportType()
{'dae': 'importDAE', 'html': 'importWebGL', 'obj': 'importOBJ', 'ifc': 'importIFC'}
>>> FreeCAD.getImportType()
{'dae': 'importDAE', 'obj': 'importOBJ', 'zip': 'importSH3D', '3ds': 'import3DS', 'ifc': 'importIFC', 'FCStd': 'FreeCAD'}
>>> import freecad.modules
>>> freecad.modules.__path__
['/home/simon/src/simonvanderveldt/freecad-build/Mod/freecad/modules']


This is produced by the following WIP code https://github.com/simonvanderveldt/Fre ... ce-modules. The import code has gotten a lot simpler.
This branch only contains the changes on the importer side, the actual modules still need to be changed, for now I just did so on my local filesystem.

Code: Select all

~/s/s/freecad-build ❯ tree -d Mod/freecad
Mod/freecad
└── modules
    ├── arch
    │   ├── Dice3DS
    │   ├── Presets
    │   └── Resources
    │       ├── icons
    │       ├── translations
    │       └── ui
    ├── startpage
    ├── test
    │   └── Gui
    │       └── Resources
    │           ├── icons
    │           └── translations
    └── tux


Note that I'm using freecad.modules as package name because I have no experience with combining this with a built-in (C based) package like FreeCAD and because package names should actually be all lower case according to PEP-8.
Next up is doing the same for the GUI init.
GUI seems to work as well :)

Code: Select all

Log: Init: Creating Gui::Application and QApplication
Log: Local server 'FreeCAD' started
Log: OpenGL version 3.0 or higher is present
Log: Run Gui init script
Log: Init: Running FreeCADGuiInit.py start script...
Log: Init: Searching for modules...
Log: Submodule freecad.modules.arch, is a package: True
Log: Initializing freecad.modules.arch
Err: During initialization the error No module named Arch_rc occurred in freecad.modules.arch
Log: Submodule freecad.modules.startpage, is a package: True
Log: Initializing freecad.modules.startpage
Err: During initialization the error No module named init_gui occurred in freecad.modules.startpage
Log: Submodule freecad.modules.test, is a package: True
Log: Initializing freecad.modules.test
Err: During initialization the error No module named init_gui occurred in freecad.modules.test
Log: Submodule freecad.modules.tux, is a package: True
Log: Initializing freecad.modules.tux
Log: Init: Loading FreeCAD GUI
looo
Posts: 788
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Postby looo » Sun Mar 19, 2017 6:57 pm

this is looking promising.

I have some questions:
- Does the imports work like before? (inside the packages)
- How would you deal with shared libraries? Currently we would add the dirs containing python packages to the FreeCAD.so (__path__ of that modul). Would this be possible with pkgutil?
- Does it solve the naming issue?
simonvanderveldt
Posts: 61
Joined: Tue Mar 14, 2017 2:11 pm

Re: Subfolder for Python modules

Postby simonvanderveldt » Sun Mar 19, 2017 8:44 pm

looo wrote:this is looking promising.

I have some questions:
- Does the imports work like before? (inside the packages)

Do you mean something like import DraftTools,DraftGui,Arch_rc,Arch,Draft_rc?

- How would you deal with shared libraries? Currently we would add the dirs containing python packages to the FreeCAD.so (__path__ of that modul). Would this be possible with pkgutil?

Where exactly does this happen? Do you mean this?
AFAIK shared libraries simply need to live in the `LD_LIBRARY_PATH` and that's it.

- Does it solve the naming issue?

Yeah, of course :) This is just normal Python stuff, everything that lives inside a package is namespaced within that package as it should be. It should never be relevant to package b what you call the modules in package a.
The only reason there were potential name-clashing issues was the abuse of adding every individual module/package's directory to the sys.path, which will definitely get you into trouble.
looo
Posts: 788
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Postby looo » Sun Mar 19, 2017 9:11 pm

Do you mean something like import DraftTools,DraftGui,Arch_rc,Arch,Draft_rc?


exactly. I am just wondering what the benefit from your method is. All in all I think it's aiming at the same thing.
The biggest benefit from your method is the possibility to have extensions anywhere. This is great as 3rd party libraries could be installed with pip.

Where exactly does this happen? Do you mean this?
AFAIK shared libraries simply need to live in the `LD_LIBRARY_PATH` and that's it.


I don't think a shared library has to live in a fixed destination. At startup we import FreeCAD (the shared library) and add the module base directories to it's __path__ variable. Then all modules are importable via FreeCAD.

import: https://github.com/FreeCAD/FreeCAD/blob ... nit.py#L34
__path__: https://github.com/FreeCAD/FreeCAD/blob ... it.py#L114

Yeah, of course :) This is just normal Python stuff, everything that lives inside a package is namespaced within that package as it should be. It should never be relevant to package b what you call the modules in package a.


ok, got it. I was just wondering if there wouldn't have to be a transition phase if we would take your approach. But I think there is no other way then moving all modules to proper python imports. This is:

Code: Select all

from FreeCAD import Module.Submodule


So we need this transition-phase anyway.
User avatar
bernd
Posts: 3924
Joined: Sun Sep 08, 2013 8:07 pm
Location: Zürich, Switzerland

Re: Subfolder for Python modules

Postby bernd » Sun Mar 19, 2017 9:27 pm

How should the imports changed in FEM to use the new system. Is this still valid ? https://github.com/looooo/FreeCAD/commi ... _issue_fem
looo
Posts: 788
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Postby looo » Sun Mar 19, 2017 9:36 pm

No we now decided to go for:

Code: Select all

from FreeCAD import Fem.module


This should work with current master.
looo
Posts: 788
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Postby looo » Mon Mar 20, 2017 10:25 am

@simonvanderveldt

It would be nice to have your suggested way to handle imports in the master. But I think there are two things that shouldn't be changed:

- import FreeCAD -> should import the shared library (we can work around by adding a dir called FreeCAD and there place a "from _FreeCAD import *")
- use FreeCAD instead of freecad for namespace packages. I think this has historic reasons. Also there is no suggestion from python to use lower case. PySide and PyQt also use upper case...
looo
Posts: 788
Joined: Mon Nov 11, 2013 5:29 pm

Re: Subfolder for Python modules

Postby looo » Mon Mar 20, 2017 10:49 am

@bernd :
I think my answer was a bit too short :D
to make fem use the new imports there have to be done the following things:
- prefix the shared library: like it is done in this branch: https://github.com/looooo/FreeCAD/blob/ ... s.txt#L295
- create a __init__.py in the Fem base dir which import all from the shared-library:

Code: Select all

from _Fem import *

- and then change all the imports
- once everything is setup you can remove the Fem dir from the sys.path with placing this line in the Init.py

Code: Select all

FreeCAD._importFromFreeCAD('Fem')

If I have time I will try to write some documentation for this.