Workbench-Starterkit refactor

[wmayer]

Can you confirm that the init.py/Init.py files are not called when FreeCAD is imported from python? Or is this another wrong assumption of me?

No I can't and it's another wrong assumption of you. Just look above:

Code: Select all

python -c "import FreeCAD"

will also print the message. But you really must use the print() statement because FreeCAD.Console.PrintMessage() at the moment doesn't write to the console any more when using FreeCAD as an external module. There was a request to stop it.

The only issue with loading the FreeCAD within a pure Python session is that by default it cannot find the FreeCAD module. On Linux systems it would be up to the packaging system to put the file to the site-packages directory.

[looo]

No I can't and it's another wrong assumption of you. Just look above:

Ah great . I tried this before but didn't see the output.

The only issue with loading the FreeCAD within a pure Python session is that by default it cannot find the FreeCAD module. On Linux systems it would be up to the packaging system to put the file to the site-packages directory.

There are several things to address here (maybe you can again destroy some wrong assumption):
- What options do we have? Maybe there is another solution than an abstraction layer like I tried here: https://github.com/looooo/freecad.pytho ... y.template
- can we agree on porting modules to the new-style? (if so can we do this in a way that doesn't break backward compatibility? https://github.com/looooo/freecad.pytho ... porting.md
- how to deal with multiple versions of FreeCAD in one system (and with the same python version) https://forum.freecadweb.org/viewtopic.php?f=8&t=40771

[wmayer]

- can we agree on porting modules to the new-style? (if so can we do this in a way that doesn't break backward compatibility?

In the long-term we should get rid of the habit to add all our module paths to sys.path. But I don't know how to keep backward compatibility as I don't know too much of the internals of Python how the module import mechanism works in detail.

- how to deal with multiple versions of FreeCAD in one system (and with the same python version)

With the Ubuntu packages there the version number is part of the directory name but doing this for Python would be a bit awkward.
But I think when a user has two versions installed one is used as production system and the other for testing/development. So, if there is a way that only the former can be directly imported in Python and for the latter the sys.path must be extended would be totally OK to me.

[looo]

But I think when a user has two versions installed one is used as production system and the other for testing/development. So, if there is a way that only the former can be directly imported in Python and for the latter the sys.path must be extended would be totally OK to me.

I guess using an environment variable to select the freecad version would be a nice option. But then the freecad-version must depend on the same python-version where the import was done... So we need to place a initialization to every python site-packages directory which has one or more freecad-versions.

Maybe there is a more elegant solution, but I feel like we are getting closer to a solution for this ubuntu-specific problem. What about other distros? Arch, redhat, opensuse? Are there parallel-versions suppprted?

[looo]

In the long-term we should get rid of the habit to add all our module paths to sys.path. But I don't know how to keep backward compatibility as I don't know too much of the internals of Python how the module import mechanism works in detail.

I tried this with the fem-module recently and this is more difficult than I thought. While I got all python modules ported from Mod/Fem to Ext/freecad/fem, I leave out the c++-files because I didn't know of the consequences of moving them into another directory.
In the end I was able to create a fem-analysis with pre and post processing. But opening one of the examples showed that the abstraction layer I tried to use (empty modules which imports everything from the transferred modules) wasn't really working. This is because anything with underscore prefix (private names like _variable) is not imported by using the "from <module> import *". Theoretically, it should be possible to fix this by importing all the things marked as private one by one.
But there is another problem: We really need to recreate the whole structure of the package to provide full backward-compatibility.

In the end I don't think it's not feasable to move a workbench like fem to the new-style packages with keeping full backward-compatibility. It might be possible, but the work needed to do so is enormous and without real benefit. So my conclusion of this is to do it the slow way by adding namespace-packages which are used as abstraction layers in the Ext-directory and this way define the future structure of the python interface.

Regarding the init.py:
If we initialize the new-style workbenches, we import the module at startup. This means the __init__.py is called at App-startup anyway. Further this means that the init.py is obsolete for the initialization and we can do everything in the __init__.py. So in the WorkbenchStarterkit we should not suggest to use init.py and instead use the __init__.py to initialize things like supported import types. In my eyes, this makes more sense for python-developers as it is the same for any other python package. But this does not mean that we should remove the possibility to have a init.py placed in a workbench (as some workbenches might use this in the future too). Maybe deprecate this option at one point, but it's only a question of consistency.

So if we can agree on the "init.py" not needed for namespace-packages I would suggest to move on with the topic which I didn't want to hijack.
No need to wait for FreeCAD to be fully moved towards namespace-packages. I guess it's best to postpone this after a major release (like 1.0) as it most likely introduces backward incompatibilities.
Also no need to fix the ubuntu multiple version not compatible with python's single version per package-and python-version-rule. I guess we can agree on an environment variable that gives us the opportunity to switch between different FreeCAD versions t select the one which is used for the python interface. I will tests this with the freecad.python package and if we find a solid solution, apply it to the master directly.

Anyway, I think it's very important to define some standards regarding the interface (FreeCAD or freecad, python imports, workbecnhes). Not doing so leads to inconsistency and frustration. That's why I think this work is very important. So thanks and please go ahead

[triplus]

Not doing so leads to inconsistency and frustration.

And doing so usually leads to inconsistency and frustration. Inconsistency being related to the fact, two solutions exists at the same time. Neither can be suggested as the default option, that usually leads to a lot of confusion. As for the frustration, such things, taking the scale of FreeCAD project into consideration, usually can take around a decade to happen. I guess similar to Py2/Py3 situation.

That's why I think this work is very important.

Sure.

P.S. IMHO i feel that we are still at the drafting stage of this proposal. But for sure we are much further, regarding understanding what is the problem we would like to resolve and on how to do it, compared to the first few debates. And as said in the past, maybe Qt 6 will introduce a more "Pythonic API", as hinted in the past. On when to start using it, that i guess is another question. As older Qt versions will likely stay supported for a while. And hence being more Pythonic on the long run, that will likely result in being less consistent and more frustrated in the mid term. In need to support both, old and new options.

[triplus]

As for i guess reducing the possibility of name clashes, using sys.path approach less, using things like __init__.py and in general to be more Pythonic. This likely is doable to great extent. I somehow doubt backward compatibility can be preserved in broader sense. As for starters the imports change, when migrating the module? As for multiple versions of FreeCAD and if i understand the problem correctly. I guess on Linux user would need to use a command like update-alternatives? Otherwise i don't know on how something like this could work:

Code: Select all

import freecad

For two versions of freecad installed.

I tried this with the fem-module recently and this is more difficult than I thought. While I got all python modules ported from Mod/Fem to Ext/freecad/fem, I leave out the c++-files because I didn't know of the consequences of moving them into another directory.

Indeed, what about such FreeCAD modules? Would they need a different (current) structure? What i am asking is would we end up with 2 or 3 types of FreeCAD modules? One being pure Python, pure C++ or being mixed? If yes, is this desirable, to have more "Workbench-Starterkit" options? And for i guess FreeCAD to initiate them differently? For example can Fem module be made to work in some straightforward fashion if we stop using the sys.path approach?

[looo]

As for starters the imports change, when migrating the module?

I guess this is the reason why we create the Startkit. I think people starting to code in python do not care much about the details how imports work as long as it works. So I guess the Starterkit should help them to get "started". I don't want anybody to dive into these details we are currently discussing.
As for other imports like "import FreeCAD" we need to define the future structure of the freecad python interface somwhow if we want to fully move to the new structure at one point. If we don't do this we accept the old structure and incompatibilities between importing from pure python and freecad will exist forever.

I guess on Linux user would need to use a command like update-alternatives? Otherwise i don't know on how something like this could work

Hmm not sure what you mean by command like update-alternitives? An environment variable for switching to select another freecad-version either temporary or permanently by adding it to the bashrc is in my eyes the easiest solution. Have a look at the freecad.python package [1].
This way we can add the directory containing the FreeCAD-library to sys.path and by importing FreeCAD we will initialize all the freecad (new-style&old-style) packages.
The difficult thing is to know where the FreeCAD-library is installed, but using the setup.py (and maybe later on the Cmake of freecad) we pass the responsibility of placing this "meta-package" in the right directory on to the maintainers. (Again by using environment-variable or a cmake-option).

In the end "from freecad import app" will have to do the same as "import sys; sys.path.append(<path_to_freecad_lib>); import FreeCAD as app". Without moving fully to new-style packages there will be still some inconvinience as you cannot "import Draft" without "import freecad" before. But I guess if we properly document this inconvenience people will understand why this is the case. Once fully moved to new-style packages this will be solved as Draft is imported via "from freecad import draft" and this will first run the one and only freecad/__init__.py which sets the sys.path and initialize all modules (non-gui).

In the end this really relies on defining some standarts across distros (which currently are not available). This is one of the main reasons for creating [2].

Indeed, what about such FreeCAD modules? Would they need a different (current) structure? What i am asking is would we end up with 2 or 3 types of FreeCAD modules? One being pure Python, pure C++ or being mixed? If yes, is this desirable, to have more "Workbench-Starterkit" options? And for i guess FreeCAD to initiate them differently?

Theoretical it shouldn't be a problem as the library is placed in PREFIX/lib anyway. So the source location should not matter very much, but you never know. I mainly feared porting the cmake-things.

[1] https://github.com/looooo/freecad.pytho ... y.template
[2] https://github.com/FreeCAD/FreeCAD-Enha ... -Proposals

[triplus]

I guess this is the reason why we create the Startkit. I think people starting to code in python do not care much about the details how imports work as long as it works. So I guess the Starterkit should help them to get "started". I don't want anybody to dive into these details we are currently discussing.

BUT, currently things are already rather simple. There is a Mod folder and a thing or two, that doesn't look Pythonic, but overall things work. Your proposals are much more involved. Future developers therefore will likely need to know much more about standard platform dependent Python module packaging, compared to Mod folder.

Hmm not sure what you mean by command like update-alternitives? An environment variable for switching to select another freecad-version either temporary or permanently by adding it to the bashrc is in my eyes the easiest solution.

As an end user yes, but as a packager packaging FreeCAD, like for example for the PPA purposes. I am guessing end users will need to use a command like update-alternitives.

P.S. Anyway, this effort for sure won't be finished anytime soon, lets see where does it take us.

[looo]

As an end user yes, but as a packager packaging FreeCAD, like for example for the PPA purposes. I am guessing end users will need to use a command like update-alternitives.

P.S. Anyway, this effort for sure won't be finished anytime soon, lets see where does it take us.

Yes, if there is a better way to handle this in ubuntu/debian then we should try it. I guess we simply have to define some standards for interfaces which can be fulfilled by any distro in one way.