Porting to python3

[triplus]

@triplus: please see my PM

As the question there was a general one i decided to answer it here. If you are interested in investing the time to make "reference builds" available sure go ahead.

As for setting up your development environment on Linux. For Qt 4 builds on Ubuntu only a few lines are needed to get all dependencies. Click on Expand button for Ubuntu:

https://www.freecadweb.org/wiki/CompileOnUnix#Ubuntu

After you need to git clone the FreeCAD source, create build directory and use cmake and make commands.

As for Qt 5 builds. In addition to what was said for Qt 4 builds the most straightforward way i know of is to use Ubuntu 16.04. As the Ubuntu 16.10 will reach EOL soon and higher versions of Ubuntu provide Qt versions above the version Qt 5.6. ATM PySide2 (needed for Qt 5 builds) won't compile against versions of Qt above 5.6:

https://bugreports.qt.io/browse/PYSIDE-323

There is a PPA available therefore you don't have to manually compile PySide2:

https://launchpad.net/~thopiekar/+archi ... pyside-git

From the PPA you would install:

Code: Select all

sudo apt-get install libpyside2-dev libshiboken2-dev pyside2-tools

ATM some workarounds are still needed:

https://forum.freecadweb.org/viewtopic. ... 20#p121669 # (libfreeimage issue was resolved in Ubuntu)
https://forum.freecadweb.org/viewtopic. ... 20#p121788

P.S. As for Python3 build of FreeCAD. I have not yet tried that. @looo likely knows best where we are (with the needed instructions) regarding this.

[looo]

P.S. As for Python3 build of FreeCAD. I have not yet tried that. @looo likely knows best where we are (with the needed instructions) regarding this.

this is how I build the py3-branch with docker and conda at the moment:

Code: Select all

> sudo docker run -i -t --name conda-forge condaforge/linux-anvil
## once the container runs interactively
> yum install -y libXt-devel libXmu-devel libXi-devel mesa-libGLU-devel rsync
> conda config --add channels freecad
> git clone https://github.com/looooo/FreeCAD_Conda
> cd FreeCAD_Conda/FreeCAD
> conda build . --python=3.5
## after the build is finished open another terminal and copy the build:
> sudo docker cp conda-forge:path_to_build place_to_store_the_build

[CADennis]

Thank your for your great input guys! I summarize, that I should get started with BuildBot+Jinja from the discussed choices {Jenkins,BuildBot} x {mako, django, jinja}. Please let me know ASAP if anyone prefers an alternative.

Let me design a prototype and then discuss its demo so it is less theoretical. I need to consider security and authentication aspects. Early feedback is always best, so here is a rough draft:

We will establish additional git repositories, "freecad-ci-master" and several "freecad-build-<targetX>.git".

Code: Select all

freecad-ci-master.git
 - master
 - bot_generic 
 - pipes

Code: Select all

freecad-build-<targetX>.git
 -* freecad-build-master.git  # submodule
 -* freecad.git               # submodule
 - bot
 - alien

Concrete targets are

Code: Select all

freecad-build-conda.git
freecad-build-windows.git
freecad-build-macos.git
freecad-build-rpm.git
freecad-build-deb-ppa.git

(have a look at http://git.blender.org/gitweb/ if you think it is overengineered for a Cpp-Python opensource project)

Each build target repository consists of jinja templates and distributor-specific artefacts, maintained in the subfolder "alien". And the bot definition itself, subclassed from bot_generic. The bot feeds provided dependency and configuration tensors ("build matrix") into its template engines. The content of "alien" might be as little as one template for the shell script you drafted above and one Dockerfile, or as much as the entire folder https://git.launchpad.net/freecad/plain/package/debian/ where only https://git.launchpad.net/freecad/plain ... an/control plus some more files are templates, the rest to be copied as is. The existing repository https://github.com/FreeCAD/homebrew-freecad could be migrated into one such "targetX" repository, e.g. "freecad-build-macos.git". Each buildbot job then checks its "freecad-build-<targetX>.git" out and pulls the submodules "freecad.git" and "freecad-ci-master.git".

I split the additions into separate repositories for authentication modalities. Wmayer might want to handle PRs from system builders as guaranteed to be separated from the master bot and pipe definitions. Doing so, bot-modifications can only cause harm inside restricted build containers. Wmayer can handle modifications to master bot and pipe definitions much more slow-paced and have them reviewed by several developers first. The repositories of type "freecad-build-<targetsX>.git" are _NEVER_ pulled as submodules into "freecad-ci-master.git", but vice versa. Thus, buildmasters never run any code from "freecad-build-<targetX>.git" inside their cloud containers. Furthermore, all PPA launchpad maintainers can get write access granted to the repository "freecad-build-deb-ppa.git" but no one else. So they can comfortably trust its contents when synchronizing it into the real Launchpad repository.

p.s. @looo:

[...]
this is how I build the py3-branch with docker and conda at the moment:

Code: Select all

> sudo docker run -i -t --name conda-forge condaforge/linux-anvil
## once the container runs interactively
> yum install -y libXt-devel libXmu-devel libXi-devel mesa-libGLU-devel rsync
> conda config --add channels freecad
> git clone https://github.com/looooo/FreeCAD_Conda
> cd FreeCAD_Conda/FreeCAD
> conda build . --python=3.5
## after the build is finished open another terminal and copy the build:
> sudo docker cp conda-forge:path_to_build place_to_store_the_build

Basing the build on "condaforge/linux-anvil" and then simply pulling a few remainders, is ultra-productive for us as a developers. And it provides me great insights into your current dependencies. In that it is very helpful, I asked you for such reference builds, thank you!

With all its opinionated preinstallments, CONDA will make any rule-based processing of a common package-dependency declaration across all OSes pretty rediculous, of course. Are you aware of their commercial freemium strategy?

[...] Conda is also included in the Continuum subscriptions of Anaconda, which provide on-site enterprise package and environment management for Python, R, Node.js, Java, and other application stacks. Conda is also available on pypi, although that approach may not be as up-to-date. [...]

You don't mind that? Do you merely use CONDA for personal, efficient development or do you wish to establish this as an official distribution channel for FreeCAD? I am a bit skeptical towards Continuoum Analytics and their freemium ecosystem. Found them too commercial and proprietary. Python end users and scholars strongly prefer CONDA over the more open, vendor-independent PIP. I noticed. But that is not very different from a "Microsoft Academic Alliance Program", put under the facade of "freemium open source". After their business model will have eventually been terminated, they might leave that landscape at ruins to truly free open-source developers. Such infrastructures come and go: codehaus-maven-service-unavailable. Or take .Net MONO. See Xamarin's license cost per seat in 2015. In the meantime aquired by Microsoft. Open source does not mean free. Continuum Analytics might just commercially exploit the temporary chaos and portability pressure introduced by Python2.7-to-Python3 incompatibilities, paired with academic involvement. That will eventually settle, unless they manage to hijack the next-gen Python userbase in the meantime. Still, I can understand any modern Python developer who wants to consume that productive ecosystem right now. So I will make sure, to integrate your CONDA layer into my prototypes.
From a heavy C++ system builder's OS dist point of view: if CONDA introduces a package distribution system redundantly stacked ontop of Win/Mac/Deb/Rpm, motivated from the very thin layer of a freemium installer wrapper around a scripting language around a C++ core, we should make sure we get FreeCAD deployed on an Ubuntu and Fedora via standard Linux package managers standalone. OPEN CASCADE SAS stood the test of time regarding FREE and fair open source involvement: 25 years. Continuum Analytics hasn't yet.
Let's find a strategy that runs another 20 years, honouring the great achievement of the people involved up to now, like you.

kind regards
Dennis

[looo]

Python end users and scholars strongly prefer CONDA over the more open, vendor-independent PIP.

pip is only for python (no other language) or at least it should not be used for any non standard python packages. I just wanted to uplaoad the pivy-package (swip-wrapper for coin) on pip, to have a version for python3 available. But this doesn't work. (missing headers)... So I would not compare pip with conda. With conda pivy can be build quite easily, although pivy uses distutils, which is the std. python distribution package.

Conda also has some problems, but for me it is simple the easiest solution currently available to distribute packages. The possibility to have virtual environments available is another thing why I like it. Running freecad in virtual environments give me the possibility to have python2 and python3 in parallel without any troubles. This could be also done with docker for example, but I think the virtual environment approach is a bit simpler.

But you are right. Once there is nobody working on the conda packages anymore we will have to build many packages ourself, and then using conda on linux wouldn't make much sense. (The list of dependencies is simple too big)

Found them too commercial and proprietary.

That is true. But it simple works. And in the end it is opensource. conda-forge is a community and already has better/more packages available then the default anaconda channel. Actually building freecad should be possible without anaconda-packages.

Do you merely use CONDA for personal, efficient development or do you wish to establish this as an official distribution channel for FreeCAD?

I think it is more a tool for developers. On Windows it is maybe a bit different. There it could be a replace of the libpacks we use. Most of the needed packages can be used from conda. Only the not so popular dependencies have to be build (occt, coin3d, pivy, simage, netgen). Conda (at the moment) is simple the best available tool to distribute packages which use c++ and python.

But packagemanger should not stop us from trying to follow our visions...

[triplus]

@looo

If i would try compiling the branch what should i expect to get as result? What should work and what should not? Is there some Python 3 related cmake "flag" to set?

P.S. And as for Qt 5 builds on Ubuntu 16.04. Don't try using cmake-gui for now as that results in Qt 4 libraries being used and compile procedure to not complete successfully. Instead "cmake -DBUILD_QT5=1" should be used without involving the cmake-gui.

[looo]

If i would try compiling the branch what should i expect to get as result? What should work and what should not? Is there some Python 3 related cmake "flag" to set?

You will see FreeCAD running with python3. Not all tests will pass. But this is more related to the modules. And as long as the base things are not in the master branch, I do not want to make changes to the modules. (more diff-> mode merge conflicts)
I am using the python3 branch for glider design (glider-workbench). So for me it's good enough. But this external module is not very dependent on FreeCAD.

cmake-flags used with conda can be seen here: https://github.com/looooo/FreeCAD_Conda ... ild.sh#L17
Most of them are conda-specific. But I think if you compile the py3-branch in an other environment this will look different.
the only thing not available directly from std. repositories is pivy. But this one is quite simple to compile.

I have also added some notes how to get a development environment with kdevelop and conda packages. https://github.com/looooo/FreeCAD_Conda ... elop-linux
But I had no success with setting up a develop environment on windows. Kdevelop on windows seems not that stable. Also running the ide inside an environment is not really supported. Maybe @peterl94 can give some hints, how to setup visual studio with the conda packages.

[triplus]

Based on the description therefore i don't really need to set any special flags when compiling your branch? And after FreeCAD compiles it should be Python 2/Python 3 compatible? If that is correct how do i make FreeCAD use Python 3 after instead of Python 2 when i run it? If both Python 2 and Python 3 are available on the system?

P.S. Note that i am talking about Ubuntu environment not conda.

[looo]

if your system is python2 by default, you have to set the python version. For ubuntu there are also some steps available here: https://www.freecadweb.org/wiki/Python_3 But it depends all on the system you are using.
A reproduce-able way (at the moment) to build is using docker and conda. But anything else will most likely need some work to get it running.

f that is correct how do i make FreeCAD use Python 3 after instead of Python 2 when i run it?

no, there is no option to switch at runtime. You cannot use python2-capi in python3 and the other way. on python level you can make everything work in python2 and python3 by using syntax supported by both versions. This is actually not very difficult, but some people are still in love with the python print statement.... But there will never be a FreeCAD-version which works with py2 and py3. Same is true for qt4 vs qt5.

[triplus]

if your system is python2 by default, you have to set the python version.

Set when compiling FreeCAD? I am guessing no? Or set in environment where running FreeCAD? I am guessing yes? Or simply Python 3 needs to be set as default on system before compiling or it won't work?

For ubuntu there are also some steps available here: https://www.freecadweb.org/wiki/Python_3 But it depends all on the system you are using.
A reproduce-able way (at the moment) to build is using docker and conda. But anything else will most likely need some work to get it running.

I'll try tomorrow to see how far i can get.

P.S. After you edited you answer this was added:

But there will never be a FreeCAD-version which works with py2 and py3. Same is true for qt4 vs qt5.

Therefore you branch shouldn't run/compile in Python 2 environment or should it?

[triplus]

First time i overlooked related section on the provided link:

https://www.freecadweb.org/wiki/Python_3

Therefore things happen at compile time and the build after isn't compatible with Python 2. Will try to do some test tomorrow to see if i can produce Python 3 FreeCAD build.