Porting to python3

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
User avatar
kkremitzki
Posts: 360
Joined: Thu Mar 03, 2016 9:52 pm
Location: Texas

Re: Porting to python3

Postby kkremitzki » Tue Feb 21, 2017 5:11 pm

flutefreak7 wrote:I just wanted to say I'm really excited to see the Python 3 work happening! I've been randomly checking the internet for some of the CAD related tools to be ported to Python 3 ever since I started with Python around 7 years ago and I'm really excited to see all the effort you guys are putting in!

I'm unfortunately not very skilled with C++, but I might could help with translating Python 2 idioms to Python 3 or end user testing or something once I'm more up to speed with FreeCAD. I feel bad just being a needy user and not helping out more, but I do want to say that your work will be appreciated!

Good to see you're willing to help! One of the first things, I think, is making sure you can built and run FreeCAD with Python 3 enabled on your machine. If you have questions or need help in doing so feel free to ask.
looo
Posts: 775
Joined: Mon Nov 11, 2013 5:29 pm

Re: Porting to python3

Postby looo » Sun Mar 12, 2017 1:35 pm

After all the diff for src/Mod of the python3 branch is in master, the rebasing isn't that time intensive anymore. Building Path with python3 is again broken, because of some new Int->Long conflicts. Also fem has some issues, but only at runtime... but that is not a big issue right now.

To make the python3 things more stable we have to get travis to work with python3. But therefor the full branch has to move to master. I don't have a plan right now how to do this. So I am open for suggestions:
how can we test python3 things automatically without moving all into master. Is there a tool to do an automatic rebase? And can we setup a branch which is tested to build with python3 for every PR?
Or should we push towards a python3 compatible master. This means that there will most likely break some things. Not that a big problem,... but it's always annoying for others...

Btw.: The up to date python3 branch has moved to https://github.com/looooo/FreeCAD/tree/py3-23
User avatar
Kunda1
Posts: 674
Joined: Thu Jan 05, 2017 9:03 pm

Re: Porting to python3

Postby Kunda1 » Mon Mar 27, 2017 6:06 pm

BTW, any commits that are related to porting to python3 please link them to issue #995
In the commit message you can write

Code: Select all

issue #995
in order to auto-link it in MantisBT. See https://freecadweb.org/wiki/tracker#Att ... o_a_ticket
simonvanderveldt
Posts: 61
Joined: Tue Mar 14, 2017 2:11 pm

Re: Porting to python3

Postby simonvanderveldt » Mon Mar 27, 2017 6:24 pm

looo wrote:After all the diff for src/Mod of the python3 branch is in master, the rebasing isn't that time intensive anymore. Building Path with python3 is again broken, because of some new Int->Long conflicts. Also fem has some issues, but only at runtime... but that is not a big issue right now.

To make the python3 things more stable we have to get travis to work with python3. But therefor the full branch has to move to master. I don't have a plan right now how to do this. So I am open for suggestions:
how can we test python3 things automatically without moving all into master. Is there a tool to do an automatic rebase? And can we setup a branch which is tested to build with python3 for every PR?
Or should we push towards a python3 compatible master. This means that there will most likely break some things. Not that a big problem,... but it's always annoying for others...

Btw.: The up to date python3 branch has moved to https://github.com/looooo/FreeCAD/tree/py3-23

I'm not 100% sure I understand the issue, but you can simply add python3 to Travis's matrix and then instruct Travis to not fail the build on Python 3 using allow_failures. See https://docs.travis-ci.com/user/customi ... ed-to-Fail for details
CADennis
Posts: 10
Joined: Tue Apr 18, 2017 10:12 am

Re: Porting to python3

Postby CADennis » Tue Apr 25, 2017 4:51 pm

looo wrote: [...] To make the python3 things more stable we have to get travis to work with python3. But therefor the full branch has to move to master. I don't have a plan right now how to do this. So I am open for suggestions:
how can we test python3 things automatically without moving all into master. Is there a tool to do an automatic rebase? And can we setup a branch which is tested to build with python3 for every PR?
Or should we push towards a python3 compatible master. This means that there will most likely break some things. Not that a big problem,... but it's always annoying for others... [...]

Do I understand you right, the main focus of stabelizations would be {QT4, QT5} x {Python2, Python3.5, Python3.6} x {Windows 7/10, macOS10.12._, Ubuntu14.04, Ubuntu14.10, Ubuntu16.04, Ubuntu16.10, Gentoo32bit, Mint***, Cent*** } ?

How about setting up an automatic branch/rebase/merge system? It shall react upon pull requests via GitHub webhooks and create for each new pull request topic a branch, e.g. "PR1._". It then runs automatically regression tests that we collect, not only functional tests but also OS variant and package dependency integration tests. For such a system we can also automate partial incremental build definitions that get executed through Travis to incrementally build up the cache per pull request branch: 1st a build bot commits the build instructions for dependencies. The build output is cached on a cleaned cache. Then a bot commits build instructions to build the FreeCAD core, then to build optional FreeCAD modules, all incrementally cached, but initially from scratch per PR branch.

After all regression tests passed - for ALL OS and Python2 AND Python3 - a bot merges the PR commit into the branch "master_auto_proposed". Wmayer will only have to look into PRs that made it up into that branch, manually review and then merge them into his master.

The CI bot's automatic rebase/merge will not always succeed. That is the whole point of having it. It must reject complications introduced by unsynced developers. The bot supports the master branch owner (Wmayer) by filtering those PRs out that would cause conflicts. The bot returns them back to sender. Each developer must get the rebase/merge working, not the master owner. In the example illustrated, you will see that PR4.0 and PR5.0 will likely fail because of rebase conflicts, even though that is not the fault of the developers not having synced. It is just a result of distributed team work. Wmayer can do nothing against it. The PR submitter can do nothing against it. But if all such conflicts were left to Wmayer to manually resolve them, he would run out of time and the next master commit would get delayed. That would prevent developers from rebasing, and we'd all run into the vicious circle of merge / rebase conflicts. bam!
Py2-Py3-branchAndMergeStrategy1c.png
Py2-Py3-branchAndMergeStrategy1c.png (354.74 KiB) Viewed 138 times

The advantage of the automated preprocessing into "master_auto_proposed" is its large-team scalability. Developers can rebase from "master_auto_proposed" to avoid rebase-conflicts. And this is the moment, when integration becomes CONTINUOUS, because all developers can pull updates from "master_auto_proposed" for quick-sync into their features branches on their satellite repositories and prepare new pull requests for the shared repository - long before the master owner is back from holidays for manual review. Your team is quite large. It needs that automated scalability.

:?: What is the infrastructure behind freecadweb.org? Does it run on Amazon AWS, Azure, Oracle? Could we then reuse unmetered but limited capacity from it? I would help you in setting up Jenkins/BuildBot + GitHub-Plugin + docker to run on EC2 or similar. , independent of their proprietary pipe services, which FreeCAD might opt for, later anytime.

What would you prefer, Jenkins or BuildBot? Both have the big advantage that they offer common programming languages or at least JSON and XML to setup build definitions. With all that trendy YAML stuff, vendors just try to lock you into their system by proprietary YAML grammar without grammar/schema editor support for Intellisense/Codecompletion. If you have never experienced Intellisense for XML or JASON schemata, try it! Don't get hyped into crap. They hope, once you learned their proprietary YAML-scheme you never want to change to another YAML scheme of the 1000000 competitors with their proprietary dialects. Just a contemporary "cloud"-implementation of vendor lock-in. And please place the file at the root of the repository so everyone can see ("Travis/CircleCI/CodeSh** was here!"). Clever marketing. But after reading the 35 pages in this thread here, I think Travis does not suite your needs:
BuildBot allows build definitions in Python. We could maintain dependency lists as Python lists and write them out to the different YAMLs and whatever using template engines such as Django. I could then share the dependency list for docker definitions with the Launchpad PPA maintenance team. They need commata for DPKG, I need spaces and line breaks, but the content is the same, the rest is just vendor locks-ins.

The two most important things we must get rid of in my opinion are
  • the single-50-minute build with Travis-caching which is not integrated with precise incremental logic of build tools and not integrated with docker caching
  • the synchronization effort manually bottle-necked at the master owner
Last edited by CADennis on Tue Apr 25, 2017 6:10 pm, edited 11 times in total.
triplus
Posts: 4993
Joined: Mon Dec 12, 2011 4:45 pm

Re: Porting to python3

Postby triplus » Tue Apr 25, 2017 5:05 pm

Hi @CADennis.

Web hosting we use doesn't offer us much in this regard. And there is the problem of capable system administrators being involved that would take care of building and maintaining the infrastructure. And if all that would get done the way i understand it what is actually needed ATM is more developers working on the porting effort itself.

P.S. And yes in the past discussions it was decided Python2/Python3/Qt4/Qt5 compatibility should be preserved for now.
User avatar
kkremitzki
Posts: 360
Joined: Thu Mar 03, 2016 9:52 pm
Location: Texas

Re: Porting to python3

Postby kkremitzki » Tue Apr 25, 2017 5:06 pm

CADennis wrote:good stuff

I can give a more fleshed out answer later, but for now:
- freecadweb is on shared hosting, i.e. not a fully-controlled server
- we recently got some servers to expand infrastructure and part of this includes a plan to use buildbot_travis (a Travis compatibility shim for our current CI tooling). Long term I think this will be great, Python-based self-hosted CI.
looo
Posts: 775
Joined: Mon Nov 11, 2013 5:29 pm

Re: Porting to python3

Postby looo » Wed Apr 26, 2017 1:15 pm

thanks for the input. I think this are quite good ideas.

CADennis wrote:Do I understand you right, the main focus of stabelizations would be {QT4, QT5} x {Python2, Python3.5, Python3.6} x {Windows 7/10, macOS10.12._, Ubuntu14.04, Ubuntu14.10, Ubuntu16.04, Ubuntu16.10, Gentoo32bit, Mint***, Cent*** } ?


hmm, I don't think we have to test every linux-distro. Conda for example uses a "any-linux" docker file based on an old centos6. This way it's theoretically possible to support all linux-distros. (Similar approach to appimage)
(linux, windows) x (python2 + python3) + mac x python2 x qt5 should be already difficult enough ;)

How about setting up an automatic branch/rebase/merge system?

this is for sure a good idea.

What would you prefer, Jenkins or BuildBot? Both have the big advantage that they offer common programming languages or at least JSON and XML to setup build definitions.


I just tried the first examples with buildbot... installed with pip and ran in conda environment.
CADennis
Posts: 10
Joined: Tue Apr 18, 2017 10:12 am

Re: Porting to python3

Postby CADennis » Wed Apr 26, 2017 6:06 pm

@looo why is Python3 x QT5 x macOS not included in your matrix? has there been trouble and common agreement to skip those newer dependencies on Apple? Looking forward to your BuildBot feedback. Please be aware of BuildBot's security concept. It has no security and must be stacked ontop of a secure IRC infrastructure. No critique, just keep that in mind while plowing through the features: (1) get dependencies, (2) unplug network, (3) switch to non-priviledged user (4) start the bots ... I don't want to lose you as the main driver behind Python3 :-)
buildbot_teaser.png
buildbot_teaser.png (26.44 KiB) Viewed 52 times

here is a security setup tutorial for Jenkins, by Microsoft: https://jenkins.io/blog/2017/04/20/secu ... -on-azure/
I must admit, I have not used Jenkins outside a corporate DMZ before, so that tutorial from Microsoft came just in time.

Sure I had to go grab a glass of fresh water and open the window when I saw that the security documentation for open source JAVA software is now sponsored by Microsoft.

Be careful and disciplined not to learn and use all the features of BuildBot just because they are there: don't!!! Fans have accumulated many non-functional wrappers. Wrappers that don't add functionality to the tool underneath but are only meant as convenience bridges to Python coders. From my industrial experience, such wrappers just come into your way during future maintenance, if there is no one to blame or financially involve. Here is an example: the cmake convenience wrapper

Code: Select all

from buildbot.plugins import steps
factory.addStep(

    steps.CMake(
        generator='Ninja',
        definitions={
            'CMAKE_BUILD_TYPE': Property('BUILD_TYPE')
        },
        options=[
            '-Wno-dev'
        ]
    )
)

Wrong design. --> Want to use it for MS Visual Studio 2017? Forget that! Wait until the open source owner adds it. Until then, patch and sync and integrate ... I think you have had enough of that recently. No fun. Avoid nonfunctional handwritten convenience wrappers by thirdparties around essential tools! Now to something completely different. Here is a Python REST convenience wrapper around Jenkins ;-) https://python-jenkins.readthedocs.io/e ... of-jenkins

Fair enough, don't miss their cool new Blue Ocean Jenkins pipe framework. Downside: Jenkinsfile is a DSL. Widespread Intellisense support in the future? - nope! Yet the pipes can be edited via GUI, maybe nice for "serious C++ code crunchers" who are not so involved in daily Python tasks. When after 5 years someone has to dig back into a fellow programmer's build automation, it is nice _not_ to find Turing-complete dynamic scripting languages but a visual editor. Yet the safest practice is betting on the availability of a Python editor in 5 years when a CI plugin is no longer so hyped that its visual editors work.
jenkins_teaser.png
jenkins_teaser.png (59.18 KiB) Viewed 52 times

For both, BuildBot and Jenkins it is commonly recommended to restrict tool integration to
(1) setting up environment variables
(2) generating or manually maintaining tool input parameter files (e.g. makefiles, shell scripts, ini files, ...
(3) integrating a single line for the generic tool invocation, for instance

Code: Select all

from buildbot.plugins import util, steps
f = util.BuildFactory()
freecadOpts = ourSuperDupaPipeLogic() //...
f.addSteps([
    steps.SVN(repourl="http://svn.example.org/Trunk/"),
    steps.ShellCommand(command=["cmake", freecadOpts])
])

That motivates a template engine for us at FreeCAD. So my question to Python power users here: what would be your favourite template engine?
  • mako,
  • django,
  • jinja
  • what else?

@triplus: please see my PM

@kkremitzki: I just had a quick look at the travis_shim that you mentioned ... . I don't like its concept. It "consumes Travis YAML" and produces BuildBot configurations. My recommendation is the other way around: consume Python-script and generate template-engine-based instructions for build tools and deployment systems. Conda packages, PIP packages, MSI packages, homebrew, APT, RPM, etc. They all need similar dependency definitions. I don't see how we could specify the following workflow in Travis Yaml and feed that into BuildBot, I think it will only work in the opposite direction:
(1) Branches are created automatically per pull request - maybe even for each build configuration a separate sub-branch.
(2) They get removed after PR completion.
Git allows to delete branches without leftovers after usage. That is Git's advantage over Subversion. BuildBot dynamically generates into them the specific build job definition, as temporarily required.

And we can exploit that flexibility of dynamically committing build instructions into branches to even "hack" around the Travis time limit and continue to use its OSX service: Each PR-job macOS subbranch builds up the cache by repeatedly committing increasingly complete build definitions into such branches with activated caching. CI bots define explicitly, when and how this cache is cleaned. In the running example that would be
branch "PR1._":

Code: Select all

{"clean cache", "PR1.0.macOS.build dependencies", "PR1.0.macOS.build core", "PR1.0.macOS.build addon1", ... ,"PR1.0.macOS.build addonN"}
{"clean cache", "PR1.1.macOS.build dependencies", "PR1.1.macOS.build core", "PR1.1.macOS.build addon1", ... ,"PR1.1.macOS.build addonN"}
branch "PR2._":
{"clean cache", "PR2.0.macOS.build dependencies", "PR2.0.macOS.build core", "PR2.0.macOS.build addon1", ... ,"PR2.0.macOS.build addonN"}
branch "PR3._":
{"clean cache", "PR3.0.macOS.build dependencies", "PR3.0.macOS.build core", "PR3.0.macOS.build addon1", ... ,"PR3.0.macOS.build addonN"}
...


Though I was initially preferring Jenkins, I'll help in what the majority here prefers. Not only Chrome, Firefox and Wireshark use BuildBot, even Blender. And we might ally with them for change requests. I have seen, they already hacked workarounds into stock BuildBot.