Good to see you're willing to help! One of the first things, I think, is making sure you can built and run FreeCAD with Python 3 enabled on your machine. If you have questions or need help in doing so feel free to ask.flutefreak7 wrote:I just wanted to say I'm really excited to see the Python 3 work happening! I've been randomly checking the internet for some of the CAD related tools to be ported to Python 3 ever since I started with Python around 7 years ago and I'm really excited to see all the effort you guys are putting in!
I'm unfortunately not very skilled with C++, but I might could help with translating Python 2 idioms to Python 3 or end user testing or something once I'm more up to speed with FreeCAD. I feel bad just being a needy user and not helping out more, but I do want to say that your work will be appreciated!
Porting to python3
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Be nice to others! Respect the FreeCAD code of conduct!
- kkremitzki
- Veteran
- Posts: 2518
- Joined: Thu Mar 03, 2016 9:52 pm
- Location: Illinois
Re: Porting to python3
Re: Porting to python3
After all the diff for src/Mod of the python3 branch is in master, the rebasing isn't that time intensive anymore. Building Path with python3 is again broken, because of some new Int->Long conflicts. Also fem has some issues, but only at runtime... but that is not a big issue right now.
To make the python3 things more stable we have to get travis to work with python3. But therefor the full branch has to move to master. I don't have a plan right now how to do this. So I am open for suggestions:
how can we test python3 things automatically without moving all into master. Is there a tool to do an automatic rebase? And can we setup a branch which is tested to build with python3 for every PR?
Or should we push towards a python3 compatible master. This means that there will most likely break some things. Not that a big problem,... but it's always annoying for others...
Btw.: The up to date python3 branch has moved to https://github.com/looooo/FreeCAD/tree/py3-23
To make the python3 things more stable we have to get travis to work with python3. But therefor the full branch has to move to master. I don't have a plan right now how to do this. So I am open for suggestions:
how can we test python3 things automatically without moving all into master. Is there a tool to do an automatic rebase? And can we setup a branch which is tested to build with python3 for every PR?
Or should we push towards a python3 compatible master. This means that there will most likely break some things. Not that a big problem,... but it's always annoying for others...
Btw.: The up to date python3 branch has moved to https://github.com/looooo/FreeCAD/tree/py3-23
Re: Porting to python3
BTW, any commits that are related to porting to python3 please link them to issue #995
In the commit message you can write in order to auto-link it in MantisBT. See https://freecadweb.org/wiki/tracker#Att ... o_a_ticket
In the commit message you can write
Code: Select all
issue #995
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
-
- Posts: 62
- Joined: Tue Mar 14, 2017 2:11 pm
Re: Porting to python3
I'm not 100% sure I understand the issue, but you can simply add python3 to Travis's matrix and then instruct Travis to not fail the build on Python 3 using allow_failures. See https://docs.travis-ci.com/user/customi ... ed-to-Fail for detailslooo wrote:After all the diff for src/Mod of the python3 branch is in master, the rebasing isn't that time intensive anymore. Building Path with python3 is again broken, because of some new Int->Long conflicts. Also fem has some issues, but only at runtime... but that is not a big issue right now.
To make the python3 things more stable we have to get travis to work with python3. But therefor the full branch has to move to master. I don't have a plan right now how to do this. So I am open for suggestions:
how can we test python3 things automatically without moving all into master. Is there a tool to do an automatic rebase? And can we setup a branch which is tested to build with python3 for every PR?
Or should we push towards a python3 compatible master. This means that there will most likely break some things. Not that a big problem,... but it's always annoying for others...
Btw.: The up to date python3 branch has moved to https://github.com/looooo/FreeCAD/tree/py3-23
Re: Porting to python3
Do I understand you right, the main focus of stabelizations would be {QT4, QT5} x {Python2, Python3.5, Python3.6} x {Windows 7/10, macOS10.12._, Ubuntu14.04, Ubuntu14.10, Ubuntu16.04, Ubuntu16.10, Gentoo32bit, Mint***, Cent*** } ?looo wrote: [...] To make the python3 things more stable we have to get travis to work with python3. But therefor the full branch has to move to master. I don't have a plan right now how to do this. So I am open for suggestions:
how can we test python3 things automatically without moving all into master. Is there a tool to do an automatic rebase? And can we setup a branch which is tested to build with python3 for every PR?
Or should we push towards a python3 compatible master. This means that there will most likely break some things. Not that a big problem,... but it's always annoying for others... [...]
How about setting up an automatic branch/rebase/merge system? It shall react upon pull requests via GitHub webhooks and create for each new pull request topic a branch, e.g. "PR1._". It then runs automatically regression tests that we collect, not only functional tests but also OS variant and package dependency integration tests. For such a system we can also automate partial incremental build definitions that get executed through Travis to incrementally build up the cache per pull request branch: 1st a build bot commits the build instructions for dependencies. The build output is cached on a cleaned cache. Then a bot commits build instructions to build the FreeCAD core, then to build optional FreeCAD modules, all incrementally cached, but initially from scratch per PR branch.
After all regression tests passed - for ALL OS and Python2 AND Python3 - a bot merges the PR commit into the branch "master_auto_proposed". Wmayer will only have to look into PRs that made it up into that branch, manually review and then merge them into his master.
The CI bot's automatic rebase/merge will not always succeed. That is the whole point of having it. It must reject complications introduced by unsynced developers. The bot supports the master branch owner (Wmayer) by filtering those PRs out that would cause conflicts. The bot returns them back to sender. Each developer must get the rebase/merge working, not the master owner. In the example illustrated, you will see that PR4.0 and PR5.0 will likely fail because of rebase conflicts, even though that is not the fault of the developers not having synced. It is just a result of distributed team work. Wmayer can do nothing against it. The PR submitter can do nothing against it. But if all such conflicts were left to Wmayer to manually resolve them, he would run out of time and the next master commit would get delayed. That would prevent developers from rebasing, and we'd all run into the vicious circle of merge / rebase conflicts. bam! The advantage of the automated preprocessing into "master_auto_proposed" is its large-team scalability. Developers can rebase from "master_auto_proposed" to avoid rebase-conflicts. And this is the moment, when integration becomes CONTINUOUS, because all developers can pull updates from "master_auto_proposed" for quick-sync into their features branches on their satellite repositories and prepare new pull requests for the shared repository - long before the master owner is back from holidays for manual review. Your team is quite large. It needs that automated scalability.
What is the infrastructure behind freecadweb.org? Does it run on Amazon AWS, Azure, Oracle? Could we then reuse unmetered but limited capacity from it? I would help you in setting up Jenkins/BuildBot + GitHub-Plugin + docker to run on EC2 or similar. , independent of their proprietary pipe services, which FreeCAD might opt for, later anytime.
What would you prefer, Jenkins or BuildBot? Both have the big advantage that they offer common programming languages or at least JSON and XML to setup build definitions. With all that trendy YAML stuff, vendors just try to lock you into their system by proprietary YAML grammar without grammar/schema editor support for Intellisense/Codecompletion. If you have never experienced Intellisense for XML or JASON schemata, try it! Don't get hyped into crap. They hope, once you learned their proprietary YAML-scheme you never want to change to another YAML scheme of the 1000000 competitors with their proprietary dialects. Just a contemporary "cloud"-implementation of vendor lock-in. And please place the file at the root of the repository so everyone can see ("Travis/CircleCI/CodeSh** was here!"). Clever marketing. But after reading the 35 pages in this thread here, I think Travis does not suite your needs:
- build time limit failures (yorik) - https://travis-ci.org/yorikvanhavre/Fre ... /222997954
- build time limit cache hacks (bernd) - https://forum.freecadweb.org/viewtopic. ... 77#p132477
- implicit python dependencies (saso) - https://forum.freecadweb.org/viewtopic. ... 28#p150928
The two most important things we must get rid of in my opinion are
- the single-50-minute build with Travis-caching which is not integrated with precise incremental logic of build tools and not integrated with docker caching
- the synchronization effort manually bottle-necked at the master owner
Last edited by CADennis on Tue Apr 25, 2017 6:10 pm, edited 11 times in total.
Re: Porting to python3
Hi @CADennis.
Web hosting we use doesn't offer us much in this regard. And there is the problem of capable system administrators being involved that would take care of building and maintaining the infrastructure. And if all that would get done the way i understand it what is actually needed ATM is more developers working on the porting effort itself.
P.S. And yes in the past discussions it was decided Python2/Python3/Qt4/Qt5 compatibility should be preserved for now.
Web hosting we use doesn't offer us much in this regard. And there is the problem of capable system administrators being involved that would take care of building and maintaining the infrastructure. And if all that would get done the way i understand it what is actually needed ATM is more developers working on the porting effort itself.
P.S. And yes in the past discussions it was decided Python2/Python3/Qt4/Qt5 compatibility should be preserved for now.
- kkremitzki
- Veteran
- Posts: 2518
- Joined: Thu Mar 03, 2016 9:52 pm
- Location: Illinois
Re: Porting to python3
I can give a more fleshed out answer later, but for now:CADennis wrote:good stuff
- freecadweb is on shared hosting, i.e. not a fully-controlled server
- we recently got some servers to expand infrastructure and part of this includes a plan to use buildbot_travis (a Travis compatibility shim for our current CI tooling). Long term I think this will be great, Python-based self-hosted CI.
Re: Porting to python3
thanks for the input. I think this are quite good ideas.
(linux, windows) x (python2 + python3) + mac x python2 x qt5 should be already difficult enough
hmm, I don't think we have to test every linux-distro. Conda for example uses a "any-linux" docker file based on an old centos6. This way it's theoretically possible to support all linux-distros. (Similar approach to appimage)CADennis wrote:Do I understand you right, the main focus of stabelizations would be {QT4, QT5} x {Python2, Python3.5, Python3.6} x {Windows 7/10, macOS10.12._, Ubuntu14.04, Ubuntu14.10, Ubuntu16.04, Ubuntu16.10, Gentoo32bit, Mint***, Cent*** } ?
(linux, windows) x (python2 + python3) + mac x python2 x qt5 should be already difficult enough
this is for sure a good idea.How about setting up an automatic branch/rebase/merge system?
I just tried the first examples with buildbot... installed with pip and ran in conda environment.What would you prefer, Jenkins or BuildBot? Both have the big advantage that they offer common programming languages or at least JSON and XML to setup build definitions.
Re: Porting to python3
@looo why is Python3 x QT5 x macOS not included in your matrix? has there been trouble and common agreement to skip those newer dependencies on Apple? Looking forward to your BuildBot feedback. Please be aware of BuildBot's security concept. It has no security and must be stacked ontop of a secure IRC infrastructure. No critique, just keep that in mind while plowing through the features: (1) get dependencies, (2) unplug network, (3) switch to non-priviledged user (4) start the bots ... I don't want to lose you as the main driver behind Python3
https://jenkins.io/blog/2017/04/20/secu ... -on-azure/
I must admit, I have not used Jenkins outside a corporate DMZ before, so that tutorial from Microsoft came just in time.
Sure I had to go grab a glass of fresh water and open the window when I saw that the security documentation for open source JAVA software is now sponsored by Microsoft.
Be careful and disciplined not to learn and use all the features of BuildBot just because they are there: don't!!! Fans have accumulated many non-functional wrappers. Wrappers that don't add functionality to the tool underneath but are only meant as convenience bridges to Python coders. From my industrial experience, such wrappers just come into your way during future maintenance, if there is no one to blame or financially involve. Here is an example: the cmake convenience wrapper
Wrong design. --> Want to use it for MS Visual Studio 2017? Forget that! Wait until the open source owner adds it. Until then, patch and sync and integrate ... I think you have had enough of that recently. No fun. Avoid nonfunctional handwritten convenience wrappers by thirdparties around essential tools! Now to something completely different. Here is a Python REST convenience wrapper around Jenkins https://python-jenkins.readthedocs.io/e ... of-jenkins
Fair enough, don't miss their cool new Blue Ocean Jenkins pipe framework. Downside: Jenkinsfile is a DSL. Widespread Intellisense support in the future? - nope! Yet the pipes can be edited via GUI, maybe nice for "serious C++ code crunchers" who are not so involved in daily Python tasks. When after 5 years someone has to dig back into a fellow programmer's build automation, it is nice _not_ to find Turing-complete dynamic scripting languages but a visual editor. Yet the safest practice is betting on the availability of a Python editor in 5 years when a CI plugin is no longer so hyped that its visual editors work. For both, BuildBot and Jenkins it is commonly recommended to restrict tool integration to
(1) setting up environment variables
(2) generating or manually maintaining tool input parameter files (e.g. makefiles, shell scripts, ini files, ...
(3) integrating a single line for the generic tool invocation, for instance
That motivates a template engine for us at FreeCAD. So my question to Python power users here: what would be your favourite template engine?
@kkremitzki: I just had a quick look at the travis_shim that you mentioned ... . I don't like its concept. It "consumes Travis YAML" and produces BuildBot configurations. My recommendation is the other way around: consume Python-script and generate template-engine-based instructions for build tools and deployment systems. Conda packages, PIP packages, MSI packages, homebrew, APT, RPM, etc. They all need similar dependency definitions. I don't see how we could specify the following workflow in Travis Yaml and feed that into BuildBot, I think it will only work in the opposite direction:
(1) Branches are created automatically per pull request - maybe even for each build configuration a separate sub-branch.
(2) They get removed after PR completion.
Git allows to delete branches without leftovers after usage. That is Git's advantage over Subversion. BuildBot dynamically generates into them the specific build job definition, as temporarily required.
And we can exploit that flexibility of dynamically committing build instructions into branches to even "hack" around the Travis time limit and continue to use its OSX service: Each PR-job macOS subbranch builds up the cache by repeatedly committing increasingly complete build definitions into such branches with activated caching. CI bots define explicitly, when and how this cache is cleaned. In the running example that would be
branch "PR1._":
Though I was initially preferring Jenkins, I'll help in what the majority here prefers. Not only Chrome, Firefox and Wireshark use BuildBot, even Blender. And we might ally with them for change requests. I have seen, they already hacked workarounds into stock BuildBot.
here is a security setup tutorial for Jenkins, by Microsoft: I must admit, I have not used Jenkins outside a corporate DMZ before, so that tutorial from Microsoft came just in time.
Sure I had to go grab a glass of fresh water and open the window when I saw that the security documentation for open source JAVA software is now sponsored by Microsoft.
Be careful and disciplined not to learn and use all the features of BuildBot just because they are there: don't!!! Fans have accumulated many non-functional wrappers. Wrappers that don't add functionality to the tool underneath but are only meant as convenience bridges to Python coders. From my industrial experience, such wrappers just come into your way during future maintenance, if there is no one to blame or financially involve. Here is an example: the cmake convenience wrapper
Code: Select all
from buildbot.plugins import steps
factory.addStep(
steps.CMake(
generator='Ninja',
definitions={
'CMAKE_BUILD_TYPE': Property('BUILD_TYPE')
},
options=[
'-Wno-dev'
]
)
)
Fair enough, don't miss their cool new Blue Ocean Jenkins pipe framework. Downside: Jenkinsfile is a DSL. Widespread Intellisense support in the future? - nope! Yet the pipes can be edited via GUI, maybe nice for "serious C++ code crunchers" who are not so involved in daily Python tasks. When after 5 years someone has to dig back into a fellow programmer's build automation, it is nice _not_ to find Turing-complete dynamic scripting languages but a visual editor. Yet the safest practice is betting on the availability of a Python editor in 5 years when a CI plugin is no longer so hyped that its visual editors work. For both, BuildBot and Jenkins it is commonly recommended to restrict tool integration to
(1) setting up environment variables
(2) generating or manually maintaining tool input parameter files (e.g. makefiles, shell scripts, ini files, ...
(3) integrating a single line for the generic tool invocation, for instance
Code: Select all
from buildbot.plugins import util, steps
f = util.BuildFactory()
freecadOpts = ourSuperDupaPipeLogic() //...
f.addSteps([
steps.SVN(repourl="http://svn.example.org/Trunk/"),
steps.ShellCommand(command=["cmake", freecadOpts])
])
- mako,
- django,
- jinja
- what else?
@kkremitzki: I just had a quick look at the travis_shim that you mentioned ... . I don't like its concept. It "consumes Travis YAML" and produces BuildBot configurations. My recommendation is the other way around: consume Python-script and generate template-engine-based instructions for build tools and deployment systems. Conda packages, PIP packages, MSI packages, homebrew, APT, RPM, etc. They all need similar dependency definitions. I don't see how we could specify the following workflow in Travis Yaml and feed that into BuildBot, I think it will only work in the opposite direction:
(1) Branches are created automatically per pull request - maybe even for each build configuration a separate sub-branch.
(2) They get removed after PR completion.
Git allows to delete branches without leftovers after usage. That is Git's advantage over Subversion. BuildBot dynamically generates into them the specific build job definition, as temporarily required.
And we can exploit that flexibility of dynamically committing build instructions into branches to even "hack" around the Travis time limit and continue to use its OSX service: Each PR-job macOS subbranch builds up the cache by repeatedly committing increasingly complete build definitions into such branches with activated caching. CI bots define explicitly, when and how this cache is cleaned. In the running example that would be
branch "PR1._":
Code: Select all
{"clean cache", "PR1.0.macOS.build dependencies", "PR1.0.macOS.build core", "PR1.0.macOS.build addon1", ... ,"PR1.0.macOS.build addonN"}
{"clean cache", "PR1.1.macOS.build dependencies", "PR1.1.macOS.build core", "PR1.1.macOS.build addon1", ... ,"PR1.1.macOS.build addonN"}
branch "PR2._":
{"clean cache", "PR2.0.macOS.build dependencies", "PR2.0.macOS.build core", "PR2.0.macOS.build addon1", ... ,"PR2.0.macOS.build addonN"}
branch "PR3._":
{"clean cache", "PR3.0.macOS.build dependencies", "PR3.0.macOS.build core", "PR3.0.macOS.build addon1", ... ,"PR3.0.macOS.build addonN"}
...
Re: Porting to python3
I didn't get deep into build-bot. This is all new stuff to me.CADennis wrote: Looking forward to your BuildBot feedback.
I don't know if anybody here has a strong opinion regarding template-engine or automatic build system... Python is for sure good, as most of the people here are familiar with it. And as long as I can build with conda, I am happy So I would vote for jinja and buildbot.
There is no big problem with python3 on mac, but I wouldn't make the matrix too complex, as in the end there will be only one version: python3-qt5.CADennis wrote:why is Python3 x QT5 x macOS not included in your matrix?
Can you give a short introduction in how the interface will look. Let's say I have a docker file which build FreeCAD in a conda-environment. What are the necessary files/steps to get automatic builds?