Thank your for your great input guys!
I summarize, that I should get started with BuildBot+Jinja from the discussed choices {Jenkins,BuildBot} x {mako, django, jinja}. Please let me know ASAP if anyone prefers an alternative.
Let me design a prototype and then discuss its demo so it is less theoretical. I need to consider security and authentication aspects. Early feedback is always best, so here is a rough draft:
We will establish additional git repositories, "freecad-ci-master" and several "freecad-build-<targetX>.git".
Code: Select all
freecad-ci-master.git
- master
- bot_generic
- pipes
Code: Select all
freecad-build-<targetX>.git
-* freecad-build-master.git # submodule
-* freecad.git # submodule
- bot
- alien
Concrete targets are
Code: Select all
freecad-build-conda.git
freecad-build-windows.git
freecad-build-macos.git
freecad-build-rpm.git
freecad-build-deb-ppa.git
(have a look at
http://git.blender.org/gitweb/ if you think it is overengineered for a Cpp-Python opensource project)
Each build target repository consists of jinja templates and distributor-specific artefacts, maintained in the subfolder "alien". And the bot definition itself, subclassed from bot_generic. The bot feeds provided dependency and configuration tensors ("build matrix") into its template engines. The content of "alien" might be as little as one template for the shell script you drafted above and one Dockerfile, or as much as the entire folder
https://git.launchpad.net/freecad/plain/package/debian/ where only
https://git.launchpad.net/freecad/plain ... an/control plus some more files are templates, the rest to be copied as is. The existing repository
https://github.com/FreeCAD/homebrew-freecad could be migrated into one such "targetX" repository, e.g. "freecad-build-macos.git". Each buildbot job then checks its "freecad-build-<targetX>.git" out and pulls the submodules "freecad.git" and "freecad-ci-master.git".
I split the additions into separate repositories for authentication modalities. Wmayer might want to handle PRs from system builders as guaranteed to be separated from the
master bot and
pipe definitions. Doing so, bot-modifications can only cause harm inside restricted build containers. Wmayer can handle modifications to
master bot and
pipe definitions much more slow-paced and have them reviewed by several developers first. The repositories of type "freecad-build-<targetsX>.git" are
_NEVER_ pulled as submodules into "freecad-ci-master.git", but vice versa. Thus, buildmasters never run any code from "freecad-build-<targetX>.git" inside their cloud containers. Furthermore, all PPA launchpad maintainers can get write access granted to the repository "freecad-build-deb-ppa.git" but no one else. So they can comfortably trust its contents when synchronizing it into the
real Launchpad repository.
p.s. @looo:
looo wrote:[...]
this is how I build the py3-branch with docker and conda at the moment:
Code: Select all
> sudo docker run -i -t --name conda-forge condaforge/linux-anvil
## once the container runs interactively
> yum install -y libXt-devel libXmu-devel libXi-devel mesa-libGLU-devel rsync
> conda config --add channels freecad
> git clone https://github.com/looooo/FreeCAD_Conda
> cd FreeCAD_Conda/FreeCAD
> conda build . --python=3.5
## after the build is finished open another terminal and copy the build:
> sudo docker cp conda-forge:path_to_build place_to_store_the_build
Basing the build on "condaforge/linux-anvil" and then simply pulling a few remainders, is ultra-productive for us as a developers. And it provides me great insights into your current dependencies. In that it is very helpful, I asked you for such reference builds, thank you!
With all its opinionated preinstallments, CONDA will make any rule-based processing of a common package-dependency declaration across all OSes pretty rediculous, of course. Are you aware of their commercial freemium strategy?
[...] Conda is also included in the Continuum subscriptions of Anaconda, which provide on-site enterprise package and environment management for Python, R, Node.js, Java, and other application stacks. Conda is also available on pypi, although that approach may not be as up-to-date. [...]
You don't mind that? Do you merely use CONDA for personal, efficient development or do you wish to establish this as an official distribution channel for FreeCAD? I am a bit skeptical towards Continuoum Analytics and their freemium ecosystem. Found them too commercial and proprietary. Python end users and scholars strongly prefer CONDA over the more open, vendor-independent PIP. I noticed.
But that is not very different from a "Microsoft Academic Alliance Program", put under the facade of "freemium open source". After their business model will have eventually been terminated, they might leave that landscape at ruins to truly free open-source developers. Such infrastructures come and go:
codehaus-maven-service-unavailable. Or take .Net MONO. See Xamarin's license cost per seat in 2015. In the meantime aquired by Microsoft. Open source does not mean free. Continuum Analytics might just commercially exploit the temporary chaos and portability pressure introduced by Python2.7-to-Python3 incompatibilities, paired with academic involvement. That will eventually settle, unless they manage to hijack the next-gen Python userbase in the meantime. Still, I can understand any modern Python developer who wants to consume that productive ecosystem right now. So I will make sure, to integrate your CONDA layer into my prototypes.
From a heavy C++ system builder's OS dist point of view: if CONDA introduces a package distribution system redundantly stacked ontop of Win/Mac/Deb/Rpm, motivated from the very thin layer of a freemium installer wrapper around a scripting language around a C++ core, we should make sure we get FreeCAD deployed on an Ubuntu and Fedora via standard Linux package managers standalone. OPEN CASCADE SAS stood the test of time regarding FREE and fair open source involvement: 25 years. Continuum Analytics hasn't yet.
Let's find a strategy that runs another 20 years, honouring the great achievement of the people involved up to now, like you.
kind regards
Dennis