@looo why is Python3 x QT5 x macOS not included in your matrix? has there been trouble and common agreement to skip those newer dependencies on Apple? Looking forward to your BuildBot feedback. Please be aware of BuildBot's security concept. It has no security and must be stacked ontop of a secure IRC infrastructure. No critique, just keep that in mind while plowing through the features: (1) get dependencies, (2) unplug network, (3) switch to non-priviledged user (4) start the bots ... I don't want to lose you as the main driver behind Python3
- buildbot_teaser.png (26.44 KiB) Viewed 1293 times
here is a security setup tutorial for Jenkins, by Microsoft:
https://jenkins.io/blog/2017/04/20/secu ... -on-azure/
I must admit, I have not used Jenkins outside a corporate DMZ before, so that tutorial from Microsoft came just in
time.
Sure I had to go grab a glass of fresh water and open the window when I saw that the security documentation for open source JAVA software is now sponsored by Microsoft.
Be careful and disciplined not to learn and use all the features of BuildBot just because they are there: don't!!! Fans have accumulated many non-functional wrappers. Wrappers that don't add functionality to the tool underneath but are only meant as convenience bridges to Python coders. From my industrial experience, such wrappers just come into your way during future maintenance, if there is no one to blame or financially involve. Here is an example: the cmake convenience wrapper
Code: Select all
from buildbot.plugins import steps
factory.addStep(
steps.CMake(
generator='Ninja',
definitions={
'CMAKE_BUILD_TYPE': Property('BUILD_TYPE')
},
options=[
'-Wno-dev'
]
)
)
Wrong design. --> Want to use it for MS Visual Studio 2017? Forget that! Wait until the open source owner adds it. Until then, patch and sync and integrate ... I think you have had enough of that recently. No fun. Avoid nonfunctional handwritten convenience wrappers by thirdparties around essential tools! Now to something completely different. Here is a Python REST convenience wrapper around Jenkins
https://python-jenkins.readthedocs.io/e ... of-jenkins
Fair enough, don't miss their cool new
Blue Ocean Jenkins pipe framework. Downside: Jenkinsfile is a DSL. Widespread Intellisense support in the future? - nope! Yet the pipes can be edited via GUI, maybe nice for "serious C++ code crunchers" who are not so involved in daily Python tasks. When after 5 years someone has to dig back into a fellow programmer's build automation, it is nice _not_ to find Turing-complete dynamic scripting languages but a visual editor. Yet the safest practice is betting on the availability of a Python editor in 5 years when a CI plugin is no longer so hyped that its visual editors work.
- jenkins_teaser.png (59.18 KiB) Viewed 1293 times
For both, BuildBot and Jenkins it is commonly recommended to restrict tool integration to
(1) setting up environment variables
(2) generating or manually maintaining tool input parameter files (e.g. makefiles, shell scripts, ini files, ...
(3) integrating a single line for the generic tool invocation, for instance
Code: Select all
from buildbot.plugins import util, steps
f = util.BuildFactory()
freecadOpts = ourSuperDupaPipeLogic() //...
f.addSteps([
steps.SVN(repourl="http://svn.example.org/Trunk/"),
steps.ShellCommand(command=["cmake", freecadOpts])
])
That motivates a template engine for us at FreeCAD. So my question to Python power users here: what would be your favourite template engine?
- mako,
- django,
- jinja
- what else?
@triplus: please see my PM
@kkremitzki: I just had a quick look at the travis_shim that you mentioned ... . I don't like its concept. It "consumes Travis YAML" and produces BuildBot configurations. My recommendation is
the other way around: consume Python-script and generate template-engine-based instructions for build tools and deployment systems. Conda packages, PIP packages, MSI packages, homebrew, APT, RPM, etc. They all need similar dependency definitions. I don't see how we could specify the following workflow in Travis Yaml and feed that into BuildBot, I think it will only work in the opposite direction:
(1) Branches are created automatically per pull request - maybe even for each build configuration a separate sub-branch.
(2) They get removed after PR completion.
Git allows to delete branches without leftovers after usage. That is Git's advantage over Subversion. BuildBot dynamically generates into them the specific build job definition, as temporarily required.
And we can exploit that flexibility of dynamically committing build instructions into branches to even "hack" around the Travis time limit and continue to use its OSX service: Each PR-job macOS subbranch builds up the cache by repeatedly committing increasingly complete build definitions into such branches with activated caching. CI bots define explicitly, when and how this cache is cleaned. In the running example that would be
branch "PR1._":
Code: Select all
{"clean cache", "PR1.0.macOS.build dependencies", "PR1.0.macOS.build core", "PR1.0.macOS.build addon1", ... ,"PR1.0.macOS.build addonN"}
{"clean cache", "PR1.1.macOS.build dependencies", "PR1.1.macOS.build core", "PR1.1.macOS.build addon1", ... ,"PR1.1.macOS.build addonN"}
branch "PR2._":
{"clean cache", "PR2.0.macOS.build dependencies", "PR2.0.macOS.build core", "PR2.0.macOS.build addon1", ... ,"PR2.0.macOS.build addonN"}
branch "PR3._":
{"clean cache", "PR3.0.macOS.build dependencies", "PR3.0.macOS.build core", "PR3.0.macOS.build addon1", ... ,"PR3.0.macOS.build addonN"}
...
Though I was initially preferring Jenkins, I'll help in what the majority here prefers. Not only Chrome, Firefox and Wireshark use BuildBot, even Blender. And we might ally with them for change requests. I have seen, they already
hacked workarounds into stock BuildBot.
FreeCAD
What is the infrastructure behind freecadweb.org? Does it run on Amazon AWS, Azure, Oracle? Could we then reuse unmetered but limited capacity from it? I would help you in setting up Jenkins/BuildBot + GitHub-Plugin + docker to run on EC2 or similar. , independent of their proprietary pipe services, which FreeCAD might opt for, later anytime.