docker reference builds for Python3 / QT5 /Ubuntu16.04

Having trouble installing or compiling FreeCAD? Get help here.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Post Reply
User avatar
CADennis
Posts: 31
Joined: Tue Apr 18, 2017 10:12 am

docker reference builds for Python3 / QT5 /Ubuntu16.04

Post by CADennis »

hi there,

I wanted to debug a bit into the native code to see if it's worth porting parts of assembly2 from python to C++, rather than making the python code statically typed in the current situation of having to support both Py2.7 and Py3.5.

To my surprise, getting FreeCAD 0.17 (same with 0.16) compiled on a recent Ubuntu is much more complicated than for example compiling blender. :o
I tried to
1. start with the launchpad PPA build definition (https://git.launchpad.net/freecad/tree/ ... 5fd3ade1a2), copy all its dependencies into a Dockerfile, but

Code: Select all

FROM ubuntu:16.04
does not find libcoin60-dev. After I changed that to libcoin80-dev, cmake would pass and make would fail at VTK.
2. Thus I changed my Dockerfile to

Code: Select all

FROM ubuntu:14.04

but then even much much more fails during compilation.
3. I tested https://github.com/FreeCAD/FreeCAD/blob ... travis.yml in my own repository's Travis integrations and after temporary manual modifications to populate an incremental build cache to work around the 50-minute build time limit of Travis that succeeds. But it does of course use its very own, weird environment, included Python2.7 and lots of dev libs and ruby hacks.

4. The next step was to reuse the dependencies from .travis.yml and get that compiled within a docker container as a reference. Then migrate the build within the container to Python3, and finally use its configuration outside docker in my development environment, but I noticed some security vulnerabilities which I am not willing to accept for my local machine.

Code: Select all

#Install Eigen 3.3.3 to reduce compiler warnings
curl -L "http://bitbucket.org/eigen/eigen/get/3.3.3.tar.gz" | tar xvz && cd eigen-*
mkdir build && cd build
cmake -DCMAKE_C_FLAGS_RELEASE=-DNDEBUG -DCMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON ..
sudo make -j2 install
Your Travis automation downloads the Eigen3.3.3 via HTTP and does not check SHA256. I went to the Eigen Website and couldn't find any checksums :roll: . I wonder if you would accept those security fixes and have hq installed on the Travis build nodes. It seems the only secure way to get your desired version of Eigen3.3.3 onto Ubuntu14.x and Ubuntu16.x

Code: Select all

hg clone https://bitbucket.org/eigen/eigen/
hg pull && hg update 3.3.3
Don't panic! NormandC's official Ubuntu Launchpad build uses securely the apt dependency system to get Launchpad's recommended libeigen3-dev package. But I want to get my builds stabelized against your latest builds and I guess I will need Eigen in version 3.3.3. Still I wonder, what the strategy will be for the final Launchpad builds...

Any suggestions or easier ways?
I guess what happened is, that most active developers recently run on Windows and Mac (brew/homebrew).
But my motivation for helping in your open source project would be to make CAD available to all people around the world, including poor nations, not only to people who have no money left for commercial software after they have spent all on gadgets in the Apple store ;-)
I think FreeCAD is great, maybe the only tool available to poor nations, who reuse and recycle our outdated phased-out industrial waste as their vital infrastructure. Enabling those people to repair essential spare parts of phased-out machinery is a nice motivation for personal open source activity. :)

My biggest respect and appreciation of NormandC's hard work at https://forum.freecadweb.org/viewtopic. ... 47#p164247 - of course I tried to find my solution in that thread but unfortunately not easily transferable to docker it seems.

p.s.:
maybe we can collect and exchange some Dockerfiles here as reference, for Mint, CentOS, Ubuntu, Debian openSuse, Fedora - whatever you have got - I would then setup some docker-based continuous integration pipes in the cloud to give you some reports
User avatar
CADennis
Posts: 31
Joined: Tue Apr 18, 2017 10:12 am

Re: docker reference builds for Python3 / QT5 /Ubuntu16.04

Post by CADennis »

here is an example Dockerfile for Ubuntu14.04 with QT4 and Eigen3.3.3 that compiles all and tests pass

Code: Select all

FROM ubuntu:14.04

MAINTAINER Its Me <CADennis@freecadweb.org>

ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

# Add General Utilities
 
RUN apt-get update
    
RUN apt-get install -y \
    apt-utils 

RUN apt-get install -y \
    bzip2 \
    ca-certificates \
    git 

RUN apt-get install -y \
	doxygen                          \
	libboost1.55-dev                 \
	libboost-filesystem1.55-dev      \
	libboost-program-options1.55-dev \
	libboost-python1.55-dev          \
	libboost-regex1.55-dev           \
	libboost-signals1.55-dev         \
	libboost-system1.55-dev          \
	libboost-thread1.55-dev          \
	libcoin80                        \
	libcoin80-dev                    \
	liboce-foundation-dev            \
	liboce-modeling-dev              \
	liboce-ocaf-dev                  \
	liboce-ocaf-lite-dev             \
	liboce-visualization-dev         \
	libpyside-dev                    \
	libqtcore4                       \
	libshiboken-dev                  \
	libxerces-c-dev                  \
	libxmu-dev                       \
	libxmu-headers                   \
	libxmu6                          \
	libxmuu-dev                      \
	libxmuu1                         \
	netgen                           \
	netgen-headers                   \
	oce-draw                         \
	pyside-tools                     \
	python-dev                       \
	python-pyside                    \
	python-matplotlib                \
	qt4-dev-tools                    \
	qt4-qmake                        \
	shiboken                         \
	swig                             \
	libvtk6-dev                      \
	libmed-dev                       \
	libmedc-dev			\
	asciidoc

RUN apt-get install -y cmake
RUN apt-get install -y mercurial

WORKDIR ~
RUN \
  mkdir eigen3.3.3 &&\
  cd eigen3.3.3 &&\
  hg clone https://bitbucket.org/eigen/eigen &&\
  pwd &&\
  ls -lsa &&\
  cd eigen &&\
  pwd &&\
  ls -lsa &&\
  hg pull && hg update 3.3.3

RUN \
  cd eigen3.3.3/eigen &&\
  mkdir build &&\
  cd build &&\
  cmake -DCMAKE_C_FLAGS_RELEASE=-DNDEBUG -DCMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON .. &&\
  make install
 
RUN useradd -ms /bin/bash dummyuser

RUN git clone https://github.com/FreeCAD/FreeCAD.git free-cad-code --depth 100 

CMD [ "/bin/bash" ]
you could run it with

Code: Select all

docker build -t docker_for_freecad .
docker run -i docker_for_freecad bash <<'EOF'
cd free-cad-code
git fetch origin
get rebase origin/master
export CMAKE_ARGS="${CMAKE_OPTS} -DPYTHON_EXECUTABLE=/usr/bin/python"
export INSTALLED_APP_PATH="/usr/local/bin/FreeCAD"
mkdir build && cd build && cmake ${CMAKE_ARGS} -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} ../
make -j2 install
${INSTALLED_APP_PATH} --run-test 0 &&
${INSTALLED_APP_PATH} --log-file ~/FreeCAD_installed.log &
sleep 10 && pkill FreeCAD
cat ~/FreeCAD_installed.log
grep --file=../.log_errors ~/FreeCAD_installed.log ; [ $? == 1 ] && echo "No errors from .log_errors file found in the log after start from /usr/local/bin" || ( echo "Error from .log_errors found!" && false )
exit
EOF
User avatar
NormandC
Veteran
Posts: 18587
Joined: Sat Feb 06, 2010 9:52 pm
Location: Québec, Canada

Re: docker reference builds for Python3 / QT5 /Ubuntu16.04

Post by NormandC »

Hi,

First things first ;)
CADennis wrote:I wanted to debug a bit into the native code to see if it's worth porting parts of assembly2 from python to C++
IMHO that would be duplicating the work of the already existing official Assembly workbench. It is not in a working state at this time, but once the massive changes in the PartDesign workbench are finalized and 0.17 is released, I believe the plan is to resume work on it (provided there are developers with time to dedicate).

As for the heart of the matter, I'm belatedly replying here (sorry about that) because you did ask for my input in the Ubuntu PPA coordination topic. But you should know that my knowledge is very specific and mostly limited to Launchpad and debian packaging. In fact I don't even compile FreeCAD that often, I have no programming knowledge, software building skills nor scripting skills. I was only maintaining the PPA because nobody with real packaging skills ever stepped in to do the grunt work. I'm an end user and an experienced CAD user, but I'm not a developer, and all I've ever wanted was to get up-to-date packages on Ubuntu because that's the OS I use. I'd rather spend my free time on helping people learn FreeCAD and using it myself than building/packaging it.
CADennis wrote: start with the launchpad PPA build definition (https://git.launchpad.net/freecad/tree/ ... 5fd3ade1a2),
This is not the PPA build definition. This is a mirror of the Github FreeCAD repository, while it is used by the PPA to get the source code as it cannot import from Github directly, the debian folder inside it has not been used in years and it is not up-to-date.

The PPA uses a separate packaging repo found here: https://git.launchpad.net/~freecad-main ... tpackaging
The build recipe clones the mirror FreeCAD repo and merges the packaging branch into it to create a tarball and build files, the whole set of files being called a source package. Then the binary packages are built from it. The "dailybuild-occt" branch is the one currently used for the Daily Builds. netgen is disabled because a newer version needs to be packaged.
CADennis wrote:The next step was to reuse the dependencies from .travis.yml and get that compiled within a docker container as a reference. Then migrate the build within the container to Python3
The Ubuntu PPA is still built with Python 2.x. I have no idea if it can actually be compiled with Python 3. AFAIK efforts to fully port FreeCAD to python 3 aren't complete? :?:
CADennis wrote:NormandC's official Ubuntu Launchpad
It's not my official Ubuntu Launchpad... ;) While I was its lone maintainer for about 4 years, it's been 2 years now since I stopped my regular involvement. I only occasionally give a hand to sgrogan who took up the reins and also manages the Windows builds.
CADennis wrote:I guess I will need Eigen in version 3.3.3.
Why? What does 3.3.3 provide that the current versions found in the Ubuntu repositories don't? The comment mentions that it's only used to reduce the compiler warnings if I understand correctly. It may just be necessary or more convenient for travis, but the PPA uses older versions of Eigen without apparently any trouble.
CADennis wrote:Still I wonder, what the strategy will be for the final Launchpad builds...
I can't speak for sgrogan, but I can guess he will continue with the same "strategy" I've always followed: use the existing packages from the official Ubuntu repositories whenever possible, and if some library versions prove to be too old/problematic for FreeCAD, backport a newer version from the more recent Ubuntu releases or from Debian unstable/experimental if one exists; if there is none, we are left to try to package a new version from scratch from the source code. As can be seen in the [Ubuntu Daily PPA] Transitioning to OCCT7, VTK7... topic, that's not an easy process for either sgrogan or myself, so it's a last resort. Without the help received we'd still be stuck...
CADennis wrote:I guess what happened is, that most active developers recently run on Windows and Mac (brew/homebrew).
Actually, many developers work on GNU/Linux... It's just that almost none of them are spending time on packaging FreeCAD. These roles (developers and packagers) are separate.

In fact, I think that on Linux sgrogan and I were until recently the only packagers who were closely associated with the FreeCAD community. Guys who create packages for Fedora, OpenSuse and other distros are not very active here.
CADennis wrote:not only to people who have no money left for commercial software after they have spent all on gadgets in the Apple store ;-)
I understand that comment was in jest, but it was nonetheless completely off the mark. Mac users are actually in the minority among devs and community members, many of us are using GNU/Linux, and we are contributing to FreeCAD not because we don't have money in our pockets, but rather for philosophical reasons. :roll:
Last edited by NormandC on Thu Apr 27, 2017 7:55 pm, edited 1 time in total.
User avatar
NormandC
Veteran
Posts: 18587
Joined: Sat Feb 06, 2010 9:52 pm
Location: Québec, Canada

Re: docker reference builds for Python3 / QT5 /Ubuntu16.04

Post by NormandC »

CADennis wrote:here is an example Dockerfile for Ubuntu14.04

Code: Select all

   liboce-foundation-dev            \
   liboce-modeling-dev              \
   liboce-ocaf-dev                  \
   liboce-ocaf-lite-dev             \
   liboce-visualization-dev         \
The Ubuntu 14.04 repository contains an utterly obsolete version of OCE (0.15, based on OCCT 6.7.0). OCCT is at the very heart of FreeCAD, the most important third-party library, and with 0.17 it is more important than ever to have the latest version (7.1.0), because it has many bug fixes, and some new modelling features depend on OCCT 6.9.0 or later.

It is why for years I was re-packaging the latest OCE version available on Debian on the PPA, and why we recently decided to directly package OCCT 7.1.0 because OCE has fallen too far behind.

If you can import packages from PPAs in a Docker file, I recommend you get the libopencascade packages from it. EDIT: forgot to mention a caveat: OCCT 7.1.0 requires netgen 5.3.1 which is not in the PPA yet. Thus netgen support was removed from the Daily PPA for the time being.
User avatar
sgrogan
Veteran
Posts: 6499
Joined: Wed Oct 22, 2014 5:02 pm

Re: docker reference builds for Python3 / QT5 /Ubuntu16.04

Post by sgrogan »

Hi CADennis,
I echo what NormandC has said, I'm not a developer or expert packager either.
For eigen3 specifically, I think 3.2.2 is the minimum version necessary for full FreeCAD functionality. There are some sparse matrix techniques that the sketcher solver uses. There are some ifdef's in the code to disable this functionality if an older version is used. This is where the compiler warnings come from (you will also get warnings in the report view at run time) On the PPA we have packages for precise and trusty that were backported from xenial (I think maybe it was wily). For later versions the official repo's caught up.

There is a strong commitment that FreeCAD will build and work with the default libs in the official repo's of non-EOL os/distro's. As NormandC has pointed out sometimes the performance loss is such that the various package managers update the libs for distribution on their platform. OCC is a case across all platforms. QT5 is an example specific to OSX. Huge bugs on OSX so the QT port was critical. On win and linux, not so much, so from a packing perspective not much work has been done (it should work if you are compiling yourself, but you must handle the dependencies yourself as well)
"fight the good fight"
User avatar
CADennis
Posts: 31
Joined: Tue Apr 18, 2017 10:12 am

Re: docker reference builds for Python3 / QT5 /Ubuntu16.04

Post by CADennis »

NormandC wrote:Hi,
[...]
CADennis wrote: start with the launchpad PPA build definition (https://git.launchpad.net/freecad/tree/ ... 5fd3ade1a2),
This is not the PPA build definition. This is a mirror of the Github FreeCAD repository, while it is used by the PPA to get the source code as it cannot import from Github directly, the debian folder inside it has not been used in years and it is not up-to-date.

The PPA uses a separate packaging repo found here: https://git.launchpad.net/~freecad-main ... tpackaging
The build recipe clones the mirror FreeCAD repo and merges the packaging branch into it to create a tarball and build files, the whole set of files being called a source package. Then the binary packages are built from it. The "dailybuild-occt" branch is the one currently used for the Daily Builds. netgen is disabled because a newer version needs to be packaged. [...]
OK, I see now, I was totally lost in the
https://git.launchpad.net/freecad/
opposed to using your
https://git.launchpad.net/~freecad-maintainers/

As reference, mostly for myself, here are the direct links to the recipes:
https://code.launchpad.net/~freecad-mai ... s/+recipes
NormandC wrote: The Ubuntu PPA is still built with Python 2.x. I have no idea if it can actually be compiled with Python 3. AFAIK efforts to fully port FreeCAD to python 3 aren't complete?
Well we can start working against Python2, as long as everything else is up-to-date as of Ubuntu16.04. That is exactly where CI with side-by-side error reports shall kick in and then stabelize both versions in parallel. But I must clarify with you expert Ubuntu packagers and release responsibles upfront,

what YOUR Python3 will look like for a Ubuntu16.04, especially at compile time, I can derive the runtime dependencies then myself if you don't have them.
???


Across the migrations from QT4 to QT5 and the Python2 to Python3, in particular in the Python wrappers for QT there have been heavy restructurings not just renamings of packages. I just can't see, what packages for Python3/QT5 Ubuntu system builders like you expect to be used on an Ubuntu / Debian eventually.
NormandC wrote: It is why for years I was re-packaging the latest OCE version available on Debian on the PPA, and why we recently decided to directly package OCCT 7.1.0 because OCE has fallen too far behind.

If you can import packages from PPAs in a Docker file, I recommend you get the libopencascade packages from it. EDIT: forgot to mention a caveat: OCCT 7.1.0 requires netgen 5.3.1 which is not in the PPA yet. Thus netgen support was removed from the Daily PPA for the time being.
yes one can run

Code: Select all

add-apt-repository ppa:freecad-maintainers/freecad-daily
apt-get update
in docker. How would I notice when netgen gets re-activated? How do you guys trace and communicate such temporary things?
sgrogan wrote:For eigen3 specifically, I think 3.2.2 is the minimum version necessary for full FreeCAD functionality.
hoooly ... another build matrix dimension. I try to remember to setup both {Eigen3.3.3, Eigen 3.2.2}.
NormandC wrote: The "dailybuild-occt" branch is the one currently used for the Daily Builds.
hmmm, can't find it. Do you mean occt7-release instead ?
screenshot-freecad-maintainers.png
screenshot-freecad-maintainers.png (45.74 KiB) Viewed 4225 times
But "occt7-release" is described as only packaging dependencies for "freecad-daily". What do you mean exactly?
screenshot-freecad-maintainers2.png
screenshot-freecad-maintainers2.png (34.99 KiB) Viewed 4225 times
User avatar
NormandC
Veteran
Posts: 18587
Joined: Sat Feb 06, 2010 9:52 pm
Location: Québec, Canada

Re: docker reference builds for Python3 / QT5 /Ubuntu16.04

Post by NormandC »

Dude, just stop with the shouting. It's damn annoying.
CADennis wrote:But I must clarify with you expert Ubuntu packagers and release responsibles upfront,
sgrogan and I have already established we are not "experts". As for the upstream packagers (official Debian/Ubuntu/etc. repositories), they are not involved with FreeCAD at all and we don't have any contact with them.
CADennis wrote:what YOUR Python3 will look like for a Ubuntu16.04
Look, it's clear to me that you are still mixing development VS. packaging. They are not the same thing, and are not necessarily managed by the same people. sgrogan and I have already said we are not developers. We have not tried to compile FreeCAD with Python3 (nor with Qt5 for that matter, for the reasons explained by sgrogan). From the Python3 development topic which I saw today you posted to this week, it is clear that python3 support has not been merged to master yet. My understanding is that FreeCAD needs to be compiled with a specific branch (not master) to use Python 3. And it is nowhere near ready to be merged, from what I could read. triplus replied to you why it is currently not possible to use only the python3 packages from the Ubuntu repositories. pyside2 is required, but does not exist yet in the Debian/Ubuntu repositories. Is it even fully functional/released yet?!?

So please do not expect guidance from the Ubuntu PPA packagers about Python3 and Qt5, we don't have any to provide because we are not there yet. Until the former is merged to master, and until the latter becomes mandatory to build FreeCAD, no energy will be dedicated to this on the PPA. I can't speak for the other platforms.
CADennis wrote:How would I notice when netgen gets re-activated? How do you guys trace and communicate such temporary things?
In the forum topic I already linked. You can subscribe to it to get email notifications of activity.
CADennis wrote:
NormandC wrote:The "dailybuild-occt" branch is the one currently used for the Daily Builds.
hmmm, can't find it. Do you mean
No. I gave you this link to the Git repo:
NormandC wrote:The PPA uses a separate packaging repo found here: https://git.launchpad.net/~freecad-main ... tpackaging
What you can see when you click on it:
FC_launchpad_freecad-maintainers_gitpackaging_01.png
FC_launchpad_freecad-maintainers_gitpackaging_01.png (88.67 KiB) Viewed 4198 times
CADennis wrote:Do you mean occt7-release instead ?
No. Please read the description in the link you posted. I would think it's pretty self-explanatory.
The occt7-release recipe wrote:This is the build recipe for the libopencascade packages based on OCCT7 to be used by the freecad-daily PPA.
CADennis wrote:But "occt7-release" is described as only packaging dependencies for "freecad-daily". What do you mean exactly?
No, it's not described as "only packaging dependencies", it's specifically and clearly described as a recipe to build the libopencascade packages, nothing else.
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: docker reference builds for Python3 / QT5 /Ubuntu16.04

Post by looo »

NormandC wrote:And it is nowhere near ready to be merged, from what I could read. triplus replied to you why it is currently not possible to use only the python3 packages from the Ubuntu repositories. pyside2 is required, but does not exist yet in the Debian/Ubuntu repositories. Is it even fully functional/released yet?!?
pyside2 is qt5 specific! and pyside is available in ubuntu:

Code: Select all

sudo apt-get install python3-pyside
User avatar
NormandC
Veteran
Posts: 18587
Joined: Sat Feb 06, 2010 9:52 pm
Location: Québec, Canada

Re: docker reference builds for Python3 / QT5 /Ubuntu16.04

Post by NormandC »

My mistake then.
User avatar
CADennis
Posts: 31
Joined: Tue Apr 18, 2017 10:12 am

Re: docker reference builds for Python3 / QT5 /Ubuntu16.04

Post by CADennis »

zaleksf wrote:Hi, CADennis, and thanks for your tip, yorik:
I applied the '-lmpi_cxx' flag to the CMAKE_CXX_COMPILER setting in CMake and FreeCAD built completely. And it even starts-up!
Now I need to play with it a little bit.
So, using vanilla-flavored Debian Stretch (as of today) and OCCT-7.1.0 libraries the FreeCAD git-master version (as of yesterday) will compile and run.
Now I want to figure out how to get netgen-5.3.1 working and I would like to transition my FreeCAD build to Python3 and QT5.
Thanks for all your work. My CMake cache file is included, if that provides info on how my system was set up.
(collected from https://forum.freecadweb.org/viewtopic. ... 58#p172058)
Attachments
CMakeCache.txt
(96.21 KiB) Downloaded 92 times
Post Reply