Libpack/Packagemanager for windows

[apeltauer]

Hi all,

My name is Manuel and for over a year I am building FreeCAD from source with the help of the Libpack 11.5 for windows. Since a while I have added my own compiled opencascade libs. Also I have added boost. Right now I handle the packages with Conan (https://conan.io)

Now I want to change qt from 4.8 to 5 and their for I want to start a thread to talk about how to handle the 3rd party libs on windows.

I hope some people who are also compiling on windows want to join the discussion how the lib stuff can be handled on windows. Also for me it is very important to have the debug libs....

Hope for some reply’s...

Br Manuel

[sgrogan]

@wmayer

@looo

@peterl94

Hope for some reply’s...

Hi all,
Manuel and I started this discussion on PM, but it's better here.
What do you think is the best strategy?

(1) Should we try to upgrade the VS2013 libpacks, we will still have the incompatibility with PyPi, but could re-use some of the debug stuff already in the 11.x based LibPacks.

(2) Switch to VS2015 and be aligned with PyPi/UCRT? If so do we leverage all looo's work on conda and try to expand to add debug libs. If so how do we handle QT on Conda?

(3) Switch to VC2015 and build the LibPack from scratch. If we choose this path how much should we use the official packages or should we just except compiling everything from scratch?

[looo]

(2) Switch to VS2015 and be aligned with PyPi/UCRT? If so do we leverage all looo's work on conda and try to expand to add debug libs. If so how do we handle QT on Conda?

Debug builds would be nice to have. But I invest already too much time into updating the builds to the new compilers, so I don't see much chance that I work on debug builds.

In principle it should not be very difficult to create a debug-build with conda. ed.: for qt:

Code: Select all

- Fork the feedstock: https://github.com/conda-forge/qt-feedstock
- change to "debug"
    unix:
    - https://github.com/conda-forge/qt-feedstock/blob/master/recipe/build.sh#L95
    - https://github.com/conda-forge/qt-feedstock/blob/master/recipe/build.sh#L153
    windows:
    - https://github.com/conda-forge/qt-feedstock/blob/master/recipe/bld.bat#L148 ?
    - https://github.com/conda-forge/qt-feedstock/blob/master/recipe/bld.bat#L157
- increase build-number:
    - https://github.com/conda-forge/qt-feedstock/blob/master/recipe/meta.yaml#L26
- remove windows-skipping:
    - https://github.com/conda-forge/qt-feedstock/blob/master/recipe/meta.yaml#L27
- install miniconda:
    - install conda-build
    - install anaconda-client
- install compilers (windows)
    - https://conda-forge.org/docs/buildwin.html#local-testing
- run conda-build
- fix all build-errors
- upload to "freecad/label/debug"

My problem is also that I don't understand how debug-builds can be mixed with release builds. My understanding is that every library has to be build in debug mode to get a full debug-build of freecad. It would be nicer if we can do it step by step.

edit: I tried this recipe but the build-process is already stopped during extracting the download... which is a bit frustrating.

[wmayer]

Right now I handle the packages with Conan (https://conan.io)

Never heard of it before. What exactly is it good for and what are its benefits? Just by looking at their website I could not figure out that.

Also for me it is very important to have the debug libs....

Yes, debug libs are important for me, too. Does conan support a way to build the boost.python debug library with the debug version of Python? In the past this was one of the most challenging stuff when building the libpack.
Another major problem is to get the QtWebEngineCore dll. When building the binaries somewhere in a sub-directory the build simply stopped because some file names exceeded the maximum length of 260 characters. After figuring this out I could build the release version but apparently the dlls are corrupted. Whenever loading them FreeCAD crashes. And building the debug libs never worked for me because the linker has never finished its work.
So, if conan could help in these cases that would be awesome.

Manuel and I started this discussion on PM, but it's better here.
What do you think is the best strategy?

I think it is time to move to VS2015 because of these reasons:

• At the moment the mesh flattening uses pybind11 which on Windows can only be built with VS2015 or later. With boost.python you can only work around some of pybind's functionality but not all of them.
• Then the latest netgen versions uses some C++14 specific keywords which makes building with VS2013 impossible.
• Recent Qt versions cannot be built any more with VS2013. IIRC the last supported version was 5.10.
• Also Python 3.6 or higher requires VS2015. Some time ago I succeeded to build it with VS2013 but I had to change some C code to make it working.

My problem is also that I don't understand how debug-builds can be mixed with release builds.

You never should do this because usually this makes your application to crash. The behaviour is similar to what we had recently with two different Coin3d versions under Linux.

[apeltauer]

Conan is a plugin for JFrog artifactory, spicily for c++ binaries. (https://bintray.com/) (https://conan.io/)
I use it to handle my binary libs and dependencies, e.g. OCC, boost...

After setting up a server (or use the conan-center (https://bintray.com/conan/conan-center)) you are able to use the libs in cmake.
For example in my cmake file this looks like this:

"include(conan.cmake)
conan_cmake_run(REQUIRES OpenCascade/7.0@OpenCascade/7.0
BASIC_SETUP
BUILD missing)
#link_directories(${OCC_LIBRARY_DIR})
include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
#conan_basic_setup()

set(OCC_INCLUDE_DIR ${CONAN_OPENCASCADE_ROOT}/include)
set(OCC_LIBRARY_DIR_DEBUG ${CONAN_OPENCASCADE_ROOT}/libd)
set(OCC_LIBRARY_DIR_RELEASE ${CONAN_OPENCASCADE_ROOT}/lib)

If i want to change the OCC version i just have to update the name of the package.
Cmake looks for the correct package and path which e.g. is
"*User*\.conan\data\OpenCascade\7.0\OpenCascade\7.0\package\88a8c94ba0d38ce69afccbcf0043e87eb56f222b\"
88a8c94ba0d38ce69afccbcf0043e87eb56f222b --> is a hash of the config i have created the package:

[settings]
arch=x86_64
build_type=Debug
compiler=Visual Studio
compiler.runtime=MDd
compiler.version=12
os=Windows

So it is possible to create the same package with different configs, e.g. different os, different compiler or different build types. And these packages can have their own binaries for that config.
The config (settings) are generating different hashs.
While running cmake, a script will create the the config hash at runtime with the current cmake config. This hash is used to find the correct package. If it is already local it will use this, if not it will be downloaded and then used.

I am not a conan expert and i am using conan just for a few packages, but for that it works like a charm.

[looo]

You never should do this because usually this makes your application to crash. The behaviour is similar to what we had recently with two different Coin3d versions under Linux.

Ok, I understand. So if we build qt in debug mode, we also have to rebuild every library which freecad or deps link to and which depends on qt. Most-likely in the end we have to recompile every library in debug-mode which is a huge task.

Conda-forge recipes are designed to allow building in whatever mode one wants, but the real problem is the explosion of the build-matrix. Currently for example netgen is build for py3.6, py3.7 / win, linux(system gcc, anaconda gcc), osx (system clang, anaconda-clang) -> 10 builds. If we now add another option to build in debug or release mode we end up with 20 builds . Porting the build-recipes to newer versions is allready very difficult (mostly because of the big build-matrix).

So from my point of view conda can work as a base for development/debug builds but first I would like to update all FreeCAD-deps to py3.7 and new-toolchain.


So if you want a fast solution for py3-debug-builds conda-forge is not an option. (If others are joining the conda-forge community (mac-guys) and help with packaging more things in parallel are possible...) If you want a more consistent/future-prove solution I guess working on conda is the best option.

[wmayer]

If it is already local it will use this, if not it will be downloaded and then used.

Does this mean that it download pre-built binaries of the requested library? Or do you still have to build the libraries locally?

[apeltauer]

In my case i have compiled OCC version 6.9, 7.0, 7.1 and 7.3 and then i have created a package for the versions and uploaded the packages to my conan server.
Since then i work with the compiled binaries.
Also it its very handy for me because i have a compile machine where i do ci. If i have a new package i just upload it to my conan server and the compile machine just downloads the package....
Also modifying a package (add a dll) is easy and i have to do that just once. After uploading the package again it works for all my machines....

[looo]

I tried to build qt in debug mode with the conda-forge-recipes. Linux worked at first try but the final result is about 500mb... For windows I didn't succeed in building debug-qt. First I had difficulties with extracting the download. After updating windows and the base environment, switching to developer-mode in windows, it was able to start the build. But the build was not successfully. But people from anaconda and conda-forge are also interested in a debug builds of qt. So I guess they will help with building the library.)

But in the end it is simple too much work for me to get this done. So I guess one of the other options should be used. I had a short look at conan and it look like a good alternative with a lot of recipes already available. Compared to conda it seems to be fully open-source. But I guess the work that has to be done is also huge. Maybe someone can give a short summary what's necessary to get conan ready for FreeCAD...

Porting the libpack is also a big task. Also here it would be interesting to have a short summary of the necessary tasks. In the end I guess everything is possible, but the work simple has to be done.

[sgrogan]

Porting the libpack is also a big task. Also here it would be interesting to have a short summary of the necessary tasks. In the end I guess everything is possible, but the work simple has to be done.

I think the task is the same for all approaches; build the whole FreeCAD dependency stack in debug mode, ideally x32/x34, release/debug, 4X4 build matrix.

To me conan looks like an alternative to conda, I don't see anything that offers more. It might be easier to make it portable.

The advantage of conda is that you've already built the release stack. We would probably need to build some of the debug versions of the deeper dependencies that are already available in release mode. Using Pybind11 works around the libboost-python issue, as it's a header only lib. Also conda helps with dependencies of dependencies and cross(common) dependencies. The drawback is no x32 support.

Building the Libpack from scratch offers the possibility to use some pre-compiled binaries from official releases. i.e. I downloaded the official PY37 Win installer and there are options to install all the Python debug libs/inculdes. I will try QT5 next but it's important that they share a compatible ssl.

My personal opinion is we should examine conda first. It's a huge task anyway we look at it, but at least the conda trail has been blazed in release mode.