Libpack/Packagemanager for windows
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Be nice to others! Respect the FreeCAD code of conduct!
Libpack/Packagemanager for windows
Hi all,
My name is Manuel and for over a year I am building FreeCAD from source with the help of the Libpack 11.5 for windows. Since a while I have added my own compiled opencascade libs. Also I have added boost. Right now I handle the packages with Conan (https://conan.io)
Now I want to change qt from 4.8 to 5 and their for I want to start a thread to talk about how to handle the 3rd party libs on windows.
I hope some people who are also compiling on windows want to join the discussion how the lib stuff can be handled on windows. Also for me it is very important to have the debug libs....
Hope for some reply’s...
Br Manuel
My name is Manuel and for over a year I am building FreeCAD from source with the help of the Libpack 11.5 for windows. Since a while I have added my own compiled opencascade libs. Also I have added boost. Right now I handle the packages with Conan (https://conan.io)
Now I want to change qt from 4.8 to 5 and their for I want to start a thread to talk about how to handle the 3rd party libs on windows.
I hope some people who are also compiling on windows want to join the discussion how the lib stuff can be handled on windows. Also for me it is very important to have the debug libs....
Hope for some reply’s...
Br Manuel
Re: Libpack/Packagemanager for windows
Hi all,
Manuel and I started this discussion on PM, but it's better here.
What do you think is the best strategy?
(1) Should we try to upgrade the VS2013 libpacks, we will still have the incompatibility with PyPi, but could re-use some of the debug stuff already in the 11.x based LibPacks.
(2) Switch to VS2015 and be aligned with PyPi/UCRT? If so do we leverage all looo's work on conda and try to expand to add debug libs. If so how do we handle QT on Conda?
(3) Switch to VC2015 and build the LibPack from scratch. If we choose this path how much should we use the official packages or should we just except compiling everything from scratch?
"fight the good fight"
Re: Libpack/Packagemanager for windows
Debug builds would be nice to have. But I invest already too much time into updating the builds to the new compilers, so I don't see much chance that I work on debug builds.
In principle it should not be very difficult to create a debug-build with conda. ed.: for qt:
Code: Select all
- Fork the feedstock: https://github.com/conda-forge/qt-feedstock
- change to "debug"
unix:
- https://github.com/conda-forge/qt-feedstock/blob/master/recipe/build.sh#L95
- https://github.com/conda-forge/qt-feedstock/blob/master/recipe/build.sh#L153
windows:
- https://github.com/conda-forge/qt-feedstock/blob/master/recipe/bld.bat#L148 ?
- https://github.com/conda-forge/qt-feedstock/blob/master/recipe/bld.bat#L157
- increase build-number:
- https://github.com/conda-forge/qt-feedstock/blob/master/recipe/meta.yaml#L26
- remove windows-skipping:
- https://github.com/conda-forge/qt-feedstock/blob/master/recipe/meta.yaml#L27
- install miniconda:
- install conda-build
- install anaconda-client
- install compilers (windows)
- https://conda-forge.org/docs/buildwin.html#local-testing
- run conda-build
- fix all build-errors
- upload to "freecad/label/debug"
edit: I tried this recipe but the build-process is already stopped during extracting the download... which is a bit frustrating.
Re: Libpack/Packagemanager for windows
Never heard of it before. What exactly is it good for and what are its benefits? Just by looking at their website I could not figure out that.Right now I handle the packages with Conan (https://conan.io)
Yes, debug libs are important for me, too. Does conan support a way to build the boost.python debug library with the debug version of Python? In the past this was one of the most challenging stuff when building the libpack.Also for me it is very important to have the debug libs....
Another major problem is to get the QtWebEngineCore dll. When building the binaries somewhere in a sub-directory the build simply stopped because some file names exceeded the maximum length of 260 characters. After figuring this out I could build the release version but apparently the dlls are corrupted. Whenever loading them FreeCAD crashes. And building the debug libs never worked for me because the linker has never finished its work.
So, if conan could help in these cases that would be awesome.
I think it is time to move to VS2015 because of these reasons:Manuel and I started this discussion on PM, but it's better here.
What do you think is the best strategy?
- At the moment the mesh flattening uses pybind11 which on Windows can only be built with VS2015 or later. With boost.python you can only work around some of pybind's functionality but not all of them.
- Then the latest netgen versions uses some C++14 specific keywords which makes building with VS2013 impossible.
- Recent Qt versions cannot be built any more with VS2013. IIRC the last supported version was 5.10.
- Also Python 3.6 or higher requires VS2015. Some time ago I succeeded to build it with VS2013 but I had to change some C code to make it working.
You never should do this because usually this makes your application to crash. The behaviour is similar to what we had recently with two different Coin3d versions under Linux.My problem is also that I don't understand how debug-builds can be mixed with release builds.
Re: Libpack/Packagemanager for windows
Conan is a plugin for JFrog artifactory, spicily for c++ binaries. (https://bintray.com/) (https://conan.io/)
I use it to handle my binary libs and dependencies, e.g. OCC, boost...
After setting up a server (or use the conan-center (https://bintray.com/conan/conan-center)) you are able to use the libs in cmake.
For example in my cmake file this looks like this:
Cmake looks for the correct package and path which e.g. is
"*User*\.conan\data\OpenCascade\7.0\OpenCascade\7.0\package\88a8c94ba0d38ce69afccbcf0043e87eb56f222b\"
88a8c94ba0d38ce69afccbcf0043e87eb56f222b --> is a hash of the config i have created the package:
[settings]
arch=x86_64
build_type=Debug
compiler=Visual Studio
compiler.runtime=MDd
compiler.version=12
os=Windows
So it is possible to create the same package with different configs, e.g. different os, different compiler or different build types. And these packages can have their own binaries for that config.
The config (settings) are generating different hashs.
While running cmake, a script will create the the config hash at runtime with the current cmake config. This hash is used to find the correct package. If it is already local it will use this, if not it will be downloaded and then used.
I am not a conan expert and i am using conan just for a few packages, but for that it works like a charm.
I use it to handle my binary libs and dependencies, e.g. OCC, boost...
After setting up a server (or use the conan-center (https://bintray.com/conan/conan-center)) you are able to use the libs in cmake.
For example in my cmake file this looks like this:
If i want to change the OCC version i just have to update the name of the package."include(conan.cmake)
conan_cmake_run(REQUIRES OpenCascade/7.0@OpenCascade/7.0
BASIC_SETUP
BUILD missing)
#link_directories(${OCC_LIBRARY_DIR})
include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
#conan_basic_setup()
set(OCC_INCLUDE_DIR ${CONAN_OPENCASCADE_ROOT}/include)
set(OCC_LIBRARY_DIR_DEBUG ${CONAN_OPENCASCADE_ROOT}/libd)
set(OCC_LIBRARY_DIR_RELEASE ${CONAN_OPENCASCADE_ROOT}/lib)
Cmake looks for the correct package and path which e.g. is
"*User*\.conan\data\OpenCascade\7.0\OpenCascade\7.0\package\88a8c94ba0d38ce69afccbcf0043e87eb56f222b\"
88a8c94ba0d38ce69afccbcf0043e87eb56f222b --> is a hash of the config i have created the package:
[settings]
arch=x86_64
build_type=Debug
compiler=Visual Studio
compiler.runtime=MDd
compiler.version=12
os=Windows
So it is possible to create the same package with different configs, e.g. different os, different compiler or different build types. And these packages can have their own binaries for that config.
The config (settings) are generating different hashs.
While running cmake, a script will create the the config hash at runtime with the current cmake config. This hash is used to find the correct package. If it is already local it will use this, if not it will be downloaded and then used.
I am not a conan expert and i am using conan just for a few packages, but for that it works like a charm.
Re: Libpack/Packagemanager for windows
Ok, I understand. So if we build qt in debug mode, we also have to rebuild every library which freecad or deps link to and which depends on qt. Most-likely in the end we have to recompile every library in debug-mode which is a huge task.wmayer wrote:You never should do this because usually this makes your application to crash. The behaviour is similar to what we had recently with two different Coin3d versions under Linux.
Conda-forge recipes are designed to allow building in whatever mode one wants, but the real problem is the explosion of the build-matrix. Currently for example netgen is build for py3.6, py3.7 / win, linux(system gcc, anaconda gcc), osx (system clang, anaconda-clang) -> 10 builds. If we now add another option to build in debug or release mode we end up with 20 builds . Porting the build-recipes to newer versions is allready very difficult (mostly because of the big build-matrix).
So from my point of view conda can work as a base for development/debug builds but first I would like to update all FreeCAD-deps to py3.7 and new-toolchain.
So if you want a fast solution for py3-debug-builds conda-forge is not an option. (If others are joining the conda-forge community (mac-guys) and help with packaging more things in parallel are possible...) If you want a more consistent/future-prove solution I guess working on conda is the best option.
Re: Libpack/Packagemanager for windows
Does this mean that it download pre-built binaries of the requested library? Or do you still have to build the libraries locally?If it is already local it will use this, if not it will be downloaded and then used.
Re: Libpack/Packagemanager for windows
In my case i have compiled OCC version 6.9, 7.0, 7.1 and 7.3 and then i have created a package for the versions and uploaded the packages to my conan server.
Since then i work with the compiled binaries.
Also it its very handy for me because i have a compile machine where i do ci. If i have a new package i just upload it to my conan server and the compile machine just downloads the package....
Also modifying a package (add a dll) is easy and i have to do that just once. After uploading the package again it works for all my machines....
Since then i work with the compiled binaries.
Also it its very handy for me because i have a compile machine where i do ci. If i have a new package i just upload it to my conan server and the compile machine just downloads the package....
Also modifying a package (add a dll) is easy and i have to do that just once. After uploading the package again it works for all my machines....
Re: Libpack/Packagemanager for windows
I tried to build qt in debug mode with the conda-forge-recipes. Linux worked at first try but the final result is about 500mb... For windows I didn't succeed in building debug-qt. First I had difficulties with extracting the download. After updating windows and the base environment, switching to developer-mode in windows, it was able to start the build. But the build was not successfully. But people from anaconda and conda-forge are also interested in a debug builds of qt. So I guess they will help with building the library.)
But in the end it is simple too much work for me to get this done. So I guess one of the other options should be used. I had a short look at conan and it look like a good alternative with a lot of recipes already available. Compared to conda it seems to be fully open-source. But I guess the work that has to be done is also huge. Maybe someone can give a short summary what's necessary to get conan ready for FreeCAD...
Porting the libpack is also a big task. Also here it would be interesting to have a short summary of the necessary tasks. In the end I guess everything is possible, but the work simple has to be done.
But in the end it is simple too much work for me to get this done. So I guess one of the other options should be used. I had a short look at conan and it look like a good alternative with a lot of recipes already available. Compared to conda it seems to be fully open-source. But I guess the work that has to be done is also huge. Maybe someone can give a short summary what's necessary to get conan ready for FreeCAD...
Porting the libpack is also a big task. Also here it would be interesting to have a short summary of the necessary tasks. In the end I guess everything is possible, but the work simple has to be done.
Re: Libpack/Packagemanager for windows
I think the task is the same for all approaches; build the whole FreeCAD dependency stack in debug mode, ideally x32/x34, release/debug, 4X4 build matrix.
To me conan looks like an alternative to conda, I don't see anything that offers more. It might be easier to make it portable.
The advantage of conda is that you've already built the release stack. We would probably need to build some of the debug versions of the deeper dependencies that are already available in release mode. Using Pybind11 works around the libboost-python issue, as it's a header only lib. Also conda helps with dependencies of dependencies and cross(common) dependencies. The drawback is no x32 support.
Building the Libpack from scratch offers the possibility to use some pre-compiled binaries from official releases. i.e. I downloaded the official PY37 Win installer and there are options to install all the Python debug libs/inculdes. I will try QT5 next but it's important that they share a compatible ssl.
My personal opinion is we should examine conda first. It's a huge task anyway we look at it, but at least the conda trail has been blazed in release mode.
"fight the good fight"