Hi all,
I am the author of a package manager called "mamba". It's very similar to "conda" and works with the same package format (but mamba is written in C++, and using a different solver backend). I think a few of you are aware of conda, since there are FreeCAD builds on conda-forge and as far as I understand, some of the final downloadable builds are derived from those.
Mamba is also cross-platform, cross-language (incl. C/C++) package manager and runs very well on Linux, OS X and Windows. We also have a "libmamba" that is just a library interface to mamba.
The idea that came up was: could "libmamba" be used as "extension manager" for FreeCAD? So that FreeCAD plugins are in fact "conda"-packages that can be installed through libmamba. This would have some nice benefits:
- proper dependency resolving, and allowing inter-dependencies between extensions
- support for native libraries (C/C++/Rust/Go dependencies)
- could inter-op / build on top of conda-forge
- FreeCAD could have it's own "channel" (repository), either as static file server, on Anaconda.org or self-hosted "quetz" instance (https://github.com/mamba-org/quetz)
This would be somewhat new territory for mamba/libmamba, but it would be very interesting for me to see if we could work something out! So far we have used "libmamba" only in the "rhumba" extension to R that brings package manager capabilities based on conda packages to the R language.
Looking forward to a discussion here!
conda / mamba based extension manager
Re: conda / mamba based extension manager
Hi wolfv and welcome to the freecad-forum!
I would like to help with getting mamba included in the conda-based bundles (especially the appimage). The main advantage I see from doing so are smaller bundles which contain only the core of freecad + mamba as dependency solver. This way we do not have to add things like (scipy, sympy, ...)
With mamba it should be possible to install compiled extensions and guarantee compatibility with freecad (examples currently already available from conda-forge are Assembly3 or openglider).
But there are for sure some difficulties which have to be solved. So I simple start with some questions:
1. Can we tell mamba where to install additional packages (best to do this local (user) and specific for the freecad version (0.19, 0.20,...)?
2. Is there a way to avoid updating libraries that are already bundled within the appimage and use these libraries as constraints for additional libraries?
3. Can we use mamba in the github-action to create the bundles (speeding up the process to create the environments)?
I also created an appimage that only includes freecad with dependencies + mamba and libmamba. I guess this is something we can experiment with. https://github.com/FreeCAD/FreeCAD-Bund ... /tag/mamba
Code: Select all
import os
from mamba import mamba_api
mamba_api.Context().conda_prefix = os.environ["PREFIX"]
Re: conda / mamba based extension manager
Hi looo,
thanks for the warm welcome!
1. Yes, installation location is completely customizable. One thing that is usually required is a "root prefix" which contains a "pkgs dir". On conda, that is usually something like `~/miniconda`, and the `~/miniconda/pkgs`. This is used to have a cache of packages, with the final installation being "hard-links" that point to the files in this cache. Since conda / mamba manage virtual environments, this can save disk-space since we can re-use the same package files across environments.
I am just thinking out loud: for the FreeCAD use-case we could have a FreeCAD-version-independent cache in `~/.freecad/plugin_pkgs` with environments unders `~/.freecad/freecad-0.19/plugins/...` as the installation location for the plugins.
It would also be possible to _not_ use this caching mechanism and always copy files into the installation directory. That is just one config variable that would need to be changed.
2. yeah, that is a good & interesting question. We already have a concept of virtual (system) packages like `__glibc`, or `__osx` to constrain on libraries / versions that are not managed by mamba / conda. For this use-case we'd probably want to extend that to freecad and other "fixed" libraries by the freecad installation. We don't have a straight-forward mechanism for that yet, but I think we can just drop a non-existent `freecad-constraints` package into the installation prefix and give it the constraints for other libraries as they are installed. But there might be some additional work to do! I'd be happy to tackle that.
3. Do you mean use mamba in general, or for this specific use-case? We could create pre-solved environments for the plugins using something like `conda-lock` (which can also use mamba & micromamba). If you're referring to the call to `conda create ...` in the current github action, then you could just substitute that with `mamba create` and I'd expect that it would work pretty much the same way as it does currently (sometimes there are slight solver differences).
I need to dive deeper into how the FreeCAD plugins work ... looking forward to this!
thanks for the warm welcome!
1. Yes, installation location is completely customizable. One thing that is usually required is a "root prefix" which contains a "pkgs dir". On conda, that is usually something like `~/miniconda`, and the `~/miniconda/pkgs`. This is used to have a cache of packages, with the final installation being "hard-links" that point to the files in this cache. Since conda / mamba manage virtual environments, this can save disk-space since we can re-use the same package files across environments.
I am just thinking out loud: for the FreeCAD use-case we could have a FreeCAD-version-independent cache in `~/.freecad/plugin_pkgs` with environments unders `~/.freecad/freecad-0.19/plugins/...` as the installation location for the plugins.
It would also be possible to _not_ use this caching mechanism and always copy files into the installation directory. That is just one config variable that would need to be changed.
2. yeah, that is a good & interesting question. We already have a concept of virtual (system) packages like `__glibc`, or `__osx` to constrain on libraries / versions that are not managed by mamba / conda. For this use-case we'd probably want to extend that to freecad and other "fixed" libraries by the freecad installation. We don't have a straight-forward mechanism for that yet, but I think we can just drop a non-existent `freecad-constraints` package into the installation prefix and give it the constraints for other libraries as they are installed. But there might be some additional work to do! I'd be happy to tackle that.
3. Do you mean use mamba in general, or for this specific use-case? We could create pre-solved environments for the plugins using something like `conda-lock` (which can also use mamba & micromamba). If you're referring to the call to `conda create ...` in the current github action, then you could just substitute that with `mamba create` and I'd expect that it would work pretty much the same way as it does currently (sometimes there are slight solver differences).
I need to dive deeper into how the FreeCAD plugins work ... looking forward to this!
Re: conda / mamba based extension manager
wolfv wrote: ↑Fri Apr 30, 2021 6:17 am 2. yeah, that is a good & interesting question. We already have a concept of virtual (system) packages like `__glibc`, or `__osx` to constrain on libraries / versions that are not managed by mamba / conda. For this use-case we'd probably want to extend that to freecad and other "fixed" libraries by the freecad installation. We don't have a straight-forward mechanism for that yet, but I think we can just drop a non-existent `freecad-constraints` package into the installation prefix and give it the constraints for other libraries as they are installed. But there might be some additional work to do! I'd be happy to tackle that.
I'm very interested in this!
"fight the good fight"
Re: conda / mamba based extension manager
This thread should be x-posted with chennes's efforts to standardie the addons with metadata (https://forum.freecadweb.org/viewtopic.php?f=10&t=57448)
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
Re: conda / mamba based extension manager
Thanks for the ping, @Kunda1 -- I've done some work on dependency resolution from within FreeCAD as part of my work developing the package metadata standard, but it's obviously a work-in-progress, and doesn't address any of the other bullet points that this does.
ETA: I should point out that when I say "dependency resolution" in this case, I am referring only to one FreeCAD package depending on another -- not the deeper sort of system-level resolution that this type of manager would enable.
ETA: I should point out that when I say "dependency resolution" in this case, I am referring only to one FreeCAD package depending on another -- not the deeper sort of system-level resolution that this type of manager would enable.
Re: conda / mamba based extension manager
Hi Friends,
I have just discovered this thread. I am very interested in this approach. Dependency management is hard especially with native c++/c/rust/.. dependencies.
I have just discovered this thread. I am very interested in this approach. Dependency management is hard especially with native c++/c/rust/.. dependencies.