Running pip from within FreeCAD
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Be nice to others! Respect the FreeCAD code of conduct!
Re: Running pip from within FreeCAD
Fortunately I have already had this problem just with the Addon Manager, so run my tests in a Virtual Machine that can easily be rolled back to a previous state. It's an important consideration, though!
-
- Veteran
- Posts: 5505
- Joined: Thu Apr 05, 2018 1:53 am
Re: Running pip from within FreeCAD
I think also it will be better to just provide the names of the dependencies needed and perhaps some general instructions for installing them. Then it is on the user if he installs something nefarious. Also, as a developer of addons I don't want these dependencies installed on my computer when I install an addon. The reason is I don't use anything in my addons that is not already included in FreeCAD packaging (for the convenience of users so they do not have to install them). If it gets installed by the addon manager I might use it not knowing other users will not have it on their systems.
Re: Running pip from within FreeCAD
Would it be an option to keep a whitelist of packages in the AddonManager that can be trusted? This would at least avoid to install arbitrary packages. It's then up to the author of a plugin to make a PR to extend this whitelist if needed and a FreeCAD developer can verify that it's a legitimate package.
Another thing is how is supposed to work if installing a package requires admin rights. Or is there a way to tell pip to install packages into the user directory?
Re: Running pip from within FreeCAD
From my point of view (AppIMage user) main concerns are the isolation from the OS.
It could be a relatevely less used case, but I think as AppIMages are very popular that it could be "the way" to mantain and have some "control" over FreeCAD installation.
Main problem that could be addressed are:
I could have some control of the search Path, I use very often:
When creating "programs" with multiple python files, to do complex things.
And it is working, so I think that using the same method to load a "user/modules" directories, like putting an ipotetic directory:
/userdir/site-packages
mimicking Python name and istruct pip to install packages here.
plus adding in using code similar to the above lines, it to sys.path, when initializing FreeCAD.
Some experiment in actual AppImage (FreeCAD 0.20, Libs: 0.20R26720 (Git)) will give:
This show that pip is installed on the actual Appimage and that the "global" pip dir
https://pip.pypa.io/en/stable/topics/configuration/
as the order is:
EDIT: new code and some caveats:
Test are telling that package is installed, but seem that is not listed in installed packages as:
pip list in will reveal
but the import line and print(dir(art)) seems to revel the correct installation and retrieve of the package.
Even exiting AppImage, relaunch them and test for a correct importing of the installed "art" package is revealing that the path is added and package is correctly found.
In meantime I've investigated some more and found this:
https://github.com/pypa/pip/issues/8063 ... -841413834
During my search I found many suggestion about to use a virtualenv, but I don't know how much disk space it will be using and if it could be integrated with actual FreeCAD way (I think that it will be a "no-go" way)
I think that for small packages that not affect too much the installed libraries, (if an AddOn will be too invasive, I think AddOn author has to choose another way to make his changes, like a dedicated fork)
If packages installed have too many dependencies, i don't know, maybe @wmayer has some advice
END EDIT
Hope it helps
Regards
Carlo D.
It could be a relatevely less used case, but I think as AppIMages are very popular that it could be "the way" to mantain and have some "control" over FreeCAD installation.
Main problem that could be addressed are:
- ability to have additional program installed on system
- difficult to have developer editor integration, in other word have a mean to have some automatic introspection over FreeCAD runnning AppImage
I could have some control of the search Path, I use very often:
Code: Select all
MODULE_PATH = os.path.dirname(__file__)
if MODULE_PATH not in sys.path:
if DBG_LOAD is True:
print("no module path")
sys.path.insert(-1, MODULE_PATH)
else:
if DBG_LOAD is True:
print("module path is present")
And it is working, so I think that using the same method to load a "user/modules" directories, like putting an ipotetic directory:
/userdir/site-packages
mimicking Python name and istruct pip to install packages here.
plus adding in using code similar to the above lines, it to sys.path, when initializing FreeCAD.
Some experiment in actual AppImage (FreeCAD 0.20, Libs: 0.20R26720 (Git)) will give:
Code: Select all
import site
site.getsitepackages() # List of global package locations
['/tmp/.mount_FreeCAo9eCC1/usr/lib/python3.9/site-packages']
Code: Select all
import pip
pip.__file__
'/tmp/.mount_FreeCAo9eCC1/usr/lib/python3.9/site-packages/pip/__init__.py'
https://pip.pypa.io/en/stable/topics/configuration/
as the order is:
I've successfully installed in and arbitrary dir a pip package with the code below, it is verbose, but maybe it will be a good starting point.Loading order
When multiple configuration files are found, pip combines them in the following order:
PIP_CONFIG_FILE, if given.
Global
User
Site
EDIT: new code and some caveats:
Code: Select all
"""pip_path.py
This code was written as an sample code
Author: Carlo Dormeletti
Copyright: 2021
Licence: CC BY-NC-ND 4.0 IT
"""
import os
import sys
import subprocess
DBG_LOAD = True
fc_dir = FreeCAD.getHomePath()
# path to python.exe
python_exe = os.path.join(fc_dir, "bin", 'python')
packages_dir = "/home/carlo-arch/test-pip"
MODULE_PATH = packages_dir
if MODULE_PATH not in sys.path:
if DBG_LOAD is True:
print("no module path")
sys.path.insert(-1, MODULE_PATH)
else:
if DBG_LOAD is True:
print("module path is present")
target_dir = "--target={}".format(packages_dir)
#cmds = ["ls", "-al", fc_dir]
#cmds = [python_exe, "-m", "pip", "install", target_dir, "color"]
cmds = [python_exe, "-m", "pip", "list"]
print(subprocess.check_output(cmds).decode('utf-8'))
import art
print(dir(art))
pip list in
Code: Select all
cmds = [python_exe, "-m", "pip", "list"]
but the import line and print(dir(art)) seems to revel the correct installation and retrieve of the package.
Even exiting AppImage, relaunch them and test for a correct importing of the installed "art" package is revealing that the path is added and package is correctly found.
In meantime I've investigated some more and found this:
https://github.com/pypa/pip/issues/8063 ... -841413834
During my search I found many suggestion about to use a virtualenv, but I don't know how much disk space it will be using and if it could be integrated with actual FreeCAD way (I think that it will be a "no-go" way)
I think that for small packages that not affect too much the installed libraries, (if an AddOn will be too invasive, I think AddOn author has to choose another way to make his changes, like a dedicated fork)
If packages installed have too many dependencies, i don't know, maybe @wmayer has some advice
END EDIT
Hope it helps
Regards
Carlo D.
GitHub page: https://github.com/onekk/freecad-doc.
- In deep articles on FreeCAD.
- Learning how to model with scripting.
- Various other stuffs.
Blog: https://okkmkblog.wordpress.com/
- In deep articles on FreeCAD.
- Learning how to model with scripting.
- Various other stuffs.
Blog: https://okkmkblog.wordpress.com/
Re: Running pip from within FreeCAD
This is a good idea: if we decide to proceed with allowing FreeCAD to do these installations, I will definitely implement a list of trusted packages.
With pip installation this should not be an issue, we can just install for the local user. Trying to do an apt/yum/pacman install is different, though. I don't know if that's resolvable. I think this issue is the real "showstopper" -- if we could have just used pip for everything I think it is doable, but with the added complication of trying to figure out whether we should use a system package manager instead, I think it's best to shelve this for the time being.Another thing is how is supposed to work if installing a package requires admin rights. Or is there a way to tell pip to install packages into the user directory?
Re: Running pip from within FreeCAD
A side note pip is not very "integrated" in many distribution, as example on Debian and Arch Linux this is not the recommended way to install packages, and sometimes pip supplied by the distribution is patched do avoid some operation.chennes wrote: ↑Fri Jan 07, 2022 2:15 am With pip installation this should not be an issue, we can just install for the local user. Trying to do an apt/yum/pacman install is different, though. I don't know if that's resolvable. I think this issue is the real "showstopper" -- if we could have just used pip for everything I think it is doable, but with the added complication of trying to figure out whether we should use a system package manager instead, I think it's best to shelve this for the time being.
As in theory with pip you could update pip itself, usually this action is not implemented, or discouraged in other ways, like warning and so on.
But as advised in:
https://packaging.python.org/en/latest/ ... jects/#pip
pipx
https://pypa.github.io/pipx/
could be a way to go with FreeCAD as it isolates (at least in the goal of developers) installed packages from those supplied by the system.
On the package manager side.
You could send using subprocess.call() or similar to install packages using distribution package manager, but, in this case almost every time you will have "administration privileges", or at least you have to use sudo or similar maybe using a special username to have the ability to put some line in /etc/sudoers to permit running package manager without insert a password, or you have to ask the user password and supply to the sudo command, (if possible, I've not done this sort of automation).
But usually involve some user interactions, that depends on the package manager itself, a totally automated installation is doable, but usually is a more complex task, at least in Debian and Arch Linux, sometimes user input is asked when installing packages, so it may lead to some random problems.
With AppImages you have to find another way, as AppIMages are separated from system.
Or maybe in AppImages there is some method that could be automated to patch an AppImage?
I know that there is a way to "decompress" the AppImage, add files and recompress, but I don't know if it is feasible on a "running" AppImage, even if as the AppImage is running in a \tmp directory, the original AppImage file should not be affected, but I don't know if there is mechanism when you launch the AppImage that do some "lock" on the original file.
Maybe for an advanced user having an version of FreeCAD as Conda Packages would be a viable alternative, but I don't know what this choice will involve in term of complexity for maintainers or disk space use.
Regards
Carlo D.
GitHub page: https://github.com/onekk/freecad-doc.
- In deep articles on FreeCAD.
- Learning how to model with scripting.
- Various other stuffs.
Blog: https://okkmkblog.wordpress.com/
- In deep articles on FreeCAD.
- Learning how to model with scripting.
- Various other stuffs.
Blog: https://okkmkblog.wordpress.com/
Re: Running pip from within FreeCAD
I personally think it is very important to allow add-ons to have dependencies and automatically install them. I recognize the problems, so here a possible workflow that may resolve some concerns (partially what I do in my addon):
1. Read in the requirement.txt from the add-on and check what is not installed (incl. version requirements etc)
2. Show the user a list of missing packages. With that he can install it from whatever package manager his system has. Add a "Check again" button for him to see whats missing
3. Give the user the option to install via pip. This installation could be done into a FreeCAD specifiy "Vendor" folder with pips "--target" option. Than this Vendor folder is added to FreeCAD path. This way we would make sure to not mess up the systems python installation, and any bad version errors etc. could simply be resolved by deleting "Vendor" again. Maybe even use "virtualenv" or "pipenv" for that.
4. Rechecking dependencies whenever the Add-on is updated is done the same way, show the list and give the options if needed
1. Read in the requirement.txt from the add-on and check what is not installed (incl. version requirements etc)
2. Show the user a list of missing packages. With that he can install it from whatever package manager his system has. Add a "Check again" button for him to see whats missing
3. Give the user the option to install via pip. This installation could be done into a FreeCAD specifiy "Vendor" folder with pips "--target" option. Than this Vendor folder is added to FreeCAD path. This way we would make sure to not mess up the systems python installation, and any bad version errors etc. could simply be resolved by deleting "Vendor" again. Maybe even use "virtualenv" or "pipenv" for that.
4. Rechecking dependencies whenever the Add-on is updated is done the same way, show the list and give the options if needed
Re: Running pip from within FreeCAD
Of course, another metadata file -- I am reading in metadata.txt now, but it seems very rudimentary, and is only used by a few Addons. Are there FreeCAD Addons that specify their requirements in a requirements.txt file?
This is what I do now: just show the list of missing required and optional python packages, and leave it up to the user to install them.2. Show the user a list of missing packages. With that he can install it from whatever package manager his system has. Add a "Check again" button for him to see what's missing
I was not aware of this option to pip!! This seems to me to be the perfect path forward. Do you think we should have a single Vendor directory for all of FreeCAD, or separate ones for each Addon?3. Give the user the option to install via pip. This installation could be done into a FreeCAD specifiy "Vendor" folder with pips "--target" option. Than this Vendor folder is added to FreeCAD path.
Right now the check is only being done on installation, not on update, but you are right that it should happen in both places.4. Rechecking dependencies whenever the Add-on is updated is done the same way, show the list and give the options if needed
Re: Running pip from within FreeCAD
actually I don't care where the dependencies are specified, requirement.txt is just the default python way, hence my example was based on that. Any other way is perfectly fine One thing to consider is the syntax for package version requirements (e.g. minimum version of a certain package needed), maybe that should be copied from the requirement.txt syntax, as this is something like a python default? (https://packaging.python.org/en/latest/ ... -Specifier)
I think both options come down to the same in the end, as there is nothing like a "pythonpath for a addon". Hence multiple vendor folders would end up all in the python path, hence they could be a single folder in the beginning.I was not aware of this option to pip!! This seems to me to be the perfect path forward. Do you think we should have a single Vendor directory for all of FreeCAD, or separate ones for each Addon?
Re: Running pip from within FreeCAD
It maybe not be so clear but in:
https://forum.freecadweb.org/viewtopic. ... 13#p559413
There is the entire mechanism
Code: Select all
import os
import subprocess
fc_dir = FreeCAD.getHomePath()
# path to python.exe
python_exe = os.path.join(fc_dir, "bin", 'python')
packages_dir = "/home/carlo-arch/test-pip"
target_dir = "--target={}".format(packages_dir)
cmds = [python_exe, "-m", "pip", "install", target_dir, "color"]
subprocess.call(cmds)
This is to extract Path of the python ececutable installed in AppImage
Code: Select all
fc_dir = FreeCAD.getHomePath()
# path to python.exe
python_exe = os.path.join(fc_dir, "bin", 'python')
Code: Select all
packages_dir = "/home/carlo-arch/test-pip"
target_dir = "--target={}".format(packages_dir)
cmds = [python_exe, "-m", "pip", "install", target_dir, "color"]
Code: Select all
MODULE_PATH = packages_dir
if MODULE_PATH not in sys.path:
sys.path.insert(-1, MODULE_PATH)
But there is also pipx that could manage things, see:
https://forum.freecadweb.org/viewtopic. ... 10#p559610
Regards
Carlo D.
GitHub page: https://github.com/onekk/freecad-doc.
- In deep articles on FreeCAD.
- Learning how to model with scripting.
- Various other stuffs.
Blog: https://okkmkblog.wordpress.com/
- In deep articles on FreeCAD.
- Learning how to model with scripting.
- Various other stuffs.
Blog: https://okkmkblog.wordpress.com/