Running pip from within FreeCAD

Need help, or want to share a macro? Post here!
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
User avatar
chennes
Veteran
Posts: 3876
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: Running pip from within FreeCAD

Post by chennes »

Fortunately I have already had this problem just with the Addon Manager, so run my tests in a Virtual Machine that can easily be rolled back to a previous state. It's an important consideration, though!
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
TheMarkster
Veteran
Posts: 5505
Joined: Thu Apr 05, 2018 1:53 am

Re: Running pip from within FreeCAD

Post by TheMarkster »

I think also it will be better to just provide the names of the dependencies needed and perhaps some general instructions for installing them. Then it is on the user if he installs something nefarious. Also, as a developer of addons I don't want these dependencies installed on my computer when I install an addon. The reason is I don't use anything in my addons that is not already included in FreeCAD packaging (for the convenience of users so they do not have to install them). If it gets installed by the addon manager I might use it not knowing other users will not have it on their systems.
wmayer
Founder
Posts: 20241
Joined: Thu Feb 19, 2009 10:32 am
Contact:

Re: Running pip from within FreeCAD

Post by wmayer »

chennes wrote: Wed Jan 05, 2022 5:53 pm No: asking pip to install a package implies installing all of that package's dependencies as well.
Would it be an option to keep a whitelist of packages in the AddonManager that can be trusted? This would at least avoid to install arbitrary packages. It's then up to the author of a plugin to make a PR to extend this whitelist if needed and a FreeCAD developer can verify that it's a legitimate package.
chennes wrote: Wed Jan 05, 2022 5:53 pm This is a serious concern, and probably undetectable by us, though if there is a way I am all ears! Maybe try to do an apt/yum/brew/whatever search before offering pip?
Another thing is how is supposed to work if installing a package requires admin rights. Or is there a way to tell pip to install packages into the user directory?
User avatar
onekk
Veteran
Posts: 6144
Joined: Sat Jan 17, 2015 7:48 am
Contact:

Re: Running pip from within FreeCAD

Post by onekk »

From my point of view (AppIMage user) main concerns are the isolation from the OS.

It could be a relatevely less used case, but I think as AppIMages are very popular that it could be "the way" to mantain and have some "control" over FreeCAD installation.

Main problem that could be addressed are:
  • ability to have additional program installed on system
  • difficult to have developer editor integration, in other word have a mean to have some automatic introspection over FreeCAD runnning AppImage
Actual state is mixed, as I could even create different configuration for different running instance of FreeCAD,

I could have some control of the search Path, I use very often:

Code: Select all

MODULE_PATH = os.path.dirname(__file__)

if MODULE_PATH not in sys.path:
    if DBG_LOAD is True:
        print("no module path")
    sys.path.insert(-1, MODULE_PATH)
else:
    if DBG_LOAD is True:
        print("module path is present")
When creating "programs" with multiple python files, to do complex things.

And it is working, so I think that using the same method to load a "user/modules" directories, like putting an ipotetic directory:

/userdir/site-packages

mimicking Python name and istruct pip to install packages here.

plus adding in using code similar to the above lines, it to sys.path, when initializing FreeCAD.

Some experiment in actual AppImage (FreeCAD 0.20, Libs: 0.20R26720 (Git)) will give:

Code: Select all

import site
site.getsitepackages() # List of global package locations

['/tmp/.mount_FreeCAo9eCC1/usr/lib/python3.9/site-packages']

Code: Select all

import pip
pip.__file__

'/tmp/.mount_FreeCAo9eCC1/usr/lib/python3.9/site-packages/pip/__init__.py'
This show that pip is installed on the actual Appimage and that the "global" pip dir

https://pip.pypa.io/en/stable/topics/configuration/

as the order is:
Loading order

When multiple configuration files are found, pip combines them in the following order:

PIP_CONFIG_FILE, if given.

Global

User

Site
I've successfully installed in and arbitrary dir a pip package with the code below, it is verbose, but maybe it will be a good starting point.

EDIT: new code and some caveats:

Code: Select all

"""pip_path.py

   This code was written as an sample code

   Author: Carlo Dormeletti
   Copyright: 2021
   Licence: CC BY-NC-ND 4.0 IT
"""
import os
import sys
import subprocess

DBG_LOAD = True

fc_dir =  FreeCAD.getHomePath()

# path to python.exe
python_exe = os.path.join(fc_dir, "bin", 'python')
 
packages_dir = "/home/carlo-arch/test-pip"

MODULE_PATH = packages_dir

if MODULE_PATH not in sys.path:
    if DBG_LOAD is True:
        print("no module path")
    sys.path.insert(-1, MODULE_PATH)
else:
    if DBG_LOAD is True:
        print("module path is present")


target_dir = "--target={}".format(packages_dir)

#cmds = ["ls", "-al", fc_dir]

#cmds = [python_exe, "-m", "pip", "install", target_dir,  "color"]

cmds = [python_exe, "-m", "pip", "list"]

print(subprocess.check_output(cmds).decode('utf-8'))

import art

print(dir(art))
Test are telling that package is installed, but seem that is not listed in installed packages as:

pip list in

Code: Select all

cmds = [python_exe, "-m", "pip", "list"]
will reveal

but the import line and print(dir(art)) seems to revel the correct installation and retrieve of the package.

Even exiting AppImage, relaunch them and test for a correct importing of the installed "art" package is revealing that the path is added and package is correctly found.

In meantime I've investigated some more and found this:

https://github.com/pypa/pip/issues/8063 ... -841413834

During my search I found many suggestion about to use a virtualenv, but I don't know how much disk space it will be using and if it could be integrated with actual FreeCAD way (I think that it will be a "no-go" way)

I think that for small packages that not affect too much the installed libraries, (if an AddOn will be too invasive, I think AddOn author has to choose another way to make his changes, like a dedicated fork)

If packages installed have too many dependencies, i don't know, maybe @wmayer has some advice

END EDIT

Hope it helps

Regards

Carlo D.
GitHub page: https://github.com/onekk/freecad-doc.
- In deep articles on FreeCAD.
- Learning how to model with scripting.
- Various other stuffs.

Blog: https://okkmkblog.wordpress.com/
User avatar
chennes
Veteran
Posts: 3876
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: Running pip from within FreeCAD

Post by chennes »

wmayer wrote: Thu Jan 06, 2022 11:49 am Would it be an option to keep a whitelist of packages in the AddonManager that can be trusted?
This is a good idea: if we decide to proceed with allowing FreeCAD to do these installations, I will definitely implement a list of trusted packages.
Another thing is how is supposed to work if installing a package requires admin rights. Or is there a way to tell pip to install packages into the user directory?
With pip installation this should not be an issue, we can just install for the local user. Trying to do an apt/yum/pacman install is different, though. I don't know if that's resolvable. I think this issue is the real "showstopper" -- if we could have just used pip for everything I think it is doable, but with the added complication of trying to figure out whether we should use a system package manager instead, I think it's best to shelve this for the time being.
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
User avatar
onekk
Veteran
Posts: 6144
Joined: Sat Jan 17, 2015 7:48 am
Contact:

Re: Running pip from within FreeCAD

Post by onekk »

chennes wrote: Fri Jan 07, 2022 2:15 am With pip installation this should not be an issue, we can just install for the local user. Trying to do an apt/yum/pacman install is different, though. I don't know if that's resolvable. I think this issue is the real "showstopper" -- if we could have just used pip for everything I think it is doable, but with the added complication of trying to figure out whether we should use a system package manager instead, I think it's best to shelve this for the time being.
A side note pip is not very "integrated" in many distribution, as example on Debian and Arch Linux this is not the recommended way to install packages, and sometimes pip supplied by the distribution is patched do avoid some operation.

As in theory with pip you could update pip itself, usually this action is not implemented, or discouraged in other ways, like warning and so on.

But as advised in:

https://packaging.python.org/en/latest/ ... jects/#pip

pipx

https://pypa.github.io/pipx/

could be a way to go with FreeCAD as it isolates (at least in the goal of developers) installed packages from those supplied by the system.

On the package manager side.

You could send using subprocess.call() or similar to install packages using distribution package manager, but, in this case almost every time you will have "administration privileges", or at least you have to use sudo or similar maybe using a special username to have the ability to put some line in /etc/sudoers to permit running package manager without insert a password, or you have to ask the user password and supply to the sudo command, (if possible, I've not done this sort of automation).

But usually involve some user interactions, that depends on the package manager itself, a totally automated installation is doable, but usually is a more complex task, at least in Debian and Arch Linux, sometimes user input is asked when installing packages, so it may lead to some random problems.


With AppImages you have to find another way, as AppIMages are separated from system.

Or maybe in AppImages there is some method that could be automated to patch an AppImage?

I know that there is a way to "decompress" the AppImage, add files and recompress, but I don't know if it is feasible on a "running" AppImage, even if as the AppImage is running in a \tmp directory, the original AppImage file should not be affected, but I don't know if there is mechanism when you launch the AppImage that do some "lock" on the original file.

Maybe for an advanced user having an version of FreeCAD as Conda Packages would be a viable alternative, but I don't know what this choice will involve in term of complexity for maintainers or disk space use.


Regards

Carlo D.
GitHub page: https://github.com/onekk/freecad-doc.
- In deep articles on FreeCAD.
- Learning how to model with scripting.
- Various other stuffs.

Blog: https://okkmkblog.wordpress.com/
ickby
Veteran
Posts: 3116
Joined: Wed Oct 05, 2011 7:36 am

Re: Running pip from within FreeCAD

Post by ickby »

I personally think it is very important to allow add-ons to have dependencies and automatically install them. I recognize the problems, so here a possible workflow that may resolve some concerns (partially what I do in my addon):

1. Read in the requirement.txt from the add-on and check what is not installed (incl. version requirements etc)
2. Show the user a list of missing packages. With that he can install it from whatever package manager his system has. Add a "Check again" button for him to see whats missing
3. Give the user the option to install via pip. This installation could be done into a FreeCAD specifiy "Vendor" folder with pips "--target" option. Than this Vendor folder is added to FreeCAD path. This way we would make sure to not mess up the systems python installation, and any bad version errors etc. could simply be resolved by deleting "Vendor" again. Maybe even use "virtualenv" or "pipenv" for that.
4. Rechecking dependencies whenever the Add-on is updated is done the same way, show the list and give the options if needed
User avatar
chennes
Veteran
Posts: 3876
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: Running pip from within FreeCAD

Post by chennes »

ickby wrote: Fri Jan 07, 2022 2:16 pm 1. Read in the requirement.txt from the add-on and check what is not installed (incl. version requirements etc)
Of course, another metadata file :) -- I am reading in metadata.txt now, but it seems very rudimentary, and is only used by a few Addons. Are there FreeCAD Addons that specify their requirements in a requirements.txt file?

2. Show the user a list of missing packages. With that he can install it from whatever package manager his system has. Add a "Check again" button for him to see what's missing
This is what I do now: just show the list of missing required and optional python packages, and leave it up to the user to install them.

3. Give the user the option to install via pip. This installation could be done into a FreeCAD specifiy "Vendor" folder with pips "--target" option. Than this Vendor folder is added to FreeCAD path.
I was not aware of this option to pip!! This seems to me to be the perfect path forward. Do you think we should have a single Vendor directory for all of FreeCAD, or separate ones for each Addon?
4. Rechecking dependencies whenever the Add-on is updated is done the same way, show the list and give the options if needed
Right now the check is only being done on installation, not on update, but you are right that it should happen in both places.
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
ickby
Veteran
Posts: 3116
Joined: Wed Oct 05, 2011 7:36 am

Re: Running pip from within FreeCAD

Post by ickby »

chennes wrote: Fri Jan 07, 2022 2:27 pm Of course, another metadata file :)
:lol: actually I don't care where the dependencies are specified, requirement.txt is just the default python way, hence my example was based on that. Any other way is perfectly fine :) One thing to consider is the syntax for package version requirements (e.g. minimum version of a certain package needed), maybe that should be copied from the requirement.txt syntax, as this is something like a python default? (https://packaging.python.org/en/latest/ ... -Specifier)
I was not aware of this option to pip!! This seems to me to be the perfect path forward. Do you think we should have a single Vendor directory for all of FreeCAD, or separate ones for each Addon?
I think both options come down to the same in the end, as there is nothing like a "pythonpath for a addon". Hence multiple vendor folders would end up all in the python path, hence they could be a single folder in the beginning.
User avatar
onekk
Veteran
Posts: 6144
Joined: Sat Jan 17, 2015 7:48 am
Contact:

Re: Running pip from within FreeCAD

Post by onekk »

chennes wrote: Fri Jan 07, 2022 2:27 pm I was not aware of this option to pip!! This seems to me to be the perfect path forward. Do you think we should have a single Vendor directory for all of FreeCAD, or separate ones for each Addon?
It maybe not be so clear but in:

https://forum.freecadweb.org/viewtopic. ... 13#p559413

There is the entire mechanism

Code: Select all

import os
import subprocess

fc_dir =  FreeCAD.getHomePath()
# path to python.exe
python_exe = os.path.join(fc_dir, "bin", 'python')
packages_dir = "/home/carlo-arch/test-pip"
target_dir = "--target={}".format(packages_dir)
cmds = [python_exe, "-m", "pip", "install", target_dir,  "color"]

subprocess.call(cmds)

It is slightly verbose to show the relevant points about dirs and "derived" dirs in the AppImage case.

This is to extract Path of the python ececutable installed in AppImage

Code: Select all

fc_dir =  FreeCAD.getHomePath()
# path to python.exe
python_exe = os.path.join(fc_dir, "bin", 'python')
This is to set the directory on which you want packages will be put.

Code: Select all

packages_dir = "/home/carlo-arch/test-pip"
target_dir = "--target={}".format(packages_dir)
cmds = [python_exe, "-m", "pip", "install", target_dir,  "color"]
This is to add the directoy to FC path to permit library use:

Code: Select all

MODULE_PATH = packages_dir

if MODULE_PATH not in sys.path:
    sys.path.insert(-1, MODULE_PATH)
The only "delicate thing" is that in this manner pip list will not show installed packages, as I've not found the way to add the "user directory" to pip.

But there is also pipx that could manage things, see:

https://forum.freecadweb.org/viewtopic. ... 10#p559610

Regards

Carlo D.
GitHub page: https://github.com/onekk/freecad-doc.
- In deep articles on FreeCAD.
- Learning how to model with scripting.
- Various other stuffs.

Blog: https://okkmkblog.wordpress.com/
Post Reply