pip-integration in addon-manager

Need help, or want to share a macro? Post here!
looo
Posts: 2744
Joined: Mon Nov 11, 2013 5:29 pm

Re: pip-integration in addon-manager

Postby looo » Thu Aug 30, 2018 1:01 pm

After some experiments I think this should work:

1. create a constraint.txt file:

Code: Select all

pip freeze >constraint.txt
2. install a package without updating (eg numpy in this case, assuming numpy was in the environment from which the constraint.txt was created)

Code: Select all

pip install template-extension  -c <path_to_constraint.txt> --install-option="--install-purelib=<package-install-dir>"
So for FreeCAD-bundles this should work. But I am still not sure if this is also useful for non-bundles freecad. My workflow for conda normally looks like this:

1. install a package with pip without installing dependencies:

Code: Select all

pip install package_a --no-deps
(assuming the library is not available from conda)

2. see what dependencies are missing and try to install them with conda

3. if not available from conda, install the dependency with 1.

So I don't know how much sense the installation of packages to %appdatadir%/Mod makes for freecad installed into an environment. So maybe keeping the pip-tools in freecad as an extern effort only available for bundles is the way to proceed? what do you think?
looo
Posts: 2744
Joined: Mon Nov 11, 2013 5:29 pm

Re: pip-integration in addon-manager

Postby looo » Thu Aug 30, 2018 4:07 pm

Ok, I tried this with the py3 freecad-bundle provided by @sgrogan.

1. create a constraint file:

Code: Select all

import subprocess as sub
import os
import FreeCAD

constraints_file = os.path.join(FreeCAD.getHomePath(), "constraints.txt")
path_to_pip = os.path.join("Scripts", "pip")
proc = sub.Popen(path_to_pip + " freeze", stdout=sub.PIPE, stderr=sub.PIPE)
out, err = proc.communicate()
FreeCAD.Console.PrintError(err.decode("utf8"))
with open(constraints_file, "wb") as fp:
    fp.write(text)

2. try to install the template extension into %appdata%\\Mod

Code: Select all

import os
import subprocess as sub
import FreeCAD

constraints_file = os.path.join(FreeCAD.getHomePath(), "constraints.txt")
path_to_pip = os.path.join("Scripts", "pip")
install_dir = os.path.join(os.environ["appdata"] ,"FreeCAD", "Mod")
install_options=""
install_options += "-c {} ".format(constraints_file)
install_options += '--install-option="--install-purelib={}" '.format(install_dir)

proc = sub.Popen(path_to_pip +" install template-extension " + install_options, stderr=sub.PIPE, stdout=sub.PIPE)
out, err = proc.communicate()
FreeCAD.Console.PrintError(err.decode("utf8"))
FreeCAD.Console.PrintMessage(out.decode("utf8"))
3. restart freecad and try to import the installed library:

Code: Select all

>>> import template_extension
>>> from freecad import template_extension
Traceback (most recent call last):
  File "<input>", line 1, in <module>
ImportError: cannot import name 'template_extension'
>>> 
so it seems there is something wrong with the sys.path of the bundled FreeCAD. Installing to %%/FreeCAD/Mod is not the best as this mixes old-style and new-style and allows to import old-style modules with new-style import syntax and the other way and confuses everone. I guess we have to think about another strategy to handle this.

edit1.:
Also I tried to install scipy but somehow adding the "--install-option=" option disables the usage of wheels and pip tries to compile scipy which (for sure) fails as there are no build-tools available in the environment...

So still some things to figure out, but I hope there are some workarounds to get this done.

edit2.:
The python way to install for multiple environments is to use the --user flag. In my case this leads to this:

Code: Select all

>>> from freecad import template_extension
>>> template_extension.__file__
'C:\\Users\\katja\\AppData\\Roaming\\Python\\Python36\\site-packages\\freecad\\template_extension\\__init__.py'
>>> 
This is good as we seperate things for different python versions but not for different freecad-versions (if the freecad-versions depend on the same python version...) So I guess this should be prefered over installation to %appdata%/FreeCAD/Mod. Also it doesn't mix old-style modules and new-style modules.

edit3:
@sgrogan: is it a problem for the update workflow if packages are installed in the portable directory?

edit4:
scipy installed without any problems using the --user flag and the given constraints from the bundle.
User avatar
sgrogan
Posts: 5252
Joined: Wed Oct 22, 2014 5:02 pm

Re: pip-integration in addon-manager

Postby sgrogan » Thu Aug 30, 2018 7:44 pm

looo wrote:
Thu Aug 30, 2018 4:07 pm
@sgrogan: is it a problem for the update workflow if packages are installed in the portable directory?
I will need some time to digest all this work.
%appdata%/FreeCAD/Mod is not sacred, I've just been able to use this without having to mess with FreeCAD's sys.path.

On Windows the issues are:
With FreeCAD installed with the installer, the user needs admin(sudo) rights to write to FreeCAD's site-packages directory.
With a portable build, there is no problem writing to the site-packages directory, but these builds last a few days are deleted when a new build is downloaded. These extra python packages need to be either re-added to the new portable build, or copied over from the previous build.

I wonder from your edit2, does FreeCAD find this without messing with FreeCAD's sys.path and would a system installed PY3.
also find this? Right now this doesn't matter because the PY3's are compatible.
My concern would be if the above is true, FreeCAD's PY2.7 and PYPI's PY2.7 would be incompatible. This is why on Win FreeCAD goes to great lengths to ignore any system Python environment variables.

I would be perfectly happy if this support was only available with a PY3/VC14 compatible pair and disabled otherwise.
If I can ever strip a Libpack out of the conda environment, I want to try building FreeCAD linked against system Python like on linux, that would be the holy grail for me.

Anyway, thanks for the work, and I will do some testing.
looo
Posts: 2744
Joined: Mon Nov 11, 2013 5:29 pm

Re: pip-integration in addon-manager

Postby looo » Thu Aug 30, 2018 9:22 pm

sgrogan wrote:On Windows the issues are:
With FreeCAD installed with the installer, the user needs admin(sudo) rights to write to FreeCAD's site-packages directory.
With a portable build, there is no problem writing to the site-packages directory, but these builds last a few days are deleted when a new build is downloaded. These extra python packages need to be either re-added to the new portable build, or copied over from the previous build.
Ok, so this is the same for all portable builds (appimages, and your approach). So the best option is to define a freecad-version-specific custom directory as default install dir. This will also give the option to use different 3rd-party directories to seperate different (incompatible) dependency-trees + sharing the same directories from different freecad versions. The question is if it's possible to have different configs for different versions of freecad. So one freecad.xy uses dirxy as user-site-package and freecad.zz uses dirzz as user-site-package.
I wonder from your edit2, does FreeCAD find this without messing with FreeCAD's sys.path and would a system installed PY3.
also find this? Right now this doesn't matter because the PY3's are compatible.
My concern would be if the above is true, FreeCAD's PY2.7 and PYPI's PY2.7 would be incompatible. This is why on Win FreeCAD goes to great lengths to ignore any system Python environment variables.
yes as far as I know FreeCAD simple uses the path from python and extent it with freecad-related directories containing modules. So the std-paths of python are not changed.
And yes the python2.7 incompatibility between freecad and python-wheels from pypi is (in my mind) a no go for future versions of FreeCAD. Therefor using similar toolchains as conda and pypi uses is the way to proceed for packaging to get the best from the python world.
sgrogan wrote:I would be perfectly happy if this support was only available with a PY3/VC14 compatible pair and disabled otherwise.
If I can ever strip a Libpack out of the conda environment, I want to try building FreeCAD linked against system Python like on linux, that would be the holy grail for me.
what is the definition of system python on windows? I guess the most used python on windows is the one from anaconda. So with conda you already have the option to get something quite similar to python on linux. Or does windows ship it's own python?

reminder: the way to alter the main location of packages installed with pip, setting the environment-variable "PIP_TARGET". I hope there is something similar for the --user option.
User avatar
sgrogan
Posts: 5252
Joined: Wed Oct 22, 2014 5:02 pm

Re: pip-integration in addon-manager

Postby sgrogan » Thu Aug 30, 2018 10:00 pm

looo wrote:
Thu Aug 30, 2018 9:22 pm
yes as far as I know FreeCAD simple uses the path from python and extent it with freecad-related directories containing modules. So the std-paths of python are not changed.
No, FreeCAD ignores any other python on the system. If it doesn't it's a problem. This is why we can't use PYTHONPATH, PYHONHOME, etc.
looo wrote:
Thu Aug 30, 2018 9:22 pm
what is the definition of system python on windows?
On Win you can install Python stand-alone https://www.python.org/downloads/windows/
It's installed to C:\ProgramFiles\Python.xx (or the like I don't have python installed on my system at the moment, I will test). This installation writes some stuff to the registry, I think, and uses the environment variables.
So because Win has no package manager/dependency solver most/all Win programs ship their own version of Python. Last time I checked I had 9 versions of Python on my system, ranging from 2.6 to 3.6. LibreOffice comes bundled with Python, Blender comes bundled with Python, FreeCAD comes bundled with Python.

All this being said, maybe we should just "force" a conda based approach? It does everything we want it to do. The users are resistant to change. We can offer an installer with limited flexibility, for those that prefer this, or a conda based approach where we use the conda based tools instead of the hacks I'm using now (to make it portable)?
looo
Posts: 2744
Joined: Mon Nov 11, 2013 5:29 pm

Re: pip-integration in addon-manager

Postby looo » Fri Aug 31, 2018 1:11 pm

No, FreeCAD ignores any other python on the system. If it doesn't it's a problem. This is why we can't use PYTHONPATH, PYHONHOME, etc.
I meant the python which is shipped with FreeCAD...
sgrogan wrote:All this being said, maybe we should just "force" a conda based approach? It does everything we want it to do. The users are resistant to change. We can offer an installer with limited flexibility, for those that prefer this, or a conda based approach where we use the conda based tools instead of the hacks I'm using now (to make it portable)?
Yes and no. The problem is still that conda is to difficult for someone who only want to use FreeCAD. In the long run people from conda will maybe come up with a solution to create a portable build, but in the meantime your approach is maybe the best solution...

Regarding the pip integration: I found a way which works for a pure conda environment and also for the portable build. I don't care much about the directory where all the user-packages are installed. Choosing it to be the default user-directory is for sure the simplest way. Maybe we can also add an option to select the user-pip-dir, but I guess this is not first priority.

So here some testing with the current pip-wrapper (I have also tested on linux with conda, but this is from the protable build on windows):

1. adding the scripts directory to the path (I guess this should be done by default (@sgrogan)

Code: Select all

>>> import os
>>> os.environ["PATH"] += "C:\\Users\\katja\\Downloads\\FreeCAD_0.18.14495_Conda_Py3QT5-WinVS2016_x64\\bin\\Scripts;"
1. import the pip-wrappers (extension module for freecad) and list all system packages. These packages are added to a temporary constraint file which is used during installation to not update these versions (stability).

Code: Select all

>>> from freecad.pip.app import pip
>>> pip.list_system()
[['certifi', '2018.8.13'], ['cycler', '0.10.0'], ['future', '0.16.0'], ['gitdb2', '2.0.4'], ['GitPython', '2.1.11'], ['kiwisolver', '1.0.1'], ['matplotlib', '2.2.2'], ['mkl-fft', '1.0.5'], ['mkl-random', '1.0.1'], ['numpy', '1.14.3'], ['pip', '18.0'], ['Pivy', '0.6.4'], ['pybind11', '2.3.dev0'], ['pyparsing', '2.2.0'], ['python-dateutil', '2.7.3'], ['pytz', '2018.5'], ['setuptools', '40.0.0'], ['six', '1.11.0'], ['smmap2', '2.0.4'], ['tornado', '5.1'], ['wheel', '0.31.1'], ['wincertstore', '0.2']]
2. list all editable packages (packages which are installed in "development mode", similar to symlinking)

Code: Select all

>>> pip.list_editable()
[]
3. list all user packages (packages installed into the --user directory)

Code: Select all

>>> pip.list_user()
[]
4. try to install scipy

Code: Select all

>>> pip.install("scipy")
Collecting scipy
  Using cached https://files.pythonhosted.org/packages/62/e2/364f0bcc641aeff79d743c732769d5dc31a1e78c27699229431412c4b425/scipy-1.1.0-cp36-none-win_amd64.whl
Requirement already satisfied: numpy==1.14.3 in c:\users\katja\downloads\freecad_0.18.14495_conda_py3qt5-winvs2016_x64\bin\lib\site-packages (from -c C:\Users\katja\AppData\Local\Temp\constraintsr9ygaxnp.txt (line 10)) (1.14.3)
Installing collected packages: scipy
Successfully installed scipy-1.1.0
5. check if it is installed in user directory

Code: Select all

>>> pip.list_user()
[['scipy', '1.1.0']]
6. try to install pandas and test it

Code: Select all

>>> pip.install("pandas")
Collecting pandas
  Using cached https://files.pythonhosted.org/packages/0e/67/def5bfaf4d3324fdb89048889ec523c0903c5efab1a64c8dbe0ac8eec13c/pandas-0.23.4-cp36-cp36m-win_amd64.whl
Requirement already satisfied: python-dateutil==2.7.3 in c:\users\katja\downloads\freecad_0.18.14495_conda_py3qt5-winvs2016_x64\bin\lib\site-packages (from -c C:\Users\katja\AppData\Local\Temp\constraintsr9ygaxnp.txt (line 15)) (2.7.3)
Requirement already satisfied: pytz==2018.5 in c:\users\katja\downloads\freecad_0.18.14495_conda_py3qt5-winvs2016_x64\bin\lib\site-packages (from -c C:\Users\katja\AppData\Local\Temp\constraintsr9ygaxnp.txt (line 16)) (2018.5)
Requirement already satisfied: numpy==1.14.3 in c:\users\katja\downloads\freecad_0.18.14495_conda_py3qt5-winvs2016_x64\bin\lib\site-packages (from -c C:\Users\katja\AppData\Local\Temp\constraintsr9ygaxnp.txt (line 10)) (1.14.3)
Requirement already satisfied: six==1.11.0 in c:\users\katja\downloads\freecad_0.18.14495_conda_py3qt5-winvs2016_x64\bin\lib\site-packages (from -c C:\Users\katja\AppData\Local\Temp\constraintsr9ygaxnp.txt (line 18)) (1.11.0)
Installing collected packages: pandas
Successfully installed pandas-0.23.4

>>> import pandas as pd
>>> pd.DataFrame(pip.list_user(), columns=["name", "version"])
     name version
0  pandas  0.23.4
1   scipy   1.1.0

>>> pd.DataFrame(pip.list(), columns=["name", "version"])
               name    version
0           certifi  2018.8.13
1            cycler     0.10.0
2            future     0.16.0
3            gitdb2      2.0.4
4         GitPython     2.1.11
5        kiwisolver      1.0.1
6        matplotlib      2.2.2
7           mkl-fft      1.0.5
8        mkl-random      1.0.1
9             numpy     1.14.3
10           pandas     0.23.4
11              pip       18.0
12             Pivy      0.6.4
13         pybind11   2.3.dev0
14        pyparsing      2.2.0
15  python-dateutil      2.7.3
16             pytz     2018.5
17            scipy      1.1.0
18       setuptools     40.0.0
19              six     1.11.0
20           smmap2      2.0.4
21          tornado        5.1
22            wheel     0.31.1
23     wincertstore        0.2

7. uninstall a user package

Code: Select all

>>> pip.uninstall("scipy")
Uninstalling scipy-1.1.0:
  Successfully uninstalled scipy-1.1.0
8. try to uninstall a system package (numpy) -> I guess this should throw an exception, currently it prints a error message

Code: Select all

>>> pip.uninstall("numpy")
pkg is not a user-package
9. install package in development mode:

Code: Select all

>>> pip.install_develop("C:\\Users\\katja\\lo_projects\\OpenGlider")
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\katja\AppData\Roaming\Python\Python36\site-packages\freecad\pip\app.py", line 38, in install_develop
    print_msg(process("pip", "install", "-e", fp, "--user", self._c_option()))
  File "C:\Users\katja\AppData\Roaming\Python\Python36\site-packages\freecad\pip\app.py", line 13, in process
    raise RuntimeError(err.decode("utf8"))
RuntimeError:   The scripts dxfaudit.exe and dxfpp.exe are installed in 'C:\Users\katja\AppData\Roaming\Python\Python36\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

 >>> pip.list_editable()
[['OpenGlider', '0.1']]
again we see the problem that bin/Scripts should be in path...
this operation also installed some dependencies:

Code: Select all

>>> pip.list_user()
[['ezdxf', '0.8.8'], ['ezodf', '0.3.2'], ['lml', '0.0.4'], ['odfpy', '1.3.5'], ['OpenGlider', '0.1'], ['pandas', '0.23.4'], ['pyexcel', '0.5.9.1'], ['pyexcel-io', '0.5.9.1'], ['pyexcel-ods', '0.5.3'], ['svgwrite', '1.1.12'], ['texttable', '1.4.0']]
so to me using pip with system packages constrained looks quite promising as this makes sense for virtual environments, FreeCAD installed in a system and bundles. Now we only have to find a way how to install to a custom user directory...
User avatar
sgrogan
Posts: 5252
Joined: Wed Oct 22, 2014 5:02 pm

Re: pip-integration in addon-manager

Postby sgrogan » Fri Aug 31, 2018 1:33 pm

looo wrote:
Fri Aug 31, 2018 1:11 pm
so to me using pip with system packages constrained looks quite promising as this makes sense for virtual environments, FreeCAD installed in a system and bundles. Now we only have to find a way how to install to a custom user directory...
Fantastic!
I've been looking at this https://www.freecadweb.org/wiki/Start_u ... figuration
For Windows it would be easy to provide a FreeCAD.bat file. In this file we could use the -P (and/or -M) switches when starting FreeCAD. The additional modules could be stored in for instance %appdata%/FreeCAD/PY27_VC12, PY36_VC14, etc. I think the .bat file could also set the PATH to the scripts directory. We would basically be setting up a psudo-environment for FreeCAD. It would be easy to bind the .bat to a desktop shortcut with a descriptive name, i.e. FreeCAD_PY36_VC14.

What do you think?

EDIT:https://stackoverflow.com/questions/683 ... -executing

EDIT2:
I created a directory %appdata%/FreeCAD/Altpython, and moved "future" that I had previously used pip to install in the builds site-packages directory (I also deleted the .egg from site packages)

I ran FreeCAD with this .bat, placed as a sibling of FreeCAD.exe

Code: Select all

SET PATH=%PATH%;%cd%/Scripts
freecad -M %appdata%/FreeCAD/Altpython
From FreeCAD's python console I get this

Code: Select all

Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 11:48:23) [MSC v.1900 64 bit (AMD64)] on win32
Type 'help', 'copyright', 'credits' or 'license' for more information.
>>> import subprocess
>>> subprocess.Popen(["pip"])
<subprocess.Popen object at 0x00000000075073C8>
>>> import future
>>> 
Next step is to install a package from FreeCAD's python console using pip into this Altpython directory.
looo
Posts: 2744
Joined: Mon Nov 11, 2013 5:29 pm

Re: pip-integration in addon-manager

Postby looo » Fri Aug 31, 2018 10:13 pm

Is there no alternative way to add bin/Sripts to the PATH? do you have any idea why it is added for a conda-env, but not for the portable one?
User avatar
sgrogan
Posts: 5252
Joined: Wed Oct 22, 2014 5:02 pm

Re: pip-integration in addon-manager

Postby sgrogan » Fri Aug 31, 2018 10:27 pm

looo wrote:
Fri Aug 31, 2018 10:13 pm
Is there no alternative way to add bin/Sripts to the PATH? do you have any idea why it is added for a conda-env, but not for the portable one?
I think the way conda creates environments is to manipulate PATH, there must be a hidden .cfg file somewhere. Maybe FreeCAD should recognize the Scripts directory, if FreeCAD has it in it's expected location in the directory tree? This all goes back to Windows programs have to ignore/prohibit other python's than their own.

In a conda environment Python is one level above where it normally is on Win (for FreeCAD at least), this was the key to making portable builds, I had to copy the python stuff so Python.exe is a sibling of FreeCAd.exe
User avatar
DeepSOIC
Posts: 6872
Joined: Fri Aug 29, 2014 12:45 am
Location: Saint-Petersburg, Russia

Re: pip-integration in addon-manager

Postby DeepSOIC » Fri Aug 31, 2018 11:22 pm

There are some problems with bat file approach.
1 is that bat screens off the icon. If one makes a shortcut, it won't have that nice FreeCAD logo on it. Windows does allow to pick an external shortcut icon, but it's annoying still. Same applies to associated files inheriting FreeCAD icon - with bat file it won't happen.

2 is semi-advanced users. They will find FreeCAD.exe, try that it actually runs, and bind it as open-with for FCStd, DXF, STEP and so on... And then maybe a few months later run into that problem of not being able to install an add-on

3 you have to channel command-line arguments though the bat. It's quite easy, apparently

4. bat file terminates ass soon as it launches FreeCAD. This can cause problems with opening temporary files, where an application would write a file to temp location, run freecad, and wait for it to finish and wipe out the file. I think, WinRAR does it...