pip-integration in addon-manager

Need help, or want to share a macro? Post here!
User avatar
DeepSOIC
Posts: 7232
Joined: Fri Aug 29, 2014 12:45 am
Location: Saint-Petersburg, Russia

Re: pip-integration in addon-manager

Postby DeepSOIC » Wed Aug 29, 2018 11:05 am

sgrogan wrote:
Wed Aug 29, 2018 11:01 am
Packaging of pip is still messed in your version. It should be fixed in the newest x64 builds.
OK, will try.


I tried in my conda install, it worked flawlessly from anaconda prompt. But it only installed the workbench for the conda freecad. Other freecads don't see the new workbench. I guess, it is supposed to be like that...
User avatar
sgrogan
Posts: 5540
Joined: Wed Oct 22, 2014 5:02 pm

Re: pip-integration in addon-manager

Postby sgrogan » Wed Aug 29, 2018 11:19 am

DeepSOIC wrote:
Wed Aug 29, 2018 11:05 am
I tried in my conda install, it worked flawlessly from anaconda prompt. But it only installed the workbench for the conda freecad. Other freecads don't see the new workbench. I guess, it is supposed to be like that...
Yes, this is what looo and I am discussing in the other thread. We need a way to get pip to install into %appdata%/freecad/mod or the like. For packages with compiled extensions we somehow have to resolve potential dependency problems.
For pure python I think this should work https://stackoverflow.com/questions/416 ... tion-path#
User avatar
looo
Posts: 3026
Joined: Mon Nov 11, 2013 5:29 pm

Re: pip-integration in addon-manager

Postby looo » Wed Aug 29, 2018 11:33 am

2 things:
1. pip.main won't work for newer versions of pip. So better use supprocess.Popen(["pip", "install", ..])

2. to install to any non-std-directory:
pip install --install-option="--install-purelib=/python/packages" package_name

or setting the environment variable "PYTHONUSERBASE", which might be the better option.
regarding: https://stackoverflow.com/questions/291 ... -using-pip
please help with my conda-packaging efforts: https://liberapay.com/looooo/
minimalistic blog: https://looooo.github.io/mini-blog/
User avatar
DeepSOIC
Posts: 7232
Joined: Fri Aug 29, 2014 12:45 am
Location: Saint-Petersburg, Russia

Re: pip-integration in addon-manager

Postby DeepSOIC » Wed Aug 29, 2018 11:45 am

looo wrote:
Wed Aug 29, 2018 11:33 am
1. pip.main won't work for newer versions of pip. So better use supprocess.Popen(["pip", "install", ..])

Code: Select all

>>> import subprocess
>>> subprocess.Popen(["pip"])
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\FreeCAD\builds-snapshots\Default\bin\lib\subprocess.py", line 390, in __init__
    errread, errwrite)
  File "C:\FreeCAD\builds-snapshots\Default\bin\lib\subprocess.py", line 640, in _execute_child
    startupinfo)
WindowsError: [Error 2] 
>>> 
That's from same portable FC. From conda freecad, it works.
User avatar
DeepSOIC
Posts: 7232
Joined: Fri Aug 29, 2014 12:45 am
Location: Saint-Petersburg, Russia

Re: pip-integration in addon-manager

Postby DeepSOIC » Wed Aug 29, 2018 11:54 am

sgrogan wrote:
Wed Aug 29, 2018 11:19 am
We need a way to get pip to install into %appdata%/freecad/mod or the like.
Actually, per-instance installs are also useful. I'd like there be an option. For example, FCGear seems to somehow cause a crash onto my debug builds, so I have to uninstall it temporarily to do C++ work.
User avatar
sgrogan
Posts: 5540
Joined: Wed Oct 22, 2014 5:02 pm

Re: pip-integration in addon-manager

Postby sgrogan » Wed Aug 29, 2018 12:14 pm

Code: Select all

>>> subprocess.Popen(["C:/Users/chris/Downloads/FreeCAD_0.18.14487_x64_dev_win/bin/Scripts/pip"])
<subprocess.Popen object at 0x0000000009CB8358>
>>> 
I tried adding the path to sys.path but it didn't work.
User avatar
looo
Posts: 3026
Joined: Mon Nov 11, 2013 5:29 pm

Re: pip-integration in addon-manager

Postby looo » Wed Aug 29, 2018 12:23 pm

sgrogan wrote:
Wed Aug 29, 2018 12:14 pm

Code: Select all

>>> subprocess.Popen(["C:/Users/chris/Downloads/FreeCAD_0.18.14487_x64_dev_win/bin/Scripts/pip"])
<subprocess.Popen object at 0x0000000009CB8358>
>>> 
I tried adding the path to sys.path but it didn't work.
sys.path is only for python. As pip is a system command you have to add it to $PATH. But I don't know what the windows equivalent is.
please help with my conda-packaging efforts: https://liberapay.com/looooo/
minimalistic blog: https://looooo.github.io/mini-blog/
User avatar
sgrogan
Posts: 5540
Joined: Wed Oct 22, 2014 5:02 pm

Re: pip-integration in addon-manager

Postby sgrogan » Wed Aug 29, 2018 1:24 pm

Code: Select all

>>> import sys
>>> import subprocess
>>> subprocess.Popen([sys.exec_prefix + "\\Scripts\\pip"])
<subprocess.Popen object at 0x000000000A5FBD30>
>>>
Should make sure we use FreeCAD's Python.
User avatar
DeepSOIC
Posts: 7232
Joined: Fri Aug 29, 2014 12:45 am
Location: Saint-Petersburg, Russia

Re: pip-integration in addon-manager

Postby DeepSOIC » Wed Aug 29, 2018 1:26 pm

looo wrote:
Wed Aug 29, 2018 12:23 pm
As pip is a system command you have to add it to $PATH. But I don't know what the windows equivalent is.
This seems to work:

Code: Select all

>>> os.environ['PATH'] = os.environ['PATH'] + r';S:\_vt\dev\PC\Qt\FreeCAD\builds-snapshots\Default\bin\Scripts' #terribly non-crossplatform, just for test
>>> subprocess.Popen(["pip"])
<subprocess.Popen object at 0x00000257A3533A58>
>>> 
Just in case, here's my content of PATH:

Code: Select all

>>> for val in os.environ['PATH'].split(';'): print(val)
... 
C:\FreeCAD\builds-snapshots\Default\bin
C:\FreeCAD\builds-snapshots\Default\Mod\Sketcher
C:\FreeCAD\builds-snapshots\Default\Mod\ReverseEngineering
C:\FreeCAD\builds-snapshots\Default\Mod\Show
C:\FreeCAD\builds-snapshots\Default\Mod\Image
C:\FreeCAD\builds-snapshots\Default\Mod\Surface
C:\FreeCAD\builds-snapshots\Default\Mod\PartDesign
C:\Users\EMB\AppData\Roaming\FreeCAD\Mod\TabBar
C:\FreeCAD\builds-snapshots\Default\Mod\Measure
C:\Users\EMB\AppData\Roaming\FreeCAD\Mod\Lattice2
C:\FreeCAD\builds-snapshots\Default\Mod\Ship
C:\FreeCAD\builds-snapshots\Default\Mod\Plot
C:\FreeCAD\builds-snapshots\Default\Mod\MeshPart
C:\FreeCAD\builds-snapshots\Default\Mod\OpenSCAD
C:\FreeCAD\builds-snapshots\Default\Mod\Fem
C:\FreeCAD\builds-snapshots\Default\Mod\Spreadsheet
C:\FreeCAD\builds-snapshots\Default\Mod\Start
C:\FreeCAD\builds-snapshots\Default\Mod\Points
C:\FreeCAD\builds-snapshots\Default\Mod\Draft
C:\Users\EMB\AppData\Roaming\FreeCAD\Mod\_disabled
C:\FreeCAD\builds-snapshots\Default\Mod\Test
C:\FreeCAD\builds-snapshots\Default\Mod\Import
C:\FreeCAD\builds-snapshots\Default\Mod\TechDraw
C:\FreeCAD\builds-snapshots\Default\Mod\Tux
C:\FreeCAD\builds-snapshots\Default\Mod\Complete
C:\FreeCAD\builds-snapshots\Default\Mod\Web
C:\FreeCAD\builds-snapshots\Default\Mod\Material
C:\FreeCAD\builds-snapshots\Default\Mod\Raytracing
C:\FreeCAD\builds-snapshots\Default\Mod\Mesh
C:\Users\EMB\AppData\Roaming\FreeCAD\Mod\Curves
C:\FreeCAD\builds-snapshots\Default\Mod\Path
C:\FreeCAD\builds-snapshots\Default\Mod\Part
C:\FreeCAD\builds-snapshots\Default\Mod\Arch
C:\FreeCAD\builds-snapshots\Default\Mod\Robot
C:\FreeCAD\builds-snapshots\Default\Mod\AddonManager
C:\Users\EMB\AppData\Roaming\FreeCAD\Mod\Part-o-Magic
C:\FreeCAD\builds-snapshots\Default\Mod\Idf
C:\FreeCAD\builds-snapshots\Default\Mod\Inspection
C:\FreeCAD\builds-snapshots\Default\Mod\Drawing
C:\Windows\system32
C:\Windows
C:\Windows\System32\Wbem
C:\Windows\System32\WindowsPowerShell\v1.0\
C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common
C:\Program Files (x86)\Skype\Phone\
C:\Qt\Git\cmd
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\WINDOWS\System32\OpenSSH\
C:\Users\EMB\AppData\Local\Microsoft\WindowsApps
C:\Qt\git\bin
C:\Users\EMB\AppData\Local\Microsoft\WindowsApps
S:\_vt\dev\PC\Qt\FreeCAD\libpack\active\bin
C:\Users\EMB\AppData\Local\GitHubDesktop\bin
S:\_vt\dev\PC\Qt\FreeCAD\builds-snapshots\Default\bin\Scripts
>>> 
User avatar
looo
Posts: 3026
Joined: Mon Nov 11, 2013 5:29 pm

Re: pip-integration in addon-manager

Postby looo » Thu Aug 30, 2018 12:39 pm

One weak point about using pip/pypi are compiled dependencies. If we are allowing users to install packages to directories shared by multiple versions of FreeCAD we have to restrict the packages which can be installed with pip to noarch-python packages (This is how pure python packages are called in the conda-world) At least we have to make sure that things like numpy / scipy / ... are updated with pip. We can achieve this by adding a constraint file which constraints packages which are already included in the freecad-bundle to there versions. This way pip will not try to update them (hopefully).

regarding https://github.com/pypa/pip/issues/3090 this can be achieved by eg.:

Code: Select all

pip install future -c constraint.txt
But somehow this doesn't work with adding an additional --target (which is the obvious way to install to a non-std directory). So maybe we should try to set the env-variable "PYTHONUSERBASE" and see if pip installs to the this directory without any additional flag. If this works, I guess adding the constraint file should also work. Then we only have to create a constraint.txt from a clean bundle (appimage / windows portable) and somehow add this constraint.txt to the bundle. Maybe adding it to Ext/freecad/pip-integration or something like this as this file should be part of the bundle and also it should be easily accessible.
please help with my conda-packaging efforts: https://liberapay.com/looooo/
minimalistic blog: https://looooo.github.io/mini-blog/