[Merged] PR #5376: Addon Manager Install Dependencies

Post here if you have re-based and finalised code to integrate into master, which was discussed, agreed to and tested in other forums. You can also submit your PR directly on github.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
User avatar
chennes
Veteran
Posts: 3868
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

[Merged] PR #5376: Addon Manager Install Dependencies

Post by chennes »

Addon manager: install dependencies

This PR implements FreeCAD and Python dependency installation when using the Addon Manager to install an Addon.

For historical and compatibility reasons, three different metadata files are examined for dependency information:
  • package.xml -- Preferred, specify all types of dependencies in this file
  • metadata.txt -- Optional, deprecated: for FreeCAD workbench and Python required and optional dependencies
  • requirements.txt -- Optional: for Python required dependencies, might be useful for some packagers
These files are consolidated into a single dependency source (note that version information is stripped from requirements.txt, and no pip options or file include options are supported in that file). This information is cached for each Addon in the Addon Manager.

When a user requests installation of a package with dependencies, the Addon Manager resolves them as follows:
  1. Dependencies on FreeCAD Workbenches are checked against the installed WB list, and if not satisfied the installation is terminated: this is an unrecoverable condition, a new copy of FreeCAD with the appropriate Workbenches is required.
  2. Dependencies on other FreeCAD Addons are checked, and if found are added to a list of required dependencies.
  3. Dependencies on Python packages are checked by attempting to import the required package, and catching the import error on failure and adding the failure to the list of required dependencies.
  4. Optional Python dependencies are checked the same way as required, but added to a separate list.
  5. If there are unmet dependencies they are presented to the user in three columns: one for FreeCAD addon dependencies (installed with the Addon Manager as a normal FreeCAD Addon), another for Python requirements (installed with pip into a FreeCAD-specific USER_APP_DATA_DIR/AdditionalPythonPackages directory), and finally a third column of optional Python packages, presented as a checkable list for the user to select from for installation.
Note that only packages listed in the new ALLOWED_PYTHON_PACKAGES.txt file are installed by the Addon Manager, as a malware-prevention strategy. Also, packages installed in AdditionalPythonPackages take lower precedence than system packages.

NOTE: Updated 2/25/2023 to clarify that package.xml is now the Addon Manager's preferred source for ALL dependency information. The other two are included for historical reasons, but are no longer needed for any type of dependency. Note also that Python dependency version information is NEVER respected, regardless of which file is used (including requirements.txt).
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
TheMarkster
Veteran
Posts: 5505
Joined: Thu Apr 05, 2018 1:53 am

Re: PR #5376: Addon Manager Install Dependencies

Post by TheMarkster »

chennes wrote: Wed Jan 12, 2022 8:40 pm Addon manager: install dependencies

This PR implements FreeCAD and Python dependency installation when using the Addon Manager to install an Addon.

For historical and compatibility reasons, three different metadata files are examined for dependency information:
  • package.xml -- for FreeCAD workbench and addon dependencies
  • metadata.txt -- for FreeCAD workbench and Python required and optional dependencies
  • requirements.txt -- for Python required dependencies
These files are consolidated into a single dependency source (note that version information is stripped from requirements.txt, and no pip options or file include options are supported in that file). This information is cached for each Addon in the Addon Manager.

When a user requests installation of a package with dependencies, the Addon Manager resolves them as follows:
  1. Dependencies on FreeCAD Workbenches are checked against the installed WB list, and if not satisfied the installation is terminated: this is an unrecoverable condition, a new copy of FreeCAD with the appropriate Workbenches is required.
  2. Dependencies on other FreeCAD Addons are checked, and if found are added to a list of required dependencies.
  3. Dependencies on Python packages are checked by attempting to import the required package, and catching the import error on failure and adding the failure to the list of required dependencies.
  4. Optional Python dependencies are checked the same way as required, but added to a separate list.
  5. If there are unmet dependencies they are presented to the user in three columns: one for FreeCAD addon dependencies (installed with the Addon Manager as a normal FreeCAD Addon), another for Python requirements (installed with pip into a FreeCAD-specific USER_APP_DATA_DIR/AdditionalPythonPackages directory), and finally a third column of optional Python packages, presented as a checkable list for the user to select from for installation.
Note that only packages listed in the new ALLOWED_PYTHON_PACKAGES.txt file are installed by the Addon Manager, as a malware-prevention strategy. Also, packages installed in AdditionalPythonPackages take lower precedence than system packages.
If my addon requires a dependency package (none of them do at this time) I can add this to package.xml? If so, what is the tag name to use and where to put it? I presume the other files would eventually be phased out in favor of everything going into package.xml.
User avatar
chennes
Veteran
Posts: 3868
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: PR #5376: Addon Manager Install Dependencies

Post by chennes »

Right now package.xml only has support for setting dependencies on other FreeCAD AddOns, via the <depend> tag. I have not decided whether to bake Python dependencies into that as well -- I played around with it a bit, but since requirements.txt is actually directly supported by pip it seemed to make more sense to just let Python dependencies reside there.
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
TheMarkster
Veteran
Posts: 5505
Joined: Thu Apr 05, 2018 1:53 am

Re: PR #5376: Addon Manager Install Dependencies

Post by TheMarkster »

Okay. If requirements.txt is used by pip, then it makes sense to leave them there if the dependencies are installed by pip.
drmacro
Veteran
Posts: 8806
Joined: Sun Mar 02, 2014 4:35 pm

Re: PR #5376: Addon Manager Install Dependencies

Post by drmacro »

chennes wrote: Wed Jan 19, 2022 7:35 pm ...
Not sure this is related...

Today's source attempt to use Addon manager:

Code: Select all

08:29:38  Traceback (most recent call last):
08:29:38    File "/home/mac/freecad-build/Mod/AddonManager/AddonManager.py", line 258, in launch
08:29:38      self.cleanup_workers()
08:29:38    File "/home/mac/freecad-build/Mod/AddonManager/AddonManager.py", line 420, in cleanup_workers
08:29:38      finished = thread.wait(QtCore.QDeadlineTimer(500))
08:29:38  AttributeError: module 'PySide2.QtCore' has no attribute 'QDeadlineTimer'
OS: Debian GNU/Linux 10 (buster) (XFCE/lightdm-xsession)
Word size of FreeCAD: 64-bit
Version: 0.20.27241 (Git)
Build type: Release
Branch: master
Hash: 2cb478fd66dc76d91695dacff15fd764e80c8c62
Python version: 3.7.3
Qt version: 5.11.3
Coin version: 4.0.0a
OCC version: 7.3.0
Locale: English/UnitedStates (en_US)
Star Trek II: The Wrath of Khan: Spock: "...His pattern indicates two-dimensional thinking."
User avatar
chennes
Veteran
Posts: 3868
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: PR #5376: Addon Manager Install Dependencies

Post by chennes »

drmacro wrote: Wed Jan 26, 2022 1:33 pm

Code: Select all

08:29:38  AttributeError: module 'PySide2.QtCore' has no attribute 'QDeadlineTimer'
That's very... interesting. QDeadlineTimer was added in Qt 5.8, so that absolutely should exist. I'll see if I can build a VM with Qt 5.11.3 to look into this.
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
User avatar
Zolko
Veteran
Posts: 2213
Joined: Mon Dec 17, 2018 10:02 am

Re: PR #5376: Addon Manager Install Dependencies

Post by Zolko »

chennes wrote: Wed Jan 26, 2022 1:42 pm
drmacro wrote: Wed Jan 26, 2022 1:33 pm

Code: Select all

08:29:38  AttributeError: module 'PySide2.QtCore' has no attribute 'QDeadlineTimer'
That's very... interesting. QDeadlineTimer was added in Qt 5.8, so that absolutely should exist. I'll see if I can build a VM with Qt 5.11.3 to look into this.
I have the same error, with latest built from GitHub

Code: Select all

08:54:09      finished = thread.wait(QtCore.QDeadlineTimer(500))
08:54:09  AttributeError: module 'PySide2.QtCore' has no attribute 'QDeadlineTimer'
it's only a wait to be sure the task is finished, is it really worth the trouble to use some fancy deadline timer ? Why not simply a QTime(500) or similar ?


OS: Debian GNU/Linux 10 (buster) (KDE//usr/share/xsessions/plasma)
Word size of FreeCAD: 64-bit
Version: 0.20.27195 (Git)
Build type: Unknown
Branch: master
Hash: 2cb478fd66dc76d91695dacff15fd764e80c8c62
Python version: 3.7.3
Qt version: 5.11.3
Coin version: 4.0.0a
OCC version: 7.3.0
Locale: English/UnitedKingdom (en_GB)
try the Assembly4 workbench for FreCAD — tutorials here and here
User avatar
chennes
Veteran
Posts: 3868
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: PR #5376: Addon Manager Install Dependencies

Post by chennes »

Zolko wrote: Thu Jan 27, 2022 7:57 am it's only a wait to be sure the task is finished, is it really worth the trouble to use some fancy deadline timer ? Why not simply a QTime(500) or similar ?
Well, it isn't supposed to be "fancy"! Regardless, I think the explicit instantiation is a relic from some debugging, so while it is supposed to work, apparently in some version of Qt and/or PySide it's broken: can you test just replacing the QtCore.QDeadlineTimer(500) with simply 500 and see if that runs for you?
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
User avatar
Zolko
Veteran
Posts: 2213
Joined: Mon Dec 17, 2018 10:02 am

Re: PR #5376: Addon Manager Install Dependencies

Post by Zolko »

chennes wrote: Thu Jan 27, 2022 1:35 pm can you test just replacing the QtCore.QDeadlineTimer(500) with simply 500 and see if that runs for you?
yes, the AddonManager seems to work with this, thank-you

As a comment, the image of the ReadMe is larger than the window, and while the text flows nicely, the image is not scaled to the window's width. It's only a cosmetic, but it would look more professional if the image was scaled too. Not a big deal though.
try the Assembly4 workbench for FreCAD — tutorials here and here
User avatar
chennes
Veteran
Posts: 3868
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: PR #5376: Addon Manager Install Dependencies

Post by chennes »

I am not happy with the display of the README, so have been experimenting with using a QWebEngineView to do the download, cache, and render of that file. You can see what that would be like in this PR: https://github.com/FreeCAD/FreeCAD/pull/5413 There are some pros and cons of doing it that way.
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
Post Reply