How to run Addon Manager in Headless mode?

Need help, or want to share a macro? Post here!
User avatar
Kunda1
Posts: 5937
Joined: Thu Jan 05, 2017 9:03 pm

How to run Addon Manager in Headless mode?

Postby Kunda1 » Tue Oct 22, 2019 9:28 pm

I'd like to attempt to modify the Addon Manager source so that it could be run headless.
I've noticed that there is a method called FreeCAD.GuiUp for logic that pertains to progress bars and dialog windows can be put in to..correct ?

Code: Select all

if FreeCAD.GuiUp:
	babyShark()
else:
	mommyShark()

What I'm trying to do specifically is first access functions within src/Mod/AddonManager/ like querying the FreeCAD-addons Github repo, and listing their contents, listing what addons currently live in the local machine, list which addons have updated, install an addon/macro, remove and addon/macro etc...

Note: I opened a FR thread https://forum.freecadweb.org/viewtopic.php?f=8&t=40297
but felt it was more appropriate to ask for help in the Python scripting subforum in a separate thread

Edit:
To test the script simply:

Code: Select all

wget https://gist.githubusercontent.com/luzpaz/f3bfaaef8aaec9c66b96e0941f6ed7a5/raw/AddonMangerHeadless.py
FreeCAD -c AddonMangerHeadless.py
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
vocx
Posts: 1886
Joined: Thu Oct 18, 2018 9:18 pm

Re: How to run Addon Manager in Headless mode?

Postby vocx » Tue Oct 22, 2019 10:04 pm

Kunda1 wrote:
Tue Oct 22, 2019 9:28 pm
...
What I'm trying to do specifically is first access functions within src/Mod/AddonManager/ like querying the FreeCAD-addons Github repo, and listing their contents, listing what addons currently live in the local machine, list which addons have updated, install an addon/macro, remove and addon/macro etc...
It seems the Addon Manager was more or less designed to be utilized from the graphical interface, so it doesn't have a proper scripting interface for non-GUI usage. The primary graphical command, what I call, a GuiCommand, is defined in AddonManager.py (class CommandAddonManager).

The top imports indicate the low level functions that are actually used by the GuiCommand.

Code: Select all

import addonmanager_utilities as utils
from addonmanager_utilities import translate # this needs to be as is for pylupdate
from addonmanager_workers import *
https://github.com/FreeCAD/FreeCAD/blob ... ger.py#L69

The Activated() function indicates which code is run when the Addon Manager is launched. It just calls launch(). This command sets up the graphical interface through the AddonManager.ui file, but then it calls the actual functions that do stuff.

Code: Select all

        # populate the list
        self.update()
So, what does update() do? The function update() calls a class that is probably defined in one of the imported modules, addonmanager_workers.

Code: Select all

 self.update_worker = UpdateWorker()
Then it does all sorts of actions defined in this UpdateWorker class.

It seems this addonmanager_workers.py module has the classes that actually do work. But they are coded in such a way that they are tightly connected with the graphical interface because they use things like progress bars and labels. In order to run these commands entirely without the graphical interface you would need to decouple the functionality of those classes.

https://github.com/FreeCAD/FreeCAD/blob ... rs.py#L557

For example, the class InstallWorkbench has a run() method which seems to do actual work to install new workbenches. But this method also accesses the elements of the graphical .ui file. You need to break this coupling.

Code: Select all

self.info_label.emit("no zip support.")
...
self.info_label.emit("Updating module...")
Said in another way, you need to write your own functions that just do stuff, but don't access the graphical interface at all. Then from the graphical interface you could call these new functions.

Then you would have something like

Code: Select all

import addonmanager_core as core

GraphicalCode():
    do_some_stuff() # GUI stuff
    arrange_labels() # GUI stuff
    result = core.call_basic_functions(repo)  # Non GUI stuff, Git or whatever behind the scenes
    updated = core.call_update(workbench)  # Non GUI stuff, Git or whatever behind the scenes
    do_more_stuff(result, updated)  # GUI stuff that does something with the results of the non-GUI functions
User avatar
sgrogan
Posts: 5472
Joined: Wed Oct 22, 2014 5:02 pm

Re: How to run Addon Manager in Headless mode?

Postby sgrogan » Tue Oct 22, 2019 11:09 pm

Kunda1 wrote:
Tue Oct 22, 2019 9:28 pm
I'd like to attempt to modify the Addon Manager source so that it could be run headless.
This would be a great GSoC project.
Addons-manager started as a macro and has grown into a full fledged module.
For workbenches, at least, the App/Gui distiction is strictly enforced.
Anything accessible from freecadcmd, should not use the Gui. In your particular case, maybe the progress bar can be replaced with "chasing dots" or whatever in the console.

Having this capability available headless opens up possibilities, CRON jobs for ex.
User avatar
Kunda1
Posts: 5937
Joined: Thu Jan 05, 2017 9:03 pm

Re: How to run Addon Manager in Headless mode?

Postby Kunda1 » Wed Oct 23, 2019 12:10 am

Thanks @vocx
super helpful as usual. Thanks for breaking it down. I was feeling inspired so I started migrating the code.
Here's a way to simply list all available official FreeCAD addons with the additional feature of knowing which ones are installed.

Code: Select all

 -*- coding: utf-8 -*-
# FreeCAD Headless AddonManager test
# (c) 2019 FreeCAD community LGPL

"""
The module can be executed with:
./FreeCAD.AppImage -c <path_to_file> AddonManagerHeadless.py
"""

import os, re, shutil, stat, sys, tempfile
import FreeCAD
import addonmanager_utilities as utils

u = utils.urlopen("https://github.com/FreeCAD/FreeCAD-addons")

if not u:
    print("Unable to open URL")
p = u.read()

if sys.version_info.major >= 3 and isinstance(p, bytes):
    p = p.decode("utf-8")
u.close()

p = p.replace("\n"," ")
p = re.findall("octicon-file-submodule(.*?)message",p)
basedir = FreeCAD.getUserAppDataDir()
moddir = basedir + os.sep + "Mod"
repos = []
# querying official addons
for l in p:
    #name = re.findall("data-skip-pjax=\"true\">(.*?)<",l)[0]
    res = re.findall("title=\"(.*?) @",l)
    if res:
        name = res[0]
    else:
        print("AddonMananger: Debug: couldn't find title in",l)
        continue
    # Print repo name by itself
    # print(name)
    #url = re.findall("title=\"(.*?) @",l)[0]
    url = utils.getRepoUrl(l)
    if url:
        addondir = moddir + os.sep + name
        #print ("found:",name," at ",url)
        if os.path.exists(addondir) and os.listdir(addondir):
            # make sure the folder exists and it contains files!
            state = 1
        else:
            state = 0
        repos.append([name,url,state])
if not repos:
    print("Unable to download addon list.")
else:
    repos = sorted(repos, key=lambda s: s[0].lower())
    print("List official available Addons:\n")
    for repo in repos:
        if repo[2] == 1:
            repo[2] = '+'
            print(repo[2] +'  '+ repo[0] +' -> '+ repo[1])
        if repo[2] == 0:
            repo[2] = 'X'
            print(repo[2] +'  '+ repo[0] +' -> '+ repo[1])
    print('\tLegend:\n\t+ = Installed\n\tX = Not Installed')
    # print("Workbenches list was updated.")
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
User avatar
Kunda1
Posts: 5937
Joined: Thu Jan 05, 2017 9:03 pm

Re: How to run Addon Manager in Headless mode?

Postby Kunda1 » Wed Oct 23, 2019 12:45 pm

How should I structure this script?
Should I use classes or just a bunch of functions?
I'm getting tripped up in sending the results of one class/function to another class/function

Edit: https://gist.github.com/luzpaz/f3bfaaef ... 941f6ed7a5

I have a class for ListRepo()

Code: Select all

class ListRepo():
    u = utils.urlopen("https://github.com/FreeCAD/FreeCAD-addons")
and a class CheckUpdateRepo()

Code: Select all

class CheckUpdateRepo():    
I'm calling them from:

Code: Select all

if __name__ == '__main__':
    print("start")
    results = ListRepo()
    print("middle")
    CheckUpdateRepo(results)
but i get nothing when I run the script.
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
vocx
Posts: 1886
Joined: Thu Oct 18, 2018 9:18 pm

Re: How to run Addon Manager in Headless mode?

Postby vocx » Wed Oct 23, 2019 3:00 pm

Kunda1 wrote:
Wed Oct 23, 2019 12:45 pm
How should I structure this script?
Should I use classes or just a bunch of functions?
...
You can use simple functions to start. I think it's easier.

I feel this type of code is inherently "imperative", and thus doesn't need classes, it just needs to do something, get the result, and use that result in another function.

Code: Select all

repo = get_repo(string)
result = update_repo(repo)
for workbench in result:
    print_update_available(workbench)
Classes should be used when you have an "object". What are the objects in your code? If you cannot answer this question simply, then it probably doesn't make much sense to use a class.

To answer your question. All classes are initialized with an __init__() function. You must specify the input parameters here.

Code: Select all

class ListRepo:
    __init__(self, string="empty"):
    self.repo = string

class CheckUpdateRepo:
    __init__(self, string="empty")
    self.repo = string
    
    def update_repo():
        # do something with self.repo here
        print(self.repo)
        return True
Then

Code: Select all

object1 = ListRepo("http://blabla")
object2 = CheckUpdateRepo(object1.repo)
object2.update_repo()
But again, since you are just passing one thing from the other thing, I don't think it makes sense to use classes, just functions.
User avatar
Kunda1
Posts: 5937
Joined: Thu Jan 05, 2017 9:03 pm

Re: How to run Addon Manager in Headless mode?

Postby Kunda1 » Wed Oct 23, 2019 3:44 pm

vocx wrote:
Wed Oct 23, 2019 3:00 pm
I feel this type of code is inherently "imperative", and thus doesn't need classes, it just needs to do something, get the result, and use that result in another function.
Roger that. I will switch to functions. Thanks for also explaining classes and objects, noted!

Ok, updated the gist https://gist.github.com/luzpaz/f3bfaaef ... 941f6ed7a5

But now my issues is when I run the script nothing is outputted except a print statement that occurs before this block

Code: Select all

if __name__ == '__main__':
    print("start")
    results = ListRepo()
    print("middle")
    CheckUpdateRepo(results)
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
vocx
Posts: 1886
Joined: Thu Oct 18, 2018 9:18 pm

Re: How to run Addon Manager in Headless mode?

Postby vocx » Wed Oct 23, 2019 3:55 pm

Kunda1 wrote:
Wed Oct 23, 2019 3:44 pm
...
But now my issues is when I run the script nothing is outputted except a print statement that occurs before this block
...
Add more print statements to see where it is failing. Your ListRepo function isn't guaranteed to always work. You should probably terminate it early if it fails at some point. Then you can analyze the causes of the failure.

For example

Code: Select all

def ListRepo():
    u = utils.urlopen("https://github.com/FreeCAD/FreeCAD-addons")

    if not u:
        print("Unable to open URL")
        return False
    p = u.read()
    ...
If I'm not able to read the URL, it should probably exit immediately, otherwise nothing else is going to work.

How are you running your code? I'd use the macro editor and just test there. Then you don't need the __name__ test.

Code: Select all

def List Repo():
...
def ShowAddon():
...

print("start")
results = ListRepo()
print("middle")
CheckUpdateRepo(results)
Last edited by vocx on Wed Oct 23, 2019 11:14 pm, edited 1 time in total.
User avatar
Kunda1
Posts: 5937
Joined: Thu Jan 05, 2017 9:03 pm

Re: How to run Addon Manager in Headless mode?

Postby Kunda1 » Wed Oct 23, 2019 4:13 pm

vocx wrote:
Wed Oct 23, 2019 3:55 pm
How are you running your code? I'd use the macro editor and just test there. Then you don't need the __name__ test.
I'm running the code through ./FreeCAD.AppImage -c AddonManagerHeadless.py

macro editor is not as powerful as the editor i like to use. But yea.. I'll use it to not make things harder.
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
User avatar
Kunda1
Posts: 5937
Joined: Thu Jan 05, 2017 9:03 pm

Re: How to run Addon Manager in Headless mode?

Postby Kunda1 » Wed Oct 23, 2019 5:56 pm

Running the code externally or through the macro editor (removed __init__ block)
and i still can't get any output. annoying
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features