Lazy loading of python modules

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
etrombly
Posts: 80
Joined: Thu Dec 05, 2019 6:50 pm

Lazy loading of python modules

Postby etrombly » Wed Apr 08, 2020 1:49 pm

While investigating the slow loading of the path workbench I found that one of the main culprits was imports of other workbenches. I was able to fix it in a few places by using deferred loading, but some modules were used in several places, and deferred importing would be awkward. I found the LazyLoader class from tensorflow, described here https://wil.yegelwel.com/lazily-importi ... n-modules/ source is here https://github.com/tensorflow/tensorflo ... _loader.py . It seems to work well from testing, and could be used by other workbenches. It would be a single file copied, so no external deps, it's apache licensed which is compatible with LGPL2+ (specifically LGPL3). Where should I put it to be usable by all workbenches? App, Base?
vocx
Posts: 3962
Joined: Thu Oct 18, 2018 9:18 pm

Re: Lazy loading of python modules

Postby vocx » Thu Apr 09, 2020 12:18 am

etrombly wrote:
Wed Apr 08, 2020 1:49 pm
... Where should I put it to be usable by all workbenches? App, Base?
Probably the best place is src/3rdParty. This should be done for all external packages that are copied to our source tree, although I think this is not followed strictly, as some workbenches include third party code inside its own src/Mod/ directory.

About lazy loading, a trick that we do in the Draft and Arch workbenches is basically load the required module at runtime, when we run a specific function.

Code: Select all

def some_operation():
    import Part
    do_something(Part.Face())
    ...
Now, I dislike this, and I think it's a bit overused, but according to the main developer of these workbenches, it was done like this because loading the Part module is very time consuming. We are currently refactoring Draft, so I think we can get rid of some of these imports, but we will keep them in some cases.

However, if you can find a more natural and Pythonic solution, that would be good. I haven't tried this LazyLoader class before, so it's great if you can investigate more about it.
Always add the important information to your posts if you need help.
To support the documentation effort, and code development, your donation is appreciated: paypal.
etrombly
Posts: 80
Joined: Thu Dec 05, 2019 6:50 pm

Re: Lazy loading of python modules

Postby etrombly » Thu Apr 09, 2020 2:09 am

I moved several imports into the function body of modules. I was looking in to using lazyloader instead because some modules have several functions that need the import statements. It tends to clutter things up. Also with lazyloader even if there's just one import, we can still put it at the top, and it won't be loaded until the first use. I've done a quick test and it seems to work well.
vocx
Posts: 3962
Joined: Thu Oct 18, 2018 9:18 pm

Re: Lazy loading of python modules

Postby vocx » Thu Apr 09, 2020 3:00 am

etrombly wrote:
Thu Apr 09, 2020 2:09 am
I moved several imports into the function body of modules...
What types of imports? From Path itself or from other workbenches. I don't know how exactly the workbench is setup, but last time I took a look at it, it had plenty of modules, and classes upon classes.
Always add the important information to your posts if you need help.
To support the documentation effort, and code development, your donation is appreciated: paypal.
chrisb
Posts: 24233
Joined: Tue Mar 17, 2015 9:14 am

Re: Lazy loading of python modules

Postby chrisb » Thu Apr 09, 2020 7:07 am

In Path this could indeed be beneficial. I would guess that the external libraries take quite some time. And if 3DSurface is not used in the model, opencamlib does not have to be loaded. However, this may require some more changes, because currently 3DSurface is enabled only if the library can be accessed.
etrombly
Posts: 80
Joined: Thu Dec 05, 2019 6:50 pm

Re: Lazy loading of python modules

Postby etrombly » Thu Apr 09, 2020 12:12 pm

vocx wrote:
Thu Apr 09, 2020 3:00 am
What types of imports? From Path itself or from other workbenches. I don't know how exactly the workbench is setup, but last time I took a look at it, it had plenty of modules, and classes upon classes.
Path Job Gui import Draft and DraftVecUtils, PathJob and several other modules import ArchPanel, etc. A lot of the modules only used the imported workbenches for a single function, so I moved those down in to the function body for testing, but it wasn't always the case. Also I'd like to keep the imports at the top of the file, mainly for readability.

When I get a chance I'll throw together a PR for this. Just wanted some input first on where to put the LazyLoader class, and the license compatibility. Unless anyone has other input I'll throw it in third party for now.
User avatar
Kunda1
Posts: 7319
Joined: Thu Jan 05, 2017 9:03 pm

Re: Lazy loading of python modules

Postby Kunda1 » Thu Apr 09, 2020 1:26 pm

etrombly wrote:
Thu Apr 09, 2020 12:12 pm
When I get a chance I'll throw together a PR for this. Just wanted some input first on where to put the LazyLoader class, and the license compatibility. Unless anyone has other input I'll throw it in third party for now.
Can you post the repo that you are conducting these tests in so we can see the code before you submit a PR ?
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
User avatar
sgrogan
Posts: 5800
Joined: Wed Oct 22, 2014 5:02 pm

Re: Lazy loading of python modules

Postby sgrogan » Thu Apr 09, 2020 9:43 pm

etrombly wrote:
Thu Apr 09, 2020 12:12 pm
Unless anyone has other input I'll throw it in third party for now.
This will be needed at runtime?
If so we will need to copy the class to the build dir and add it to the INSTALL project.
etrombly
Posts: 80
Joined: Thu Dec 05, 2019 6:50 pm

Re: Lazy loading of python modules

Postby etrombly » Thu Apr 09, 2020 11:17 pm

I'm working on it here https://github.com/etrombly/FreeCAD/tree/lazyloader . I moved it out of path and into 3rd Party, so it's not working currently. Trying to figure out the cmake file for it. I think I have it copying to the build dir and installing correctly, but can't get it to add to the python path. Just learning cmake now.
vocx
Posts: 3962
Joined: Thu Oct 18, 2018 9:18 pm

Re: Lazy loading of python modules

Postby vocx » Thu Apr 09, 2020 11:51 pm

etrombly wrote:
Thu Apr 09, 2020 11:17 pm
... but can't get it to add to the python path. Just learning cmake now.
I think you can install it to the Ext/ directory. It's supposed to be in the Python path.

Alternatively, you may wish to add it to Mod/ as this should also be in the path, but that doesn't look as nice.
Always add the important information to your posts if you need help.
To support the documentation effort, and code development, your donation is appreciated: paypal.