Lazy loading of python modules

[etrombly]

While investigating the slow loading of the path workbench I found that one of the main culprits was imports of other workbenches. I was able to fix it in a few places by using deferred loading, but some modules were used in several places, and deferred importing would be awkward. I found the LazyLoader class from tensorflow, described here https://wil.yegelwel.com/lazily-importi ... n-modules/ source is here https://github.com/tensorflow/tensorflo ... _loader.py . It seems to work well from testing, and could be used by other workbenches. It would be a single file copied, so no external deps, it's apache licensed which is compatible with LGPL2+ (specifically LGPL3). Where should I put it to be usable by all workbenches? App, Base?

[vocx]

... Where should I put it to be usable by all workbenches? App, Base?

Probably the best place is src/3rdParty. This should be done for all external packages that are copied to our source tree, although I think this is not followed strictly, as some workbenches include third party code inside its own src/Mod/ directory.

About lazy loading, a trick that we do in the Draft and Arch workbenches is basically load the required module at runtime, when we run a specific function.

Code: Select all

def some_operation():
    import Part
    do_something(Part.Face())
    ...

Now, I dislike this, and I think it's a bit overused, but according to the main developer of these workbenches, it was done like this because loading the Part module is very time consuming. We are currently refactoring Draft, so I think we can get rid of some of these imports, but we will keep them in some cases.

However, if you can find a more natural and Pythonic solution, that would be good. I haven't tried this LazyLoader class before, so it's great if you can investigate more about it.

[etrombly]

I moved several imports into the function body of modules. I was looking in to using lazyloader instead because some modules have several functions that need the import statements. It tends to clutter things up. Also with lazyloader even if there's just one import, we can still put it at the top, and it won't be loaded until the first use. I've done a quick test and it seems to work well.

[vocx]

I moved several imports into the function body of modules...

What types of imports? From Path itself or from other workbenches. I don't know how exactly the workbench is setup, but last time I took a look at it, it had plenty of modules, and classes upon classes.

[chrisb]

In Path this could indeed be beneficial. I would guess that the external libraries take quite some time. And if 3DSurface is not used in the model, opencamlib does not have to be loaded. However, this may require some more changes, because currently 3DSurface is enabled only if the library can be accessed.

[etrombly]

What types of imports? From Path itself or from other workbenches. I don't know how exactly the workbench is setup, but last time I took a look at it, it had plenty of modules, and classes upon classes.

Path Job Gui import Draft and DraftVecUtils, PathJob and several other modules import ArchPanel, etc. A lot of the modules only used the imported workbenches for a single function, so I moved those down in to the function body for testing, but it wasn't always the case. Also I'd like to keep the imports at the top of the file, mainly for readability.

When I get a chance I'll throw together a PR for this. Just wanted some input first on where to put the LazyLoader class, and the license compatibility. Unless anyone has other input I'll throw it in third party for now.

[Kunda1]

When I get a chance I'll throw together a PR for this. Just wanted some input first on where to put the LazyLoader class, and the license compatibility. Unless anyone has other input I'll throw it in third party for now.

Can you post the repo that you are conducting these tests in so we can see the code before you submit a PR ?

[sgrogan]

Unless anyone has other input I'll throw it in third party for now.

This will be needed at runtime?
If so we will need to copy the class to the build dir and add it to the INSTALL project.

[etrombly]

I'm working on it here https://github.com/etrombly/FreeCAD/tree/lazyloader . I moved it out of path and into 3rd Party, so it's not working currently. Trying to figure out the cmake file for it. I think I have it copying to the build dir and installing correctly, but can't get it to add to the python path. Just learning cmake now.

[vocx]

... but can't get it to add to the python path. Just learning cmake now.

I think you can install it to the Ext/ directory. It's supposed to be in the Python path.

Alternatively, you may wish to add it to Mod/ as this should also be in the path, but that doesn't look as nice.