Python workbenches debugging

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
ediloren
Posts: 209
Joined: Wed May 08, 2013 9:23 pm
Location: Italy
Contact:

Python workbenches debugging

Postby ediloren » Mon Apr 01, 2019 9:46 pm

As I found some lessons learning while coding and debugging my Python workbench, I would like to share my experience and collect some further advise, with the ultimate goal to improve the documentation (and make good use of the advice myself). So you may well assume that this topic is intermediate between the "Developers corner" and "Wiki", and you will excuse me for selecting "Developers" as I believe that the post has more relevance here for the initial discussion.

Also, Python workbenches debugging is strictly related also to Python workbenches reloading. So I'm summarizing here my understanding about that as well.

Let's start with debugging. Python debugging is actually described in two WIki pages:

However, much more information can be found searching in the Forum, but in the end (as these are Forum questions) it is not well structured and sometimes also misleading, as spread over many years and different FreeCAD versions. Also, most of it deals on how to debug Macros, not specifically Modules/Workbenches.

So here's my summary:

Method 1: Using pdb

This is my preferred, as it works out of the box (requires no installation of anything beyond the standard FreeCAD release), it is the same for Windows and Linux, and can debug pretty much everything Python.

Basic usage for pdb in FreeCAD is discussed in https://forum.freecadweb.org/viewtopic.php?f=10&t=28256

However, the suggestion is to insert the statement

Code: Select all

import pdb; pdb.set_trace()
in your code, instead of print statements. This is not quite right, as in general I do not want to change the code for debugging; and more so when dealing with workbenches, where I fall into the secondary problem on how to reload a workbench!

Instead, as we have the Python console available, from the console you just:
  • break into pdb with

    Code: Select all

    import pdb; pdb.set_trace()
  • set a breakpoint as below, where <yourpythonmodule> is the module name as you would import it (i.e. import <yourpythonmodule>)

    Code: Select all

    b <yourpythonmodule>:<yourlinenumber>
  • resume execution typing

    Code: Select all

    c
Now when FreeCAD hits the breakpoint, pdb will kick-in in the Python console, and you can inspect/change variables, arguments, execute Python statements, etc.

Another advantage of pdb is that it allows you post-mortem analysis. If you hit an exception, you can run from the Python console

Code: Select all

pdb.pm()
and you have access at the last backtrace, so you can dig into what happened.

Cons of this approach:
  • It is textual only (well, for me this is not necessarily a cons)
  • FreeCAD mess up a bit the prompt (pdb) when debugging on a breakpoint, the prompt appears sometimes in the report window, so it might be a bit confusing
Method 2: using winpdb

As the name implies, this is a GUI version of pdb (not a Windows-only version, mind you! win=GUI here). However, to make it work, you need some configuration, which is different for Windows and Linux, as of course you need to install winpdb, and the debug is done attaching winpdb to a remote target (is not embedded within FreeCAD; FreeCAD is the remote target to debug).

This is the method described by Werner in many posts (I believe this is his preferred?), see:

https://forum.freecadweb.org/viewtopic. ... 80&p=26446
https://forum.freecadweb.org/viewtopic.php?f=10&t=3884

and ultimately appearing in the Wiki in the post I already cited:

https://www.freecadweb.org/wiki/Debugging

Again all the discussion is mainly for debugging Macros, and the example in the Wiki page tells you to run a script. Actually this is not needed, again you can just insert a breakpoint in your code and work in FreeCAD until that part of the code is reached.

Linux version:
  • install winpdb from a terminal with

    Code: Select all

    sudo apt-get install winpdb
  • start winpdb. Go to menu File->Password" and set the password to e.g. "test"
  • from the Python console in FreeCAD type:

    Code: Select all

    import rpdb2
    rpdb2.start_embedded_debugger("test")
  • in winpdb, pause the execution (FreeCAD becomes unresponsive)
  • in winpdb, open your source file, and set the breakpoint
  • in winpdb, resume execution
Note that installing winpdb automatically installs rpdb2 in your Python environment.

Windows version:

The problem under Windows is that Python for FreeCAD is installed bundled with FreeCAD, and separately from any other Python installation you may have.
So when you need to install Python extensions, you need to do that in a way that FreeCAD sees it.
However, you don't need winpdb to be installed into FreeCAD; this application opens a socket for remote debugging, so you can launch it in any other way. In my case, as I have Python installed under Win, I just installed it from the DOS shell using:

Code: Select all

pip install winpdb-reborn
However, from within FreeCAD you need rpdb2. I looked around a bit in the Forum and found the discussion https://forum.freecadweb.org/viewtopic.php?t=16851 . So I gave it a try and it actually worked, at least in my case - the thread warns that this method may fail, I'm not sure here about the conditions when this could happen.

Code: Select all

import pip
pip.main(['install'] + ['winpdb']
One important remark: if you have FreeCAD installed in "Program Files" or equivalent directory that is under Admin supervisory right, this method of installation will fail. You MUST start FreeCAD with "Run as Administrator".

Now you are all set, and you can follow similar steps as in Linux case:
  • start winpdb. Go to menu File->Password" and set the password to e.g. "test"
  • from the Python console in FreeCAD type:

    Code: Select all

    import rpdb2
    rpdb2.start_embedded_debugger("test")
  • in winpdb, pause the execution (FreeCAD becomes unresponsive)
  • in winpdb, open your source file, and set the breakpoint
  • in winpdb, resume execution

The obvious advantage of winpdb is the availability of a GUI, so it is more user-friendly than pdb alone. However, I find it more tricky to start (as you need to work on two different windows, coordinating the actions) and the author of winpdb says

Cons of this approach:

Method 3: using Visual Studio

This is described in detail in the thread https://forum.freecadweb.org/viewtopic.php?f=22&t=28901

However, I admit I was not able to make it works, possibly because (for some reasons not relevant here) I have only VS2015 installed.

But besides that:

Cons of this approach:
  • Only Windows
  • Tied to Visual Studio, that is fully proprietary (I know that FreeCAD under Win compiles under VS, but whenever I can, I prefer open tools)
Method 4: using FreeCAD embedded Python debugger/editor

This would be the preferred method, but actually the debugger is not documented (or at least I could not find the relevant Wiki page), and I could find no way to make it work for debugging a workbench.
So your option here is creating a stub i.e. a Macro that calls the functions of your workbench that you need to debug, so that you can load the macro in the editor, set breakpoints and run it.

Cons of this approach:
  • Not really suited to debug workbenches, only macros
Reloading workbenches

Ok, so now you debugged your code, found the issue and fixed it. How can you reload your workbench without closing/re-opening FreeCAD?

There are two main posts that deal with the topic:

https://forum.freecadweb.org/viewtopic.php?t=320
https://forum.freecadweb.org/viewtopic. ... 5&p=296289

First of all a note: as far as my understanding and my experience goes, you cannot really reload a full workbench, as there are some functions that loaded by C/C++ code. In particular:
  • the Workbench description class is transfered to FreeCAD while start
  • FreeCADGui.addCommand('MyModule', MyModule())
To reload everything else, you can use in Python 2:

Code: Select all

import someModule
reload(someModule)
or in Python 3:

Code: Select all

import someModule
from importlib import reload
reload(someModule)
Now a word of caution. What can make this method fail is forgetting about the namespaces you are in. In particular, if your workbench definition file, say someModule.py, imports other files with

Code: Select all

from someTool import *
then you are actually importing the definitions in the namespace someModule.

So while in general you can import * and even reload it, with

Code: Select all

import someTool 
reload( someTool  )
from someTool  import *
this is not possible from the Python console as you are not in the correct namespace (someModule)

(note: hint available at last row of the text appearing when typing import this, but probably you all already know that).

What you can do (this was suggested by Microelly I believe) is to tweak your someModule to contain, instead of the plain from someTool import *, the full code above.

Then you just need to import someModule and then reload someModule and the trick is done.

I'm not sure about the performance impact of leaving the full reload code in someModule in a production release of the workbench, but at least you can leave it there until you are done with your testing.


I realize I made an extra-long post, sorry about that and thanks for reading so far. I would welcome everybody's comments, as soon as this is cleared up I will translate the post into the Wiki pages, ideally marking https://www.freecadweb.org/wiki/Python_ ... nvironment as old, while improving https://www.freecadweb.org/wiki/Debugging .
Of course debugging Macros, and not workbenches, is a subset of workbench debugging, and considerably easier.

Ciao!
Enrico
triplus
Posts: 8802
Joined: Mon Dec 12, 2011 4:45 pm

Re: Python workbenches debugging

Postby triplus » Mon Apr 01, 2019 10:39 pm

Thanks for sharing and for trying to improve the developer oriented documentation.
User avatar
Kunda1
Posts: 5949
Joined: Thu Jan 05, 2017 9:03 pm

Re: Python workbenches debugging

Postby Kunda1 » Tue Apr 02, 2019 2:21 pm

Awesome summary!
This needs to be part of FreeCAD #documentation because debugging can save a ton of time if you know how use it.

@kkremitzki recently turned me on to pdb and it's seriously awesome.

There are also cool refinements that people should know about, there is a dropin replacement for pdb called pdbplusplus (pdbpp) that has an awesome sticky mode. I learned about it from this fantastic pdb intro talk back from pycon 2015:
phpBB [video]


Watching the whole thing is well worth it.

Actually, I think it would be an amazing gift if we had a GSOC project that was dedicated to integrating a debugger (continuing @mumme's awesome work in @ediloren's mentioned thread)
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
User avatar
iogui
Posts: 94
Joined: Tue Mar 19, 2019 3:44 pm
Location: São Paulo
Contact:

Re: Python workbenches debugging

Postby iogui » Tue Apr 02, 2019 7:26 pm

I will post a really newbie question here as I'm new booth to FreeCAD macro's development and its internal api and to python programming. Is it possible to open a macro file or a code from a workspace from inside an IDE like intellij and debug it from there? (I use to do this kind of thing with nodejs, php on a browser, javascript on a browser, java as a remote connection and so on)
Be the change you want to see in the world. - Mahatma Gandhi
ediloren
Posts: 209
Joined: Wed May 08, 2013 9:23 pm
Location: Italy
Contact:

Re: Python workbenches debugging

Postby ediloren » Wed Apr 03, 2019 9:13 pm

@Kunda1 : thanks for the pointer to pdbpp, I did not know that. However I am not sure to which extent we can use it in FreeCAD, as it actually tries to repaint the screen (this is the whole purpose of the sticky mode), color the output etc. so I wonder how's that in the FreeCAD console. I am not testing it for the time being, as it replaces pdb.py, so I don't want to mess up the installation when I need to 'pip uninstall' (probably nothing bad will happen, but I am not ready for a test now). If anybody is giving that a shot, I'd welcome knowing the results (both for Windows as well as for Linux).

@iogui : I do not use intellij, but what you ask for is possible, provided you have the right workspace. Actually this is exactly method 3 in my list, where the workspace IDE is Visual Studio (which is quite powerful, to be frank - only I don't like to stick to proprietary closed-source software that may change the license later on; plus (or minus..) this is only for Windows). Other IDEs may be able to attach to a remote Python process for debugging, just like VS does, or using rpdb as winpdb does (which works also through an LAN connection), but I am not able at present to suggest you other options besides VS.

Enrico
User avatar
iogui
Posts: 94
Joined: Tue Mar 19, 2019 3:44 pm
Location: São Paulo
Contact:

Re: Python workbenches debugging

Postby iogui » Wed Apr 03, 2019 10:14 pm

ediloren wrote:
Wed Apr 03, 2019 9:13 pm
@iogui : I do not use intellij, but what you ask for is possible, provided you have the right workspace. Actually this is exactly method 3 in my list, where the workspace IDE is Visual Studio (which is quite powerful, to be frank - only I don't like to stick to proprietary closed-source software that may change the license later on; plus (or minus..) this is only for Windows). Other IDEs may be able to attach to a remote Python process for debugging, just like VS does, or using rpdb as winpdb does (which works also through an LAN connection), but I am not able at present to suggest you other options besides VS.
Well Intellij is quite powerful too and has a community version that is open-source indeed and it works nicely on windows, mac or linux. As I am a Linux only user I do prefer it over Visual Studio and I also own an Intellij commercial license. I have seen that Intellij has a remote debugging on python option but I have never tried it. I've made a small search in my environment to find their "pycharm-debug-egg" python file but I haven't found nothing. It is really nice to have an IDE editing and debugging tool as you develop something so it would be useful. Unfortunately, I think it will not work with intellij... sad... :(
Be the change you want to see in the world. - Mahatma Gandhi
hlg
Posts: 19
Joined: Fri Jul 12, 2019 10:11 am

Re: Python workbenches debugging

Postby hlg » Fri Jul 12, 2019 10:30 am

I successfully managed to use PyCharm for debugging a workbench script. Remote debugging is indeed not available in the Community Edition and that might be the reason why you do not find the egg.

However you don't need remote debugging as long as PyCharm and the FreeCAD application are running on the same machine. You can just attach the remote debugger to a running FreeCAD instance and off you go. In PyCharm this can be found under Run / Attach To Process or with Ctrl-Alt-F5. You are presented with a list of processes to select from. You can configure the processes to appear in this list under File / Settings / Build, Execution, Deployment / Python Debugger. There you can specify a string. All the processes containing this string in their name will appear in the process list to select from later on. The initial search string is "python" such that FreeCAD will not appear in the list. Changing it to "FreeCAD" should work. When in doubt, inspect the list of running processes with the tools of your OS and also note the process ID.

I don't know whether and how you are able to access this functionality when using the IntelliJ plugin instead of the standalone PyCharm application, but it might be worth having a look around the menu.
User avatar
iogui
Posts: 94
Joined: Tue Mar 19, 2019 3:44 pm
Location: São Paulo
Contact:

Re: Python workbenches debugging

Postby iogui » Fri Jul 12, 2019 12:58 pm

hlg wrote:
Fri Jul 12, 2019 10:30 am
Remote debugging is indeed not available in the Community Edition and that might be the reason why you do not find the egg.
No it isn't. I'm not using the community edition. I'm using the ultimate edition of Intellij that has indeed the python plugin.
hlg wrote:
Fri Jul 12, 2019 10:30 am
You can just attach the remote debugger to a running FreeCAD instance and off you go.
Ok. I don't know if it will work, but I will give it a try later when I have the time to do it.
Be the change you want to see in the world. - Mahatma Gandhi