FEM Workbench Testing - GSoC Proposal

[Kunda1]

@m42kus thank you so much for getting the ball rolling on this. It's sounds a PITA to have 2 different frameworks for documenting FC (doxygen and sphinx for c++ and python respectively) but honestly it makes sense if it helps us get around the rut that we've been in stalling on having robust documentation for FC. I'm not sure what the technical future pitfalls may be in pursuing both frameworks at the same time. I will say though that I love the examples that you've created and am actually quite excited. So thank you!

[m42kus]

@m42kus thank you so much for getting the ball rolling on this. It's sounds a PITA to have 2 different frameworks for documenting FC (doxygen and sphinx for c++ and python respectively) but honestly it makes sense if it helps us get around the rut that we've been in stalling on having robust documentation for FC. I'm not sure what the technical future pitfalls may be in pursuing both frameworks at the same time. I will say though that I love the examples that you've created and am actually quite excited. So thank you!

I think a big problem (from the view of documentation) is that in FreeCAD the C++ and Python Parts are both huge. This means that we need a good efficient solution for both languages. In the C++ world people are used to using Doxygen and in the python world Sphinx is king. Also in FreeCAD it is possible to archive most programming tasks in both languages which is great but easily leads do redundant and confusing documentation. We also must not forget to consider the usability of the documentation in addition of issues with writing and maintaining it. It would be sad if we invest many hours into writing good documentation that people don't use because the software doesn't work well for the programming language (indexing, search and so on).

[Kunda1]

Also in FreeCAD it is possible to archive most programming tasks in both languages which is great but easily leads do redundant and confusing documentation.

So true, lets really think of a strategy for this!

We also must not forget to consider the usability of the documentation in addition of issues with writing and maintaining it. It would be sad if we invest many hours into writing good documentation that people don't use because the software doesn't work well for the programming language (indexing, search and so on).

Yea, sphinx makes more sense for python. It does use .rst right?

[vocx]

...
I think that the best approach would be to use doxygen for C++ and sphinx for python. I think that is done to a certain extend already but not very consistent. Honestly I am a little confused about what the current state of documentation is in FreeCAD . What do you think about that?

I just finished editing some relevant pages in the wiki.

• CompileOnUnix: I edited this page because I wanted to have clear instructions to compile FreeCAD myself. I mostly reviewed the packages needed for Debian/Ubuntu, as I use Ubuntu myself, and I tried to clarify the steps that you must follow in general in Linux for compilation.
• Source documentation: this page explains how to create the documented sources; it follows the same process as compiling FreeCAD in general, that is, cmake and then make DevDoc, instead of just make. FreeCAD produces its online documentation in a way that I still don't understand very well. You don't run the doxygen executable directly, but with the cmake+make process mentioned before, so Doxygen uses the appropriate paths, variables, and configuration (like HTML headers and footers). The specifics are still a bit of a mystery, as I think the calls to cmake create temporary files (CMakeLists?), or include other files, with the right configurations.
• Doxygen: this page was short so I expanded it with many details explaining how to use Doxygen in general, and for C++ and Python files that are used in FreeCAD.

Normally, Doxygen cannot read very well the docstrings of Python source code. However, the program doxypypy can be used to convert the docstrings to a format that Doxygen can understand better. So, when running Doxygen, doxypypy can be used as a filter, through which all Python files go, in order to pre-process the comments, and produce better documentation for Python files. This is explained in the same Doxygen page. It is a long document, but I provide examples, so hopefully it isn't too hard to follow. At least for now I think this is the best solution to keep using Doxygen with both C++ and Python.

In the last section of the document I mention Sphinx. I wanted to do something else so I didn't investigate further this part. It seems that, just like Doxygen is ideal for C++, Sphinx is ideal for Python. While Doxygen can process Python files with the help of doxypypy, it seems Sphinx can do the same but in the opposite direction. Sphinx can naturally process Python files, and it can document C++ files with the use of an intermediate program, like Breathe, which itself uses Doxygen. This deserves more investigation.

Clear, Functional C++ Documentation with Sphinx + Breathe + Doxygen + CMake

I'm not entirely sure what the "Domains" are in Sphinx, but the latest documentation seems to have a C++ Domain in Sphinx 3.0, which is still in development. I don't know if this means that in the future Sphinx will be able to parse C++ natively, but it sounds promising.

Sphinx The C++ Domain

So I think there are two possibilities:

Code: Select all

C++ and Python ----> Doxygen ----> doxypypy ----> HTML

C++ and Python ----> Sphinx ----> (natively, Breathe/Doxygen?) ----> HTML

Since I realize that you've made some progress on testing Sphinx, I would strongly suggest that you document the process in a Sphinx wiki page. I wanted to do this myself, but as I said, I got sidetracked doing something else. I want to refactor the Python code of the Draft Workbench (which is entirely in Python) precisely to test the Python documentation generation, as if the sources aren't properly commented, automatic systems like Doxygen, doxypypy, and Sphinx don't work quite well.

Personally, I think it should be possible to document everything with Sphinx. Having a nice programming interface like that of SciPy, Numpy, and Matplolib would be amazing. Or even the documentation of Blender; I believe the internals are in C++ but all public functions are in Python.

[m42kus]

So true, lets really think of a strategy for this!

Yes, a consisten strategy for all of FreeCAD would do wonders. I'd gladly help to work out a good solution, is there a good place to discuss this already or do you want to make a new thread somewhere? For me the first step would be to write down the current state of the documentation. As far as I know there are a few different systems in use in different parts of FreeCAD for all kinds of purposes.

Yea, sphinx makes more sense for python. It does use .rst right?

Yes it is essentially some kind of structured rst with a way of cross referencing things and a lot of plugins (e.g. for source code documentation extraction) and themes.

[m42kus]

Hi, thanks for this detailed post, I definitely should go to bed now but I'll read through all of your stuff tomorrow

[vocx]

Hi, thanks for this detailed post, I definitely should go to bed now but I'll read through all of your stuff tomorrow

Also read this post.

...
How is the documentation style currently in FreeCAD? Till now I only looked a bit into the FEM workbench, which does not give the whole picture. I reckon that much of the code is documented in doxygen style (or something similar) that can be parsed in a nice format (like html or pdf in case of doxygen).

Allow me to be blunt on this. The documentation style of FreeCAD is a mess! Simply put. The original developers didn't exactly care too much about documenting everything in an accessible way, and I believe this is an issue when it comes to making it easy for others to join in extending FreeCAD. Simply put, there isn't enough description of the core components of FreeCAD. As long as nobody touches it, and it works, then people will just build workbenches on top.

...

The reason Yorik basically says "do whatever you want" is precisely because there is no overall standard guideline for the documentation. You can do whatever you want! It's a mess! This is not a knock on Yorik. He's the one keeping FreeCAD thriving, but it's basically the reality. At this point in time FreeCAD is still a growing project so it seems most people favor a quick development pace, and don't worry too much with boring stuff; you guessed it, documentation!

I personally suggest you to use Python guidelines (pdoc, PEP8, look at the documentation of numpy, matplotlib, etc.) to document Python code
...
(Hint: I wished you'd make a second proposal to GSoC to organize FreeCAD's documentation, and refactor modules to a more Pythonic standard.)

[Kunda1]

lets split off this tangent in to another thread. Is there one already or can someone suggest a name?

[yorik]

About sphinx/ doxygen:

Some years ago I tried to use sphinx to generate the python API docs because, indeed, they look nicer. However, they fall short at the same place as Doxygen: They are unable to document C++-generated modules and classes. So at that time we made some "fake" modules (that you can still find in TemplatePyMod). that would allow sphinx to crawl through them. But obviosuly it was a PITA to maintain and got quickly obsolete.

Also, maintaining two systems is also annoying, once doxygen is able to parse python code too. So I myself basically gave up of sphinx. The code is still there but it's not used anywhere anymore.

At some point, I think we should build some tools that automatically build fake python files from selected FreeCAD modules.. It would actually not be too hard, python has many tools to do that.

The real problem with doxygen is that the html it produces is rather ugly. But there are several other possibilities that we could try, that I saw on the net, such as generating markdown files...

[m42kus]

I just finished editing some relevant pages in the wiki.

Very nice work. Especially the compilation page would have been helpful for me when I started using FreeCAD One thing I noticed is that you used triple single quotes for python docstrings. I would suggest changing that to triple double quotes because thats what PEP 257 -- Docstring Convention says: "For consistency, always use """triple double quotes""" around docstrings. Use r"""raw triple double quotes""" if you use any backslashes in your docstrings. For Unicode docstrings, use u"""Unicode triple-quoted strings""" ". I think most projects use the double quotes for this reason.

Clear, Functional C++ Documentation with Sphinx + Breathe + Doxygen + CMake

Good Article, I especially agree with the author on the following paragraph:

On a more fundamental level, Doxygen’s style of documentation is listing out all the API entities along with their associated comments in a more digestible, searchable manner. It’s essentially paraphrasing the header files, to take a phrase from Robert Ramey[1]; embedding things like rationale, examples, notes, or swapping out auto-generated output for hand-written is not very well supported. In Sphinx however, the finer-grained control gives you the ability to write documentation which is truly geared towards getting people to learn and understand your library. -- Simon Brand

I'm not entirely sure what the "Domains" are in Sphinx, but the latest documentation seems to have a C++ Domain in Sphinx 3.0, which is still in development. I don't know if this means that in the future Sphinx will be able to parse C++ natively, but it sounds promising.

Domains are just namespaces for roles and directives. E.g. with domains you can have a class directive named "class" for Python (py:class) and C++ (cpp:class). So unfortunately this doesn't have anything to do with parsing. A parser would be a plugin like "autodoc" for python. The "Breath" project is actually a extension just like "autodoc" but it depends on the xml output of doxygen instead of the source directly. So the next step would be to replace it with a plugin that parses the C++ source itself but even that it would define new directives and roles for itself. As far as I know the C++ domain is for manually writing API documentation just like the py domain.

Personally, I think it should be possible to document everything with Sphinx. Having a nice programming interface like that of SciPy, Numpy, and Matplolib would be amazing. Or even the documentation of Blender; I believe the internals are in C++ but all public functions are in Python.

I would not generally say that C++ is internal and Python is public (except for scripting). The core of FreeCAD exposes C++ interfaces that are used by the workbenches and should be documented for the same reason their Python equivalents should be documented. I'd even argue that the internals of the FreeCAD core should also be documented to enable other developers to understand and enhance the core.

So I think there are two possibilities:

Code: Select all

C++ and Python ----> Doxygen ----> doxypypy ----> HTML

C++ and Python ----> Sphinx ----> (natively, Breathe/Doxygen?) ----> HTML

As I'm not aware of any other projects supporting python and doxygen I would also say that these are very good options. I still think that using both sphinx and doxygen is an option we could think about if sphinx proves to be insufficient for C++ or there is too much opposition to the idea from the community.