This is now a thread about documentation

[David_D]

Hey, I'm making an attempt to expand Arch's doxygen documentation, and I noticed that all of the class names are prefixed with an underscore.

This will result in none of the commands having documentation generated for them, which is a major gap in the documentation. Underscore prefixed variables/classes/functions are excluded per the web documentation doxygen config file's EXCLUDE_SYMBOLS variable. (See src/Doc/BuildWebDoc.cfg.in)

I wanted to know why this was done in the Arch workbench. Is it purely a stylistic choice? Would Arch's maintainers be willing to part with the underscores? Removing the underscore from the doxygen configuration file may or may not break something else in the documentation, I have yet to figure out why it's in there.

[vocx]

...
I wanted to know why this was done in the Arch workbench. Is it purely a stylistic choice? Would Arch's maintainers be willing to part with the underscores? ...

Using the underscore was a mistake, of course. The main author of the workbench probably didn't anticipate that it would cause problems in documentation when he initially started the workbench back in the distant past.

We could of course change the name of the classes, but this would break the loading of older files. When a file is loaded, each object is recreated by loading its class. So, if you have an Arch_Wall, it will try to reload ArchWall._Wall. If it cannot be found, the geometry will load, but as a static object, no longer as a fully parametric object.

So, if you want to rename the classes, you need to set up an alias so that the file loads the object it is looking for.

Code: Select all

# ArchWall.py

class Wall
    ...

_Wall = Wall

Once that is done, then you need to be sure the object can be opened and works correctly. This is the reason I suggest creating unit tests.

I suggest writing a test file that has all Arch objects. This is a sample file that you create and can open with the current master version. Then as you make modifications to Arch, you need to try to open the sample file. If all Arch objects open correctly, it means the migration from the old class to the new class was successful, and there won't be broken files.

See Draft/drafttests/draft_test_objects.py for a sample file for Draft. We would like to have something similar for Arch.

Another possibility is to write code that upgrades an object from an older class to a newer class. It would basically recreate the object, using the new class name. It would iterate over all the older object's properties, and assign them in the new object.

Code: Select all

# old object is loaded
old_obj = App.ActiveDocument.OldWall

# New object is created
obj = App.ActiveDocument.addObject("Part::FeaturePython", "Wall")
Wall(obj)  # New class

# iteration over all properties of old_obj
obj.PropertySomething = old_obj.PropertySomething
obj.Shape = old_obj.Shape

Check all properties inside obj.PropertiesList. The same process for the view provider classes.

[David_D]

That is quite the pickle. I'll look into the unit tests. I've also noticed a few other strange things about how the documentation is generated so this may be an appropriate time to fix several things.

[vocx]

... I've also noticed a few other strange things about how the documentation is generated so this may be an appropriate time to fix several things.

That sounds good. See source documentation and Doxygen. Doxygen is pretty powerful but it's a bit hard to get everything right. It particular it's hard to make live tests with FreeCAD because even small changes to the code results in big recompilation times of the documentation.

I had one issue, maybe you can look into it. Once you generate the Doxygen documentation, in the HTML pages there are some arrows that expand subtopics, for example, C++ classes show which other C++ classes they inherit. In my computer these arrows don't work. It works in the online API pages, but not locally.

And also, I think a task that should receive priority is investigating a way to handle both C++ and Python in the best way possible to create good documentation. Since FreeCAD is a combination of C++ code with Python, it's actually a bit hard to manage both languages in a simple way.

One way of handling things is using Doxygen completely. It would parse C++ without issues, but then it would use the program "doxypypy" to process Python sources, so that the generated documentation looks better. I investigated this briefly and it seems to work, but I didn't test on a wide scale, just in my computer. See Doxygen.

The other path is using Sphinx to handle everything. It would parse the Python modules natively, and when it comes to the C++ code, you'd use an intermediate program called "Breathe" to call Doxygen in order to parse the C++ code.

If you can find clever ways on how other projects handle this combination C++-Python, that would be great.

Fixing Doxygen Module descriptions

[David_D]

And also, I think a task that should receive priority is investigating a way to handle both C++ and Python in the best way possible to create good documentation. Since FreeCAD is a combination of C++ code with Python, it's actually a bit hard to manage both languages in a simple way.

I've found two solutions to documenting mixed python and C++.

1) Stick with doxygen

So after some experimentation, I've discovered the most elegant solution I could find, which isn't very elegant. An undocumented feature of doxygen is that if you add an exclamation mark at the start of a docstring, doxygen will render the docstring appropriately. By appropriately, I mean showing line breaks, and rendering markup formatting.



Upsides: We get to keep using doxygen without any changes.

Downsides: All the docstrings have an exclamation mark at the start of them, which might confuse people using help(), or an autocomplete. Also, we're stuck with all the current things we dislike about doxygen.

2) Move to sphinx

I believe this would produce higher quality documentation overall. I've never used sphinx in this particular use case, but personally, I find it much friendlier as both a user and a developer. I think this is the way to go.

Upsides:
- Documentation generated quickly and easily. Lowers the bar to entry for those who want to document things.
- Greater flexibility in how the documentation is structured, instead of sticking to doxygen's rather c++ oriented structure. Currently, I find the api doc's structure rather arbitrary. I think we can make it much more user friendly.
- Good looking.
- I could just implement this for Arch, and we can see how that turns out, and expand it if it's acceptable.

Downsides:
- This turns this into a larger project.
- Sphinx being more flexible could down the road, end up causing this documentation to start edging into the wiki's territory.

I'm happy to pursue either option, on your recommendation.

[vocx]

...
I'm happy to pursue either option, on your recommendation.

Since we have never truly explored Sphinx, I suggest looking into this more. A long time ago the developers did create some tentative documents, but the efforts were not pursued further. You can see the few files tested in src/Doc/sphinx.

[wandererfan]

Hey, I'm making an attempt to expand Arch's doxygen documentation, and I noticed that all of the class names are prefixed with an underscore.

In Python, leading underscore is a convention saying "this is a private function, don't use it directly". But it is only a convention. https://dbader.org/blog/meaning-of-unde ... -in-python

In C++ leading underscore names are used to prevent name collisions between "implementation" names and application names. IIRC, application programmers are not supposed to use leading underscores.

[David_D]

Since we have never truly explored Sphinx, I suggest looking into this more. A long time ago the developers did create some tentative documents, but the efforts were not pursued further. You can see the few files tested in src/Doc/sphinx.

Alrighty, I'll see if I can produce something that works better than the current system, and report back when I've got a better idea of it's limitations.

[David_D]

Since we have never truly explored Sphinx, I suggest looking into this more. A long time ago the developers did create some tentative documents, but the efforts were not pursued further. You can see the few files tested in src/Doc/sphinx.

Ok, I'm pretty certain that sphinx is the way to go.

I've produced some sphinx documentation that can happily extract stuff from both python and c++, as you can see here:



It's also got some scripts that will automatically regenerate the documentation when you make changes to things. The key point here is that these scripts will only regenerate the specific parts of the documentation that need to be changed, to keep down the doc's build time. it is currently far faster than doxygen.

For documenting c++, you keep using doxygen documentation syntax. For documenting python, you use the google docstring styleguide, as per here, with examples here. This keeps the doc strings human readable, so they're suitable for reading straight from the code, or from the FreeCAD python interpreter's autosuggestions. It also allows sphinx to extract syntax, as you'd expect from auto-generated api documentation. I think this is a very elegant solution, that will not effect anyone who wants to keep using pure doxygen.

Sphinx gives us the control to choose what's included in the API documentation and not, so we don't end up with a voluminous tome of auto-generated documentation without any useful content.

The downside is that it's a little more complex than I'd like, and that as far as I can tell, it'll only work on linux and mac systems. It works very nicely once you've got all the auto generation terminals open though.

You can see the repo here: https://github.com/DavidDaish/freecad_sphinx_docs

As a proof of concept for this method of api documentation, I'll document Arch, and come back when there's something pretty and neat to look at.

[vocx]

Ok, I'm pretty certain that sphinx is the way to go.

I've produced some sphinx documentation that can happily extract stuff from both python and c++, as you can see here:

About C++, it's not only about documenting a single class or function, but rather automatically generating the relationships between them and creating the diagrams that show the relationship between the classes. That is very helpful when using Doxygen.

For documenting c++, you keep using doxygen documentation syntax. For documenting python, you use the google docstring styleguide,...

Can you check if we can use Numpy style for the docstrings? I'm not against Google style on principle, however, I think since FreeCAD is a technical program that deals with mathematics, I feel that it's more appropriate to use the same style as Numpy, SciPy, and Matplotlib. This is what I've been doing since I started adding docstrings to various files. See for example, WorkingPlane.py.

I normally use Spyder to write Python code. Spyder is an IDE primarily meant for scientific use, so by default it parses docstrings in Numpy style and displays them nicely. I actually encourage people to use Spyder every time I can, but it seems to me not many people use it, probably only those who use Numpy and Matplotlib regularly.

Sphinx gives us the control to choose what's included in the API documentation and not, so we don't end up with a voluminous tome of auto-generated documentation without any useful content.

Ideally we would specify the Python workbenches to run through Sphinx, and the rest of the C++ workbenches to run through Doxygen. The problem is you can also include Python commands in C++ workbenches without much problem. For example, see src/Mod/PartDesign/WizardShaft, FeatureHole, InvoluteGearFeature, etc., these are Gui Commands and classes to create scripted objects in Python. So, how to tie the two worlds would be interesting.

The downside is that it's a little more complex than I'd like, and that as far as I can tell, it'll only work on linux and mac systems. It works very nicely once you've got all the auto generation terminals open though.

Ideally we can run the documentation generation script through "make DevDoc", just like we do right now. Or how do you intend to generate it? Also, why would it only work in Linux and MacOS?

Do you mean you are running some scripts with Bash? Could you write an equivalent .bat file, or powershell script to run the right commands in Windows? This should be set through the initial cmake. If we are on Windows cmake should set the proper script to run when doing "make DevDoc".

Or if you want something more general, we could use Python itself as a scripting language to run the necessary programs. We can use subprocess.Popen to run external programs with it.

Or do you mean Sphinx itself doesn't quite work on Windows?