Testing Tools Survey (FEM Workbench Testing - GSoC)

[abdullah]

There are two reasons why are I posting:
1. I would like to have more verbosity when unit tests fail. Currently the why is rather hidden. See for example:
https://travis-ci.org/FreeCAD/FreeCAD/b ... tification

2. If you are going to work both unit tests and CI, in particular Travis, this may be relevant for you.

The thread were I log my tests is:
https://forum.freecadweb.org/viewtopic. ... &start=130

Hi, this will definitely be relevant for me at some point. If I understood correctly the status right now is that all tests run successfully except a few tests of the Path Workbench because they make use of the Gui module which can't be used in the windows virtual environment?

Yes, that is the situation.

However, take with a grain of salt what I wrote, because in the meantime I discovered the following:

git-bash is not a "proper" windows console. When Python is used therein (apparently a Python compiled with ncurses), the output is mangled: incomplete and out of the real order. To solve this, one can use winpty, which is bundled with git-bash. However, winpty requires, without options, a proper windows console, which git-bash is not, and it will through an error "stdin is not a tty". Nevertheless, when using winpty with the flag "-Xallow-non-tty", one can by-pass the check of whether it is a proper tty. However, it generates problems when it tries to rewind, which you can do with a tty, but not with a file. You can the use the winpty flag, "-Xplain" for this. Those flags are not documented and can only be found in the code of winpty. In any case, when winpty is executed like this, the screen output is beautiful (well just correct, but after having seen it wrong, it looks beautiful to me), the exit code (return code) of FreeCADCmd is not passed correctly to git-bash, so it detects it as "return 0 - success" even if tests fail. The solution I found is to use tee to put the same content that goes to the screen to a file, then grep is used against this file to search for FAILED and make the build fail if found (I am currently missing the -q flag to grep to make it quiet and avoid duplicating the FAILED line to the screen).

So, all-in-all. The output is ok now:
https://travis-ci.org/FreeCAD/FreeCAD/jobs/537777414

There some FEM and Path test that fail, but now we know why!! (yes, I know it sounds stupid, but some days ago I felt like the guys in the 50s searching for actual bugs in the relays):

Code: Select all

======================================================================
ERROR: test_pyimport_all_FEM_modules (femtest.testcommon.TestFemCommon)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\travis\build\FreeCAD\FreeCAD\build\Mod\Fem\femtest\testcommon.p
y", line 113, in test_pyimport_all_FEM_modules
    __import__('{0}'.format(mod))
  File "C:\Users\travis\build\FreeCAD\FreeCAD\build\Mod\Fem\feminout\importFenic
sMesh.py", line 34, in <module>
    import FreeCADGui
ImportError: DLL load failed: The specified module could not be found.
======================================================================
ERROR: TestPathApp (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: TestPathApp
Traceback (most recent call last):
  File "C:\Users\travis\build\FreeCAD\FreeCAD\build\bin\lib\unittest\loader.py",
 line 153, in loadTestsFromName
    module = __import__(module_name)
  File "C:\Users\travis\build\FreeCAD\FreeCAD\build\Mod\Path\TestPathApp.py", li
ne 33, in <module>
    from PathTests.TestPathDepthParams        import depthTestCases
  File "C:\Users\travis\build\FreeCAD\FreeCAD\build\Mod\Path\PathTests\TestPathD
epthParams.py", line 25, in <module>
    import PathScripts.PathUtils as PU
  File "C:\Users\travis\build\FreeCAD\FreeCAD\build\Mod\Path\PathScripts\PathUti
ls.py", line 26, in <module>
    import FreeCADGui
ImportError: DLL load failed: The specified module could not be found.

With this output the experts of each workbench will be able to fix it.

So maybe it is not so relevant for you after all.

[PaulG]

Thats a very ambitious system. I imagine that it took quite a while to reach it's current state.

Yes, the R system was custom built over many years, and a lot of it well before the tools that are available now.

I actually have a few questions about your project. What tool chain do you imagine? I couldn't work out how you use Travis CI in your git repo for testing? Do you use a unit test framework, or is it a collection of scripts that are executed?

It uses Sphinx, so the example code chunks are mixed with text in Sphinx rst files. (The intro.rst file at https://github.com/pdgilbert/EgFreeCAD/ ... ter/source will give you an idea of how it works.) When the document is built the code chunks are run, so the build fails if the code chunks do not run. Typically the Sphinx build just makes sure it runs, it doesn't make sure it actual gives proper results. But the tests I've added in the code chunks give some certainty that proper results are produced. I would not call this unit testing. It is not quite system testing but the chucks can be much more substantial than what would normally be called units. The travis.yml file is somewhat standard, I'm not an expert in travis by any means. It just installs freecad and Sphinx and does' make doctest' and 'make htlm' for the Sphinx document. It took me awhile to figure out how to push the html built from Sphinx back to gh-pages.

The main problems I see at the moment are:

- I can't run 0.19-pre because the freecad-daily PPA does not yet have systems that are available on Travis. If the daily worked then a cron job could trigger rebuilds so there would be a regular indication of what just got broken in the daily. (At least when the sphinx document is comprehensive and stable.)

-Travis does not yet have support for systems sufficiently new to uses freecad and python3.

-My examples need to be re-organized into sections (chapter?) so that success/failure can be established for different parts more easily. For example, if you were to build FEM WB tests you would like to see the results for that on Travis and on my web pages separate from the success or failure of some other workbench tests.

-My understanding of freecad needs improvement. I struggle a lot with understand if problems are my own or freecad bugs.

[m42kus]

Wow thanks for the detailed responses Paul and abdullah Those are both topics I'm sure I have to spend some time with in the future. Please let me come back to you in the next weeks. Right now I'm quite busy getting the GSoC project started. I still have to sort out much basic stuff before I can concern myself with travis and user contributed stuff

I just implemented a very shallow copy of basic FreeCAD runtime components which I will use to develop a prototype of the testing framework. If you are interested I posted something about it in the GSoC thread: https://forum.freecadweb.org/viewtopic. ... 26#p310926

Deciding on a C++ Unit Test Library is also more that overdue. I will post a survey much like the one for python later today

[abdullah]

Deciding on a C++ Unit Test Library is also more that overdue. I will post a survey much like the one for python later today

Most probably you already know it, the current tests rely on unittest from Python, which, as I read it is some kind of Python version of JUnit:
https://docs.python.org/2/library/unittest.html

[m42kus]

Not quite today but almost To move on we need to decide on a C++ unit test library. As far as I know FreeCAD doesn't use any kind of C++ unit tests yet. That of course means that it also doesn't use a unit test library for C++. Also there is no standard unit test library that ships with the standard library (like the unittest module for python). This gives us the freedom of freely choosing a suitable library.

The most important property in my opinion is it's popularity and how familiar it feels to us developers. Of course ease of use and its feature set are also important. For consistency and to minimize confusion the library should also feel similar to the python unittest module.

To fulfill all those points the unit test library must be a xUnit test framework. There are quite many xUnit test libraries out there. One of the most popular libraries out there is Google Test (https://github.com/google/googletest). In my opinion if fulfills all of the above points very well. It is also quite easy to to get started with Google Test as there is not that much you need to know if you only want to write a few tests. A test looks like the following:

Code: Select all

    
TEST(FactorialTest, HandlesPositiveInput)
{
      EXPECT_EQ(Factorial(1), 1);
      EXPECT_EQ(Factorial(2), 2);
      EXPECT_EQ(Factorial(3), 6);
      EXPECT_EQ(Factorial(8), 40320);
}

Tests register themselfs automatically (they don't need to be listed anywhere else). That means that only the above code is needed to add a new test. In addition to many other test macros besides `EXPECT_EQ` there are also more advanced features like common setup and teardown methods and much more.

Alternatives include the very similar Boost.Test library and the more modern catch2 (https://github.com/catchorg/Catch2) library. Both also support automatic test registration. I haven't used Boost.Test yet but it is supposed to be very similar to Google Test though most find Google Test a little easier to use. catch2 is an excellent library in my opinion but it may be a little too exotic for our purpose.

If it were up to me I would choose Google Test, but what do you guys have to say about the topic?

[kkremitzki]

+1 to adding C++ tests via Google Test. Re: pytest, I've often heard that it's nicer to use as a test runner even if you don't actually make tests with it (with which I've heard there are issues, but I'm not aware of what they are in detail.)

[yorik]

Not a big fan of having anything named "google" in FreeCAD, but, it's proper, bona-fide BSD3-licensed FOSS, so I guess there is no harm there... I'd say, try it!

[m42kus]

Not a big fan of having anything named "google" in FreeCAD, but, it's proper, bona-fide BSD3-licensed FOSS, so I guess there is no harm there... I'd say, try it!

Yes it's from google and they don't especially try to hide that fact xD. But I agree that it is a proper FOSS Project. It's also a small enough project that it can be maintained by it's community and is not depended solely on google.
Nice attitude I'm going to write some test cases using Google Test this week.

[apeltauer]

I can offer my help with gtest. My complete workbench is tested with gtest. Plus a great benefit is, I use gtest also for my code coverage check with OpenCppCoverage (https://github.com/OpenCppCoverage/OpenCppCoverage)

[m42kus]

I can offer my help with gtest. My complete workbench is tested with gtest. Plus a great benefit is, I use gtest also for my code coverage check with OpenCppCoverage (https://github.com/OpenCppCoverage/OpenCppCoverage)

Hi, very nice! Is your workbench open source? I haven't come across OpenCppCoverage yet. Do you know if it works on linux?