Testing Tools Survey (FEM Workbench Testing - GSoC)

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
m42kus
Posts: 159
Joined: Tue Sep 02, 2014 9:41 am

Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby m42kus » Sun May 19, 2019 2:47 pm

Hi everyone,

as most of you probably know I have been given the opportunity to work a lot on the stability of FreeCADs FEM module during the summer because of the Google Summer of Code initiative. My project is to work out a testing approach for the Fem module. One important part of any testing approach is a sufficient amount of automation to reduce the amount of repetitive work and the response time in which the developer gets his/her test results.

Automation is only possible with the help of tools. Which tools to use is a delicate question in a community as big as we are. Because of that I'm starting my project with this survey and invite every developer of FreeCAD to comment on my proposals, post his or hers experiences with the following tools, suggest additional ways to obtain a even higher degree of automation and everything else you guys think might help :)

Before I start the survey I'd like to say that although I'm only targeting the FEM module it is likely that at least some ideas are later applied to the whole of FreeCAD. Because of that I'm posting this here and I'm very interested to hear what developers of other workbenches have to say. Now without further ado lets continue with the actual questions we must ask ourselves. There are a few kinds of tool useful for any kind of automated testing. I've divided those into the following categories:

- Unit Test Frameworks for Python and C++
- Code Coverage Measurement for Python and C++
- Automated Test Execution (already done via Travis CI... I think)
- Automated Code Coverage Report

Each of those categories deserve a little time to be discussed. For that reason I won't post all at once but split it up into multiple posts. The next post will start with the python unit test framework.

regards,
Markus
My GSoC Devlog: ceeli.github.io/gsoc
m42kus
Posts: 159
Joined: Tue Sep 02, 2014 9:41 am

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby m42kus » Sun May 19, 2019 3:02 pm

Sooo, lets get started with the python unit test tool. Currently FreeCAD uses the included unittest module of python. I'd like to try using pytest (pytest.org) instead. pytest is a external python module that can be installed via pip and most system package manager. In my opinion pytest has a number of advantages over the standard unittest module:
  • easier to use for beginners: classes are not required; uses standard operators like == instead of assert methods like self.assertEqual(...)
  • fixtures: objects that are needed by test function can be passed as arguments and are automatically created when the test for each test that specifies them. For example if a test function needs a temporary file one can just write

    Code: Select all

    def test_something(tempfile)
    If there exists a correctly implemented tempfile fixture a temporary file will be created for each test as it needs it. The fixture feature is extendable which means we could use it to create geometry objects, mesh objects or any other object that might be needed for tests. Those fixtures are reusable. A extensive library of FreeCAD fixtures would greatly simplify and speed up test creation.
  • better tool support via plugins (e.g. automatic test converage report)
The disadvantage is of course that it isn't included in a default python installation. But one has to keep in mind that pytest will only be needed by developer and not by the end users. What do you guys think about pytest? It's of course something new and and an additional dependency but is has some pretty amazing features is very cool to use.
My GSoC Devlog: ceeli.github.io/gsoc
User avatar
yorik
Site Admin
Posts: 11552
Joined: Tue Feb 17, 2009 9:16 pm
Location: São Paulo, Brazil
Contact:

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby yorik » Mon May 20, 2019 2:36 pm

Looks nice, but switching to it would require to rewrite the entire test suite of FreeCAD, which is a problem :D
Unless we keep both, but that would probably be more confusing than helping.

But I'm just raising counterarguments here... Not really against it otherwise.
chrisb
Posts: 18859
Joined: Tue Mar 17, 2015 9:14 am

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby chrisb » Mon May 20, 2019 3:43 pm

To be honest, I'm not a big fan of additional modules. Even if it works now, we don't know what the future brings.

The argument with the classes does not count very much to me, because - as you say later - this tool is used by developers, and they probably know how to use classes.
I cannot say really much about the second point, because I don't understand the consequences in depth.
The third point would bring yet more dependencies, so following my argument from above I'm not a big fan of this either.

To be not too discouraging: How about making creation of tests more comfortable by giving developers a smart interface that can create all the stuff needed in the background.
This may be somehow reinventing the wheel, but we could customize this to our special needs and - again - as it is used by developers it doesn't have to be completely idiot proof.
User avatar
sgrogan
Posts: 5419
Joined: Wed Oct 22, 2014 5:02 pm

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby sgrogan » Mon May 20, 2019 8:50 pm

chrisb wrote:
Mon May 20, 2019 3:43 pm
To be not too discouraging: How about making creation of tests more comfortable by giving developers a smart interface that can create all the stuff needed in the background.
This may be somehow reinventing the wheel, but we could customize this to our special needs and - again - as it is used by developers it doesn't have to be completely idiot proof.
pytest seems an easy package to maintain (It's pure python), and readily available in Ubuntu.
A quick look https://docs.pytest.org/en/latest/ says it runs unittest out of the box.
Keep in mind, that we have a good group of "pseudo-developers" that compile themselves and run these tests from the Test-framework WB. We also sometimes ask users to run these tests to trouble shoot their installation.

In general I agree with chrisb about adding dependencies, but if we get code coverage reports it may be worth the cost.

Interesting project, I wish you success.
abdullah
Posts: 3174
Joined: Sun May 04, 2014 3:16 pm

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby abdullah » Wed May 22, 2019 12:04 pm

Hi there,

For the last weeks I have been trying to move Windows CI to travis.

The current Windows Travis CI tests of master are "faked", in the sense that even if they fail it shows pass.

Windows Travis CI executes the unit tests in a git-bash environment.

There are several issues that make, specially Path WB tests, fail (We managed Fem and had to remove Loft from PartDesign).

There are two reasons why are I posting:
1. I would like to have more verbosity when unit tests fail. Currently the why is rather hidden. See for example:
https://travis-ci.org/FreeCAD/FreeCAD/b ... tification

2. If you are going to work both unit tests and CI, in particular Travis, this may be relevant for you.

The thread were I log my tests is:
https://forum.freecadweb.org/viewtopic. ... &start=130

It all started with Appveyor, that is the reason for the title of the thread, but most of it today is about Travis.
PaulG
Posts: 37
Joined: Mon Feb 25, 2019 5:38 pm

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby PaulG » Fri May 24, 2019 6:25 pm

(I'm a freecad newbie so I may be wrong on details, and sorry this is so long.)

I think the testing mechanisms mentioned will rely on the developer also developing the tests, so I would like to draw your attention to other models. The main idea is to put in place a system which makes it easy for users to contribute tests. This wider participation of the community can be extremely valuable.

A variant of this has been very successful in another open source project I'm familiar with (R). That project has contributed packages that would be somewhat comparable to workbenches, with only a core of the most important packages being distributed with R. Most R package developers would be considered advanced users rather than developers of R itself. I think that distinction may also apply in the case of many freecad WB developers?

There is an extremely well developed format and mechanism for contributing packages to the main R distribution site CRAN. (It is documented in Writing R Extension linked to at https://cran.r-project.org/manuals.html.) The packaging system enforces a standard for documentation and in particular encourages examples. With some exceptions, the package will not be accepted unless examples work. This is enforced automatically by the package checking process, before the package is accepted. The format also allows for testing scripts and demos which must also work. These are advertised as examples and tests of the package but they form an enormous test suite for R itself. Being examples, they are typically a bit broader ranging than typical unit testing. The effect on packages of daily R development changes are summarized at https://cran.r-project.org/web/checks/. There are now many thousand packages, but in the early days of R there were about as many packages as there are freecad workbenches at the moment. Even then, the package tests were an important part of development because the package tests could be run to see what was broken by development changes, so substantial changes could be undertaken with reasonable expectation that the impact would be understood.

An important point here is the vetting process. To be accepted as a CRAN package the examples must work when the package is accepted. So when they don't work latter it is because of some change. That might be an intended change, so the package needs to be fixed, or it might be a bug introduced in the core development.

I worked briefly on another model that would allow all users to contribute code snippets to test packages, but that was in the days before CI tools were commonly available. This would have brought an even wider user group into the testing process. I gave up because the support effort at the time would have been too much, but I think it is more feasible now.

So, I encourage you to think about testing possibilities that would draw into the process the larger community. I have been working on something that might give you some ideas, at https://pdgilbert.github.io/EgFreeCAD/. It is in a very rudimentary state, I have just been writing down things as I work through them myself learning freecad.
m42kus
Posts: 159
Joined: Tue Sep 02, 2014 9:41 am

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby m42kus » Mon May 27, 2019 8:42 am

Thanks for all the great feedback! :)
yorik wrote:
Mon May 20, 2019 2:36 pm
Looks nice, but switching to it would require to rewrite the entire test suite of FreeCAD, which is a problem :D
Unless we keep both, but that would probably be more confusing than helping.
The unittest module and the pytest module can run quite happily beside on another. That's because the test runner of pytest treats unittest test cases like it's own thous executing them automatically. I think that would be a acceptable short to mid term but as you said it would still not make a good long term solution. But the transition could be made very smooth.
chrisb wrote:
Mon May 20, 2019 3:43 pm
To be honest, I'm not a big fan of additional modules. Even if it works now, we don't know what the future brings.

The argument with the classes does not count very much to me, because - as you say later - this tool is used by developers, and they probably know how to use classes.
I cannot say really much about the second point, because I don't understand the consequences in depth.
The third point would bring yet more dependencies, so following my argument from above I'm not a big fan of this either.

To be not too discouraging: How about making creation of tests more comfortable by giving developers a smart interface that can create all the stuff needed in the background.
This may be somehow reinventing the wheel, but we could customize this to our special needs and - again - as it is used by developers it doesn't have to be completely idiot proof.
The pytest package is actually very stable and used a lot by the python community. I don't think that it is likely to disappear anytime soon. I can't really say what concrete advantages it offers for FreeCAD developers because I haven't used it for this purpose yet. My motivation for using it are mainly to make life easier for the developer.
sgrogan wrote:
Mon May 20, 2019 8:50 pm
Keep in mind, that we have a good group of "pseudo-developers" that compile themselves and run these tests from the Test-framework WB. We also sometimes ask users to run these tests to trouble shoot their installation.
Thats a very good point we should definitely keep in mind.

All of you raised very good counter arguments. I think the two main concerns are the technicalities of adding and maintaining a new module as well as forcing developers to learn something new. During my project I will definitely develop support tools to make testing easier for us be it for pytest or for unittest.

However I'm really against reinventing something that a very well supported module already offers. Self made stuff will never be as good as what quality modules have to offer and maintaining that code would definitely be more work that just using an additional module. I understand you points of view very well, so for now my plan is to stick to unittest, but I may come back to that topic when I have a concrete plan on how my support framework will look like. GSoC has already began today so it won't take long ;)
My GSoC Devlog: ceeli.github.io/gsoc
m42kus
Posts: 159
Joined: Tue Sep 02, 2014 9:41 am

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby m42kus » Mon May 27, 2019 9:54 am

PaulG wrote:
Fri May 24, 2019 6:25 pm
(I'm a freecad newbie so I may be wrong on details, and sorry this is so long.)
Hi Paul, don't feel sorry, my posts always end up being that long :D
PaulG wrote:
Fri May 24, 2019 6:25 pm
The main idea is to put in place a system which makes it easy for users to contribute tests. This wider participation of the community can be extremely valuable.
That is a very nice system that would definitely benefit FreeCAD a lot. I like the idea of taking a little work of the core developers shoulders. And I think it would suite the very active FreeCAD Community well.
PaulG wrote:
Fri May 24, 2019 6:25 pm
A variant of this has been very successful in another open source project I'm familiar with (R). That project has contributed packages that would be somewhat comparable to workbenches, with only a core of the most important packages being distributed with R. Most R package developers would be considered advanced users rather than developers of R itself. I think that distinction may also apply in the case of many freecad WB developers?
Yes!
PaulG wrote:
Fri May 24, 2019 6:25 pm
There is an extremely well developed format and mechanism for contributing packages to the main R distribution site CRAN. (It is documented in Writing R Extension linked to at https://cran.r-project.org/manuals.html.) The packaging system enforces a standard for documentation and in particular encourages examples. With some exceptions, the package will not be accepted unless examples work. This is enforced automatically by the package checking process, before the package is accepted. The format also allows for testing scripts and demos which must also work. These are advertised as examples and tests of the package but they form an enormous test suite for R itself.
Thats a very ambitious system. I imagine that it took quite a while to reach it's current state. The end result is very much what I'd like to have for FreeCAD but I don't think that we can start straight away with that idea. I think starting with a traditional testing approach is a better way to build up the required infrastructure (services as well as support libraries). Than supplementing that more traditional way of testing with user contributed tests later on is the best way of doing it I think. I don't have anything like that in my proposal, but If I have enough time I could put some work into implementing this. Also if you are still interested and have a bit of time to spare in the next months we could work out something pushing those features together :)

I actually have a few questions about your project. What tool chain do you imagine? I couldn't work out how you use Travis CI in your git repo for testing? Do you use a unit test framework, or is it a collection of scripts that are executed?

Thanks for your contribution :)
Last edited by m42kus on Mon May 27, 2019 12:00 pm, edited 1 time in total.
My GSoC Devlog: ceeli.github.io/gsoc
m42kus
Posts: 159
Joined: Tue Sep 02, 2014 9:41 am

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby m42kus » Mon May 27, 2019 12:00 pm

abdullah wrote:
Wed May 22, 2019 12:04 pm
There are two reasons why are I posting:
1. I would like to have more verbosity when unit tests fail. Currently the why is rather hidden. See for example:
https://travis-ci.org/FreeCAD/FreeCAD/b ... tification

2. If you are going to work both unit tests and CI, in particular Travis, this may be relevant for you.

The thread were I log my tests is:
https://forum.freecadweb.org/viewtopic. ... &start=130
Hi, this will definitely be relevant for me at some point. If I understood correctly the status right now is that all tests run successfully except a few tests of the Path Workbench because they make use of the Gui module which can't be used in the windows virtual environment?
My GSoC Devlog: ceeli.github.io/gsoc