Testing Tools Survey (FEM Workbench Testing - GSoC)

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
User avatar
apeltauer
Posts: 212
Joined: Fri Aug 11, 2017 2:19 pm

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby apeltauer » Tue Jun 11, 2019 3:07 am

My workbench is not open source.
I don't think that open coverage works on Linux....
As fare as I can see it depends on pdb files for the debug information. I don’t know if Linux has the same files?!?
m42kus
Posts: 159
Joined: Tue Sep 02, 2014 9:41 am

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby m42kus » Thu Jun 13, 2019 10:30 am

Sorry for the late repose, unfortunately I am a little sick at the moment...
apeltauer wrote:
Tue Jun 11, 2019 3:07 am
My workbench is not open source.
Alright, than I have a few questions. If you can't or don't want to answer them, no problem ;)
Did you start the FreeCAD application before executing the tests? If so how did you do it? I just developed a first version of a test runner that starts FreeCAD in advance. I didn't run in to any problems yet but I'm sure that I will at some point. (https://github.com/ceeli/FreeCAD_bhb/co ... 4ad9115463)

Do you also use python tests? If so do you run them separately or do you somehow combine them? I think it would be very nice to have a test runner that runs both C++ and python tests.

Do you use some kind of automated test coverage reports? Incooperated into the cmake build system or something similar?
apeltauer wrote:
Tue Jun 11, 2019 3:07 am
I don't think that open coverage works on Linux....
As fare as I can see it depends on pdb files for the debug information. I don’t know if Linux has the same files?!?
It seems that it doesn't work and the author has no plan on porting it: https://github.com/OpenCppCoverage/Open ... /issues/31. This is not too bad though because gcov does the job well enough I think :) It's also the most popular choice and is supported by many tools.
My GSoC Devlog: ceeli.github.io/gsoc
User avatar
apeltauer
Posts: 212
Joined: Fri Aug 11, 2017 2:19 pm

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby apeltauer » Thu Jun 13, 2019 1:11 pm

Just because the workbench is not open source, it doesn’t mean that I don’t want to help, answer or share some sample code :)

Right now I have no access to my source code during my holidays. But I will try to answers it fast.

I have several gtest applications, e.g. one for my business logic and all algorithms with open cascade. A other one is for testing my Qt stuff (Widgets etc.) Those Test’s don’t need Freecad to be started because I have encapsulated the Freecad stuff.
When it comes to testing my Freecad specific stuff (GUI handling....) I am starting Freecad and pass a python script. This script is calling a workbench function. This function starts my gtest inside of Freecad... :)

I sounds easy but it was a little bit tricky, but now it works.... :)

For more detailed questions and answers I think I need my source code... :)

I only test c++ stuff. But I think you also could to start python tests.

OpenCppCoverage generates a xml report. Those xml files I parse with a little tool and check that my coverage is not below 95%
ascaron
Posts: 12
Joined: Sat Jun 01, 2019 3:59 pm

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby ascaron » Fri Jul 26, 2019 7:13 am

I'm not a big fan of Google either.
What do you think about https://cpputest.github.io/ ?
It's a little smaller than Google Test and maintained by one of the agile manifesto guys, James Grenning. I've used it throughout his book on unit testing of embedded devices and liked it so far. Though I have to say I never used Google Test before and so can not compare.
m42kus
Posts: 159
Joined: Tue Sep 02, 2014 9:41 am

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby m42kus » Fri Jul 26, 2019 10:17 am

ascaron wrote:
Fri Jul 26, 2019 7:13 am
I'm not a big fan of Google either.
What do you think about https://cpputest.github.io/ ?
It's a little smaller than Google Test and maintained by one of the agile manifesto guys, James Grenning. I've used it throughout his book on unit testing of embedded devices and liked it so far. Though I have to say I never used Google Test before and so can not compare.
Hi, I just had a look at it and it appears to be very similar to googletest. It's also a xUnit inspired testing library and those are all not that much different from one another meaning that technical differences are less significant in my opinion. I think much more important is the user base. I (and I think most developers) don't want to learn all the macros of a new framework which in the end is more or less the same than every other framework. For pure C++ googletest seems to be the default option, has a very large userbase and there are many tutorials and resources from the community. That why I opted for googletest. Yes it's comes from google but it's BSD licensed everyone can contribute. I don't wan't to say that googletest is the only option or that CppUTest is bad, I just wanted to express what I think our priorities should be when choosing a framework.
My GSoC Devlog: ceeli.github.io/gsoc
ascaron
Posts: 12
Joined: Sat Jun 01, 2019 3:59 pm

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby ascaron » Fri Jul 26, 2019 3:24 pm

Just wanted to throw in an alternative :)
I have no problem using gtest, I simply associate mostly bad stuff with Google. Randomly abandoning projects, keeping the project lead of open source projects in the company and so on. There was an interesting discussion about the Go Language, whether it's a Google or community language.

Anyway, I'll give it a try on the rather small Robot WB by following your blog.
m42kus
Posts: 159
Joined: Tue Sep 02, 2014 9:41 am

Re: Testing Tools Survey (FEM Workbench Testing - GSoC)

Postby m42kus » Fri Jul 26, 2019 3:53 pm

ascaron wrote:
Fri Jul 26, 2019 3:24 pm
Just wanted to throw in an alternative :)
I have no problem using gtest, I simply associate mostly bad stuff with Google. Randomly abandoning projects, keeping the project lead of open source projects in the company and so on. There was an interesting discussion about the Go Language, whether it's a Google or community language.
Yes you are of course right, google doesn't have the best track record. I think with unit test frameworks it's not so bad because there is normally very little innovation in a unit test library and I'm positive that even if it is shut down that the community will continue to support it. One alternative I also considered is Boost.Test. It's of course also a good option but I have not worked with it yet.
ascaron wrote:
Fri Jul 26, 2019 3:24 pm
Anyway, I'll give it a try on the rather small Robot WB by following your blog.
Cool, keep in mind that at the moment the C++ unit tests are compiled into a separate executable. This was more like a proof of concept till now. My plan is to somehow make a similar workflow possible for python and C++ unit tests which means that there may be a lot of changes during the next month. This and probably next week I'm still busy writing python tests but after that I'm going to start working on the infrastructure and overall structure of the test case organization. If you have any questions feel free to contact me :)
My GSoC Devlog: ceeli.github.io/gsoc