Running unit tests automatically

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Post Reply
mlampert
Veteran
Posts: 1772
Joined: Fri Sep 16, 2016 9:28 pm

Running unit tests automatically

Post by mlampert »

I was wondering how folks feel about having the unit tests run automatically as part of the build, and make their successful completion a gating criteria for merging changes into master.

Path has been bitten a bit lately by regressions coming in through changes in other components and/or FC core, all of which could have been prevented/flagged with proper tests if they were executed continuously.
User avatar
PrzemoF
Veteran
Posts: 3520
Joined: Fri Jul 25, 2014 4:52 pm
Contact:

Re: Running unit tests automatically

Post by PrzemoF »

I wouldn't force it - you can always make a one liner to run it with the build if you want (at least on linux).

make -j8 && bin/FreeCAD -t 0

The tests are happening here [1] and for each pull request on github (tavis & appveyor). Example here [2].
[1] https://travis-ci.org/FreeCAD/FreeCAD
[2] https://github.com/FreeCAD/FreeCAD/pull/458
User avatar
sgrogan
Veteran
Posts: 6499
Joined: Wed Oct 22, 2014 5:02 pm

Re: Running unit tests automatically

Post by sgrogan »

mlampert wrote:I was wondering how folks feel about having the unit tests run automatically as part of the build, and make their successful completion a gating criteria for merging changes into master.
As PrzemoF has mentioned the tests are run (non-gui tests to be specific) at each build on the CI engines. When the main devs choose to merge a pull request they can ignore it. Unfortunately or CI process can still be improved. There are still false failures. I'm aware of the issues you are referring to, can you find the commit/pull request from https://github.com/FreeCAD/FreeCAD/commits/master? It would be interesting to see what the indications were. See here: https://github.com/FreeCAD/FreeCAD/pull/433 also, this error seams to float around.
EDIT: See here: https://travis-ci.org/FreeCAD/FreeCAD/j ... 4928#L6553 Now the failure is only with clang on OSX?
"fight the good fight"
User avatar
kkremitzki
Veteran
Posts: 2511
Joined: Thu Mar 03, 2016 9:52 pm
Location: Illinois

Re: Running unit tests automatically

Post by kkremitzki »

The sentiment of more tests is great; like you said, they will help flag regressions when people are working on code that can affect parts of the app they're not familiar with. My only concern is that the builds already take so long. Is there any way to make things go faster? How's FreeCAD's build time compared to other open source projects of comparable size, if such a thing can be determined?
Like my FreeCAD work? I'd appreciate any level of support via Patreon, Liberapay, or PayPal! Read more about what I do at my blog.
User avatar
PrzemoF
Veteran
Posts: 3520
Joined: Fri Jul 25, 2014 4:52 pm
Contact:

Re: Running unit tests automatically

Post by PrzemoF »

My record build time is ~45 seconds and I have a screencast somewhere to prove it :D Travis can handle it in 7 to 20 minutes with ccache populated. I don't think there is an easy way to accelerate the build. I strongly recommend to install ccache if you build FreeCAD yourself.
User avatar
Kunda1
Veteran
Posts: 13434
Joined: Thu Jan 05, 2017 9:03 pm

Re: Running unit tests automatically

Post by Kunda1 »

kkremitzki wrote:The sentiment of more tests is great; like you said, they will help flag regressions when people are working on code that can affect parts of the app they're not familiar with. My only concern is that the builds already take so long. Is there any way to make things go faster? How's FreeCAD's build time compared to other open source projects of comparable size, if such a thing can be determined?
Just a superficial comparison (none of these are of comparable size and they're all via travis so not super accurate)
Example of other FOSS projects:
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
mlampert
Veteran
Posts: 1772
Joined: Fri Sep 16, 2016 9:28 pm

Re: Running unit tests automatically

Post by mlampert »

Build times are always an issue. In my experience it comes down to the question if you want a machine to spend the time running your unit tests continuously and flagging regressions as early as possible? Or do you want a person to spend the time triaging an issue and finding the root cause (or at least narrow it down to a commit)? And both times will increase the more complex FC becomes, the latter one not linearly.
User avatar
kkremitzki
Veteran
Posts: 2511
Joined: Thu Mar 03, 2016 9:52 pm
Location: Illinois

Re: Running unit tests automatically

Post by kkremitzki »

PrzemoF wrote:My record build time is ~45 seconds and I have a screencast somewhere to prove it :D Travis can handle it in 7 to 20 minutes with ccache populated. I don't think there is an easy way to accelerate the build. I strongly recommend to install ccache if you build FreeCAD yourself.
Thanks, I'm not super familiar with the C++ tooling ecosystem. I'd never heard of ccache (but I also only heard about ccmake, the ncurses UI to cmake, about a month ago.) Last time I timed it, it took me 15 minutes with make - j$(nproc) on my 8 core desktop. I just did timed the cached build and it took about 2.

So thanks for the tip!
Like my FreeCAD work? I'd appreciate any level of support via Patreon, Liberapay, or PayPal! Read more about what I do at my blog.
User avatar
sgrogan
Veteran
Posts: 6499
Joined: Wed Oct 22, 2014 5:02 pm

Re: Running unit tests automatically

Post by sgrogan »

Back to the beginning ;)
mlampert wrote:I was wondering how folks feel about having the unit tests run automatically as part of the build, and make their successful completion a gating criteria for merging changes into master.
We do? as PrzemoF has pointed out every pull request and Merge to master goes through these tests. See here for the results: https://github.com/FreeCAD/FreeCAD/commits/master Click on the green check mark or the red x.
mlampert wrote:Build times are always an issue. In my experience it comes down to the question if you want a machine to spend the time running your unit tests continuously and flagging regressions as early as possible? Or do you want a person to spend the time triaging an issue and finding the root cause (or at least narrow it down to a commit)? And both times will increase the more complex FC becomes, the latter one not linearly.
I agree with this whole heartedly. We need to find where the process failed and improve it!
"fight the good fight"
User avatar
saso
Veteran
Posts: 1920
Joined: Fri May 16, 2014 1:14 pm
Contact:

Re: Running unit tests automatically

Post by saso »

I think the confusion here might be that, as I understand, recently some unit tests were added for the path wb but this tests don't run, if I am not mistaken they don't even run when one runs the unit tests manually from the gui. I guess they were not integrated properly... I did not recheck all this so sorry if I am mistaken :roll:

https://github.com/FreeCAD/FreeCAD/comm ... 30571eb9af ?

But as has been said before we are running unit tests 24/7 on all CI builds and those that we run now don't take a lot of time, so at least for now more new unit tests are IMO very welcome :)
Post Reply