Running unit tests automatically
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Be nice to others! Respect the FreeCAD code of conduct!
Running unit tests automatically
I was wondering how folks feel about having the unit tests run automatically as part of the build, and make their successful completion a gating criteria for merging changes into master.
Path has been bitten a bit lately by regressions coming in through changes in other components and/or FC core, all of which could have been prevented/flagged with proper tests if they were executed continuously.
Path has been bitten a bit lately by regressions coming in through changes in other components and/or FC core, all of which could have been prevented/flagged with proper tests if they were executed continuously.
Re: Running unit tests automatically
I wouldn't force it - you can always make a one liner to run it with the build if you want (at least on linux).
make -j8 && bin/FreeCAD -t 0
The tests are happening here [1] and for each pull request on github (tavis & appveyor). Example here [2].
[1] https://travis-ci.org/FreeCAD/FreeCAD
[2] https://github.com/FreeCAD/FreeCAD/pull/458
make -j8 && bin/FreeCAD -t 0
The tests are happening here [1] and for each pull request on github (tavis & appveyor). Example here [2].
[1] https://travis-ci.org/FreeCAD/FreeCAD
[2] https://github.com/FreeCAD/FreeCAD/pull/458
Re: Running unit tests automatically
As PrzemoF has mentioned the tests are run (non-gui tests to be specific) at each build on the CI engines. When the main devs choose to merge a pull request they can ignore it. Unfortunately or CI process can still be improved. There are still false failures. I'm aware of the issues you are referring to, can you find the commit/pull request from https://github.com/FreeCAD/FreeCAD/commits/master? It would be interesting to see what the indications were. See here: https://github.com/FreeCAD/FreeCAD/pull/433 also, this error seams to float around.mlampert wrote:I was wondering how folks feel about having the unit tests run automatically as part of the build, and make their successful completion a gating criteria for merging changes into master.
EDIT: See here: https://travis-ci.org/FreeCAD/FreeCAD/j ... 4928#L6553 Now the failure is only with clang on OSX?
"fight the good fight"
- kkremitzki
- Veteran
- Posts: 2511
- Joined: Thu Mar 03, 2016 9:52 pm
- Location: Illinois
Re: Running unit tests automatically
The sentiment of more tests is great; like you said, they will help flag regressions when people are working on code that can affect parts of the app they're not familiar with. My only concern is that the builds already take so long. Is there any way to make things go faster? How's FreeCAD's build time compared to other open source projects of comparable size, if such a thing can be determined?
Re: Running unit tests automatically
My record build time is ~45 seconds and I have a screencast somewhere to prove it Travis can handle it in 7 to 20 minutes with ccache populated. I don't think there is an easy way to accelerate the build. I strongly recommend to install ccache if you build FreeCAD yourself.
Re: Running unit tests automatically
Just a superficial comparison (none of these are of comparable size and they're all via travis so not super accurate)kkremitzki wrote:The sentiment of more tests is great; like you said, they will help flag regressions when people are working on code that can affect parts of the app they're not familiar with. My only concern is that the builds already take so long. Is there any way to make things go faster? How's FreeCAD's build time compared to other open source projects of comparable size, if such a thing can be determined?
Example of other FOSS projects:
- Scribus https://travis-ci.org/scribusproject/scribus/ ~40
- Qgis https://travis-ci.org/qgis/QGIS/builds ~16
- GIMP https://travis-ci.org/Swyter/gimp-git/builds ~30
- LibreCAD https://travis-ci.org/LibreCAD/LibreCAD/builds ~10
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
Re: Running unit tests automatically
Build times are always an issue. In my experience it comes down to the question if you want a machine to spend the time running your unit tests continuously and flagging regressions as early as possible? Or do you want a person to spend the time triaging an issue and finding the root cause (or at least narrow it down to a commit)? And both times will increase the more complex FC becomes, the latter one not linearly.
- kkremitzki
- Veteran
- Posts: 2511
- Joined: Thu Mar 03, 2016 9:52 pm
- Location: Illinois
Re: Running unit tests automatically
Thanks, I'm not super familiar with the C++ tooling ecosystem. I'd never heard of ccache (but I also only heard about ccmake, the ncurses UI to cmake, about a month ago.) Last time I timed it, it took me 15 minutes with make - j$(nproc) on my 8 core desktop. I just did timed the cached build and it took about 2.PrzemoF wrote:My record build time is ~45 seconds and I have a screencast somewhere to prove it Travis can handle it in 7 to 20 minutes with ccache populated. I don't think there is an easy way to accelerate the build. I strongly recommend to install ccache if you build FreeCAD yourself.
So thanks for the tip!
Re: Running unit tests automatically
Back to the beginning
We do? as PrzemoF has pointed out every pull request and Merge to master goes through these tests. See here for the results: https://github.com/FreeCAD/FreeCAD/commits/master Click on the green check mark or the red x.mlampert wrote:I was wondering how folks feel about having the unit tests run automatically as part of the build, and make their successful completion a gating criteria for merging changes into master.
I agree with this whole heartedly. We need to find where the process failed and improve it!mlampert wrote:Build times are always an issue. In my experience it comes down to the question if you want a machine to spend the time running your unit tests continuously and flagging regressions as early as possible? Or do you want a person to spend the time triaging an issue and finding the root cause (or at least narrow it down to a commit)? And both times will increase the more complex FC becomes, the latter one not linearly.
"fight the good fight"
Re: Running unit tests automatically
I think the confusion here might be that, as I understand, recently some unit tests were added for the path wb but this tests don't run, if I am not mistaken they don't even run when one runs the unit tests manually from the gui. I guess they were not integrated properly... I did not recheck all this so sorry if I am mistaken
https://github.com/FreeCAD/FreeCAD/comm ... 30571eb9af ?
But as has been said before we are running unit tests 24/7 on all CI builds and those that we run now don't take a lot of time, so at least for now more new unit tests are IMO very welcome
https://github.com/FreeCAD/FreeCAD/comm ... 30571eb9af ?
But as has been said before we are running unit tests 24/7 on all CI builds and those that we run now don't take a lot of time, so at least for now more new unit tests are IMO very welcome