Test bundling with "make check"

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Post Reply
User avatar
kkremitzki
Veteran
Posts: 2511
Joined: Thu Mar 03, 2016 9:52 pm
Location: Illinois

Test bundling with "make check"

Post by kkremitzki »

I was recently doing some development for LibreOffice and saw that they have a "make check" Makefile rule which is a sort of unified test runner for new builds. I thought this might be interesting to implement for FreeCAD, but there's already FreeCAD -t 0 to run console tests. However, I was experimenting and found this way to fire the full GUI tests from the shell with FreeCADCmd:

Code: Select all

cat | freecadcmd-daily &
fc_pid=$!
run_py () {
    echo "$1" > "/proc/$fc_pid/fd/0"
}
run_py "import FreeCADGui"
run_py "FreeCADGui.showMainWindow()"
run_py "import QtUnitGui"
run_py "QtUnitGui.addTest('TestApp.All')"
run_py "QtUnitGui.setTest('TestApp.All')"
run_py "import qtunittest, gc"
run_py "__qt_test__=qtunittest.QtTestRunner(0,'')"
run_py "__qt_test__.runClicked()"
run_py "del __qt_test__"
run_py "gc.collect()"

I don't think this will work on Windows... but would it be useful to use something like this to fire all GUI tests via make check after a successful make?
Last edited by kkremitzki on Wed Mar 08, 2017 11:02 pm, edited 1 time in total.
Like my FreeCAD work? I'd appreciate any level of support via Patreon, Liberapay, or PayPal! Read more about what I do at my blog.
ian.rees
Posts: 696
Joined: Sun Jun 15, 2014 3:28 am
Contact:

Re: Test bundling with "make check"

Post by ian.rees »

Regardless of what goes in to the Makefiles (which really means cmake source), I think we should be able to run GUI tests via CLI arguments - you'd need that to run them via make anyways.

To me, it doesn't seem worth cluttering the cmake source just to avoid such a simple command line argument - the CLI argument is discoverable in the usual way (somecommand --help), but a hypothetical make target wouldn't be. -Ian-

Edit:ed following kkremitzki's edit ;)
Last edited by ian.rees on Wed Mar 08, 2017 11:13 pm, edited 1 time in total.
User avatar
sgrogan
Veteran
Posts: 6499
Joined: Wed Oct 22, 2014 5:02 pm

Re: Test bundling with "make check"

Post by sgrogan »

kkremitzki wrote:I don't think this will work on Windows... but would it be useful to use something like this to fire all GUI tests via make clean after a successful make?
Result after pasting code into FreeCAD console (I remove -daily because we don't have this on win)

Code: Select all

[FreeCAD Console mode <Use Ctrl-Z plus Return to exit.>]
>>> cat | freecadcmd &
  File "<stdin>", line 1
    cat | freecadcmd &
                     ^
SyntaxError: invalid syntax
>>> fc_pid=$!
  File "<stdin>", line 1
    fc_pid=$!
           ^
SyntaxError: invalid syntax
>>> run_py () {
  File "<stdin>", line 1
    run_py () {
              ^
SyntaxError: invalid syntax
>>>     echo "$1" > "/proc/$fc_pid/fd/0"
  File "<stdin>", line 1
    echo "$1" > "/proc/$fc_pid/fd/0"
    ^
IndentationError: unexpected indent
>>> }
  File "<stdin>", line 1
    }
    ^
SyntaxError: invalid syntax
>>> run_py "import FreeCADGui"
  File "<stdin>", line 1
    run_py "import FreeCADGui"
                             ^
SyntaxError: invalid syntax
>>> run_py "FreeCADGui.showMainWindow()"
  File "<stdin>", line 1
    run_py "FreeCADGui.showMainWindow()"
                                       ^
SyntaxError: invalid syntax
>>> run_py "import QtUnitGui"
  File "<stdin>", line 1
    run_py "import QtUnitGui"
                            ^
SyntaxError: invalid syntax
>>> run_py "QtUnitGui.addTest('TestApp.All')"
  File "<stdin>", line 1
    run_py "QtUnitGui.addTest('TestApp.All')"
                                            ^
SyntaxError: invalid syntax
>>> run_py "QtUnitGui.setTest('TestApp.All')"
  File "<stdin>", line 1
    run_py "QtUnitGui.setTest('TestApp.All')"
                                            ^
SyntaxError: invalid syntax
>>> run_py "import qtunittest, gc"
  File "<stdin>", line 1
    run_py "import qtunittest, gc"
                                 ^
SyntaxError: invalid syntax
>>> run_py "__qt_test__=qtunittest.QtTestRunner(0,'')"
  File "<stdin>", line 1
    run_py "__qt_test__=qtunittest.QtTestRunner(0,'')"
                                                     ^
SyntaxError: invalid syntax
>>> run_py "__qt_test__.runClicked()"
  File "<stdin>", line 1
    run_py "__qt_test__.runClicked()"
                                    ^
SyntaxError: invalid syntax
>>> run_py "del __qt_test__"
  File "<stdin>", line 1
    run_py "del __qt_test__"
                           ^
SyntaxError: invalid syntax
>>> run_py "gc.collect()"
  File "<stdin>", line 1
    run_py "gc.collect()"
                        ^
SyntaxError: invalid syntax
>>>

ian.rees wrote:Tests certainly should not happen after "clean" - that target already has a well established meaning: delete all intermediate compilation files. -Ian-
+1
on win it could be a seperate project, like
make
make guitest
make install
"fight the good fight"
User avatar
kkremitzki
Veteran
Posts: 2511
Joined: Thu Mar 03, 2016 9:52 pm
Location: Illinois

Re: Test bundling with "make check"

Post by kkremitzki »

ian.rees wrote:Tests certainly should not happen after "clean" - that target already has a well established meaning: delete all intermediate compilation files. -Ian-
Typo, I meant "make check". Fixed.
Like my FreeCAD work? I'd appreciate any level of support via Patreon, Liberapay, or PayPal! Read more about what I do at my blog.
wmayer
Founder
Posts: 20245
Joined: Thu Feb 19, 2009 10:32 am
Contact:

Re: Test bundling with "make check"

Post by wmayer »

"make check" is something you get almost for free when using the autotools. In the past when FreeCAD was using it we implemented the make dist & make distcheck targets which created a source tarball and the latter additionally unpacked the tarball and compiled them in order to be sure that no file was missing.

After the move to cmake as universal build system this wasn't possible any more because unlike to the Makefile.am of automake which is using a white list of all files that should go into the tarball cmake keeps a black list which doesn't work very well because you ended up with a lot of junk in the tarballs.

Now after the move to git we again had a good way to mimic the dist/distcheck targets because we can use git's archive function and attribute file to create proper tarballs. For makefile based builds we have it back as make dist-git and make distcheck-git.

So, IMO it would be nice to also support make check
Post Reply