Test bundling with "make check"

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
User avatar
kkremitzki
Posts: 348
Joined: Thu Mar 03, 2016 9:52 pm
Location: Texas

Test bundling with "make check"

Postby kkremitzki » Wed Mar 08, 2017 10:37 pm

I was recently doing some development for LibreOffice and saw that they have a "make check" Makefile rule which is a sort of unified test runner for new builds. I thought this might be interesting to implement for FreeCAD, but there's already FreeCAD -t 0 to run console tests. However, I was experimenting and found this way to fire the full GUI tests from the shell with FreeCADCmd:

Code: Select all

cat | freecadcmd-daily &
fc_pid=$!
run_py () {
    echo "$1" > "/proc/$fc_pid/fd/0"
}
run_py "import FreeCADGui"
run_py "FreeCADGui.showMainWindow()"
run_py "import QtUnitGui"
run_py "QtUnitGui.addTest('TestApp.All')"
run_py "QtUnitGui.setTest('TestApp.All')"
run_py "import qtunittest, gc"
run_py "__qt_test__=qtunittest.QtTestRunner(0,'')"
run_py "__qt_test__.runClicked()"
run_py "del __qt_test__"
run_py "gc.collect()"



I don't think this will work on Windows... but would it be useful to use something like this to fire all GUI tests via make check after a successful make?
Last edited by kkremitzki on Wed Mar 08, 2017 11:02 pm, edited 1 time in total.
ian.rees
Posts: 577
Joined: Sun Jun 15, 2014 3:28 am
Contact:

Re: Test bundling with "make check"

Postby ian.rees » Wed Mar 08, 2017 10:57 pm

Regardless of what goes in to the Makefiles (which really means cmake source), I think we should be able to run GUI tests via CLI arguments - you'd need that to run them via make anyways.

To me, it doesn't seem worth cluttering the cmake source just to avoid such a simple command line argument - the CLI argument is discoverable in the usual way (somecommand --help), but a hypothetical make target wouldn't be. -Ian-

Edit:ed following kkremitzki's edit ;)
Last edited by ian.rees on Wed Mar 08, 2017 11:13 pm, edited 1 time in total.
User avatar
sgrogan
Posts: 3042
Joined: Wed Oct 22, 2014 5:02 pm

Re: Test bundling with "make check"

Postby sgrogan » Wed Mar 08, 2017 11:01 pm

kkremitzki wrote:I don't think this will work on Windows... but would it be useful to use something like this to fire all GUI tests via make clean after a successful make?

Result after pasting code into FreeCAD console (I remove -daily because we don't have this on win)

Code: Select all

[FreeCAD Console mode <Use Ctrl-Z plus Return to exit.>]
>>> cat | freecadcmd &
  File "<stdin>", line 1
    cat | freecadcmd &
                     ^
SyntaxError: invalid syntax
>>> fc_pid=$!
  File "<stdin>", line 1
    fc_pid=$!
           ^
SyntaxError: invalid syntax
>>> run_py () {
  File "<stdin>", line 1
    run_py () {
              ^
SyntaxError: invalid syntax
>>>     echo "$1" > "/proc/$fc_pid/fd/0"
  File "<stdin>", line 1
    echo "$1" > "/proc/$fc_pid/fd/0"
    ^
IndentationError: unexpected indent
>>> }
  File "<stdin>", line 1
    }
    ^
SyntaxError: invalid syntax
>>> run_py "import FreeCADGui"
  File "<stdin>", line 1
    run_py "import FreeCADGui"
                             ^
SyntaxError: invalid syntax
>>> run_py "FreeCADGui.showMainWindow()"
  File "<stdin>", line 1
    run_py "FreeCADGui.showMainWindow()"
                                       ^
SyntaxError: invalid syntax
>>> run_py "import QtUnitGui"
  File "<stdin>", line 1
    run_py "import QtUnitGui"
                            ^
SyntaxError: invalid syntax
>>> run_py "QtUnitGui.addTest('TestApp.All')"
  File "<stdin>", line 1
    run_py "QtUnitGui.addTest('TestApp.All')"
                                            ^
SyntaxError: invalid syntax
>>> run_py "QtUnitGui.setTest('TestApp.All')"
  File "<stdin>", line 1
    run_py "QtUnitGui.setTest('TestApp.All')"
                                            ^
SyntaxError: invalid syntax
>>> run_py "import qtunittest, gc"
  File "<stdin>", line 1
    run_py "import qtunittest, gc"
                                 ^
SyntaxError: invalid syntax
>>> run_py "__qt_test__=qtunittest.QtTestRunner(0,'')"
  File "<stdin>", line 1
    run_py "__qt_test__=qtunittest.QtTestRunner(0,'')"
                                                     ^
SyntaxError: invalid syntax
>>> run_py "__qt_test__.runClicked()"
  File "<stdin>", line 1
    run_py "__qt_test__.runClicked()"
                                    ^
SyntaxError: invalid syntax
>>> run_py "del __qt_test__"
  File "<stdin>", line 1
    run_py "del __qt_test__"
                           ^
SyntaxError: invalid syntax
>>> run_py "gc.collect()"
  File "<stdin>", line 1
    run_py "gc.collect()"
                        ^
SyntaxError: invalid syntax
>>>


ian.rees wrote:Tests certainly should not happen after "clean" - that target already has a well established meaning: delete all intermediate compilation files. -Ian-

+1
on win it could be a seperate project, like
make
make guitest
make install
User avatar
kkremitzki
Posts: 348
Joined: Thu Mar 03, 2016 9:52 pm
Location: Texas

Re: Test bundling with "make check"

Postby kkremitzki » Wed Mar 08, 2017 11:02 pm

ian.rees wrote:Tests certainly should not happen after "clean" - that target already has a well established meaning: delete all intermediate compilation files. -Ian-

Typo, I meant "make check". Fixed.
wmayer
Site Admin
Posts: 11332
Joined: Thu Feb 19, 2009 10:32 am

Re: Test bundling with "make check"

Postby wmayer » Thu Mar 09, 2017 9:01 am

"make check" is something you get almost for free when using the autotools. In the past when FreeCAD was using it we implemented the make dist & make distcheck targets which created a source tarball and the latter additionally unpacked the tarball and compiled them in order to be sure that no file was missing.

After the move to cmake as universal build system this wasn't possible any more because unlike to the Makefile.am of automake which is using a white list of all files that should go into the tarball cmake keeps a black list which doesn't work very well because you ended up with a lot of junk in the tarballs.

Now after the move to git we again had a good way to mimic the dist/distcheck targets because we can use git's archive function and attribute file to create proper tarballs. For makefile based builds we have it back as make dist-git and make distcheck-git.

So, IMO it would be nice to also support make check