Set exit status code based upon unit test results

Merged, abandoned or rejected pull requests are moved here to clear the main Pull Requests forum.
blacey
Posts: 367
Joined: Tue Dec 08, 2015 11:28 pm

Set exit status code based upon unit test results

Postby blacey » Thu Mar 10, 2016 6:27 am

Currently FreeCAD provides the option to run unit tests from the command line using the --run-test command-line argument however the application does not check the test results and exit with a non-zero status code if any tests fail. In order to adhere to proper continuous integration best-practices, we should exit with a status code indicative of the test pass/failure results. The Python unittest module provides a TextTestRunner() method to execute tests that will return a TestResult object that encapsulates the test result details (# passed, #failed, etc.).

The scope of this pull-request is simply to enhance the TestApp.py FreeCAD module to return the TestResult object to the caller, in this specific case, FreeCADTest.py, so it can interrogate the TestResult object to see if all tests passed using the wasSuccessful() method and exit with the proper zero (if all passed) and non-zero (if any tests failed) exit status.

This change provides a clean, cross-platform mechanism to execute the unit test suite from the command-line for continuous integration. With this change, for example, we can change travis configuration from:

Code: Select all

PYTHONPATH=$(pwd)/lib/ python -c "import sys, unittest, FreeCAD, TestApp; sys.exit(0 if unittest.TextTestRunner().run(TestApp.All()).wasSuccessful() else 1)"
to:

Code: Select all

$(pwd)/bin/FreeCAD --run-test 0
Whereby we are running the unit tests within the fully linked executable and follows standard unix exit status conventions that can be leveraged by any script/process that calls FreeCAD. Future travis configs, for most platforms, will be able to run the unit tests against the "installed" executable that more closely mimics an end-user's environment.

Finally, FreeCADTest.py calls os._exit(exitStatusCode) instead of sys.exit(exitStatusCode) because FreeCAD Base::Interpreter:SystemExitException swallows the system exit code so it is not available to the calling try/catch block that currently traps the SystemExitException exception and exits with a 0 status code. Calling os._exit() for this specific use case is the least intrusive change without refactoring Base::Interpreter.
wmayer
Site Admin
Posts: 14896
Joined: Thu Feb 19, 2009 10:32 am

Re: Set exit status code based upon unit test results

Postby wmayer » Thu Mar 10, 2016 10:19 am

Merged.