External full Python WB Unit Tests

[apeltauer]

Hi all,

I just figured out that with the commit:

---------------------------------------------------------------------------------------
Commit: 404b42d0a6059faa668c72255d80b7a6932c56ce [404b42d]
Parents: 559a5e6e58
Author: wmayer <wmayer@users.sourceforge.net>
Date: Tuesday, 28 August 2018 19:39:34
Committer: wmayer
unit tests of Arch and Draft require GUI, improve output of units tests
---------------------------------------------------------------------------------------

the unit test does not work on windows 10 anymore.
The function "def testSchemes(self)" which was added throws an error: ("src\Mod\Test\UnitTests.py" line 53)

---------------------------
Error: testSchemes (UnitTests.UnitBasicCases)
---------------------------
Traceback (most recent call last):
File "C:\Work\FreeCAD_dev\bin_debug\bin\lib\unittest\case.py", line 329, in run
testMethod()
File "C:\Work\FreeCAD_dev\bin_debug\Mod\Test\UnitTests.py", line 61, in testSchemes
self.assertAlmostEqual(1, v.Value, msg="Failed with \"{0}\" scheme: {1} != 1 (delta: {2})".format(schemes, v.Value, self.delta), delta=self.delta)
AttributeError: 'UnitBasicCases' object has no attribute 'delta'

---------------------------
OK Hide Details...
---------------------------

Any idea?
Thanks in advance.

My config:
OS: Windows 10
Word size of OS: 64-bit
Word size of FreeCAD: 64-bit
Version: 0.18.14487 +903 (Git)
Build type: Debug
Branch: pre_integration_cirunner
Hash: 74d67584618491e77bd455e1c368b9de429716ab
Python version: 2.7.8
Qt version: 4.8.7
Coin version: 4.0.0a
OCC version: 7.0.0
Locale: English/Germany (en_DE)

[gflorent]

2. Is it possible to add a test suite for an external WB to the Test WB (without editing the Python code in the Test WB of the installed FreeCAD) or should the usual test runners from a testing framework be used?

Yes, you can add your tests to the Test WB without changing its source. However, a few things must be polished a bit to improve the integration. Assuming you have a module called MyTest which resides in the Mod directory then you can do the following things:

• Create a file __init__.py
• Create a directory Tests with another __init__.py and e.g. TestCase.py
• The content of Tests/__init__.py can e.g. be:

  Code: Select all

  import QtUnitGui
  QtUnitGui.addTest("MyTest.Tests.TestCase")
  

• and TestCase.py

Thanks a lot for the answer Werner. I will try and see how it goes.

[wmayer]

Traceback (most recent call last):
File "C:\Work\FreeCAD_dev\bin_debug\bin\lib\unittest\case.py", line 329, in run
testMethod()
File "C:\Work\FreeCAD_dev\bin_debug\Mod\Test\UnitTests.py", line 61, in testSchemes
self.assertAlmostEqual(1, v.Value, msg="Failed with \"{0}\" scheme: {1} != 1 (delta: {2})".format(schemes, v.Value, self.delta), delta=self.delta)
AttributeError: 'UnitBasicCases' object has no attribute 'delta'

This is weird. All what I have changed is to also show the value of "delta" in the error message in case the test fails and "self.delta" is already used at the end of the line.

Furthermore, the Windows build on AppVeyor and the Linux builds on travis work all, too. Only the macOS build fails but this apparently has something to do with TestFem.

[apeltauer]

Thanks Werner for your reply.
I will debug the script tomorrow and check it again...

[wmayer]

File "C:\Work\FreeCAD_dev\bin_debug\Mod\Test\UnitTests.py", line 61, in testSchemes
self.assertAlmostEqual(1, v.Value, msg="Failed with \"{0}\" scheme: {1} != 1 (delta: {2})".format(schemes, v.Value, self.delta), delta=self.delta)
AttributeError: 'UnitBasicCases' object has no attribute 'delta'

It looks like your file UnitTests.py is corrupted as the function testSchemes starts in line 69 and not in line 61.

[apeltauer]

I just figured out that my branch is corrupted since a while. After reverting to a clean stand, everything works fine.....
Sorry for the noise.....

[gflorent]

Yes, you can add your tests to the Test WB without changing its source. However, a few things must be polished a bit to improve the integration. Assuming you have a module called MyTest which resides in the Mod directory then you can do the following things:

• Create a file __init__.py
• Create a directory Tests with another __init__.py and e.g. TestCase.py
• The content of Tests/__init__.py can e.g. be:

  Code: Select all

  import QtUnitGui
  QtUnitGui.addTest("MyTest.Tests.TestCase")
  

• and TestCase.py

  Code: Select all

  import unittest
  
  class MyTestCases(unittest.TestCase):
    def setUp(self):
      pass
  
    def testNothing(self):
      self.assertTrue(True)
  
    def tearDown(self):
      pass
  

I just tested that and it works really well. Now, for a real life WB, I think it is faster to run the tests from IDE/Command line as you do not have to restart FreeCAD each time (or maybe I missed a kind of 'reload' function for workbenches). Yet in IDE/Command line tests, you need something like the following snippet, that is platform dependant (easy to manage) but also installation dependant (devs will not always run FreeCAD from the same folder, so committing such a path will not work out of the box for another dev after a clone).

Code: Select all

sys.path.append("C:\\Program Files\\FreeCAD 0.17-13528\\lib")

If you run from the FreeCAD UI, the path to the lib folder is obviously not an issue as all the imports have been done on startup; yet, the test-correct-test again workflow is slower. Maybe the best way is to run from IDE/Command line in early stages and to move to to the UI runner when things get more stable.

[ceremcem]

Wouldn't macros perfectly fit for this purpose?

(Original conversation, Asm3 discussion)

TL;DR;

"User definable tests" Idea is as simple as follows:

1. Start recording a macro. (Might be default for any project in the end and those "logs" will be saved like the project file.)
2. Create a part from scratch or load it from a file.
3. Perform relevant operations.
4. Stop the macro recording.
5. Create a Spreadsheet named Tests that will hold the expected values (lengths, angles etc.). It should contain "Description" (optional), "Expected" and "Observed" columns. Expected column should be hard coded and Observed column will be calculated by a formula.
6. Use another macro, named "Test.FCMacro", which will simply "Compare the Expected and Observed column values in the Tests Spreadsheet.

This proposal actually makes things boil down to the point: Macros ("logs") become the actual file format and FCStd files become the "cache" files. I guess this approach lets virtually an unbreakable file format. With tests per project, we can use/test any FreeCAD commit in production without fear.

How does it sound?