External full Python WB Unit Tests

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
User avatar
gflorent
Posts: 49
Joined: Fri Aug 10, 2018 10:05 am
Location: France
Contact:

External full Python WB Unit Tests

Postby gflorent » Thu Aug 23, 2018 3:46 pm

I have some questions, for the more experienced folks with FreeCAD, regarding the preferred unit testing strategy for external workbenches.

1. Is any code organisation and testing framework ok (as long as it works) or should an organisation and framework (unittest I guess) be preferred for potential integration with the Test WB?

2. Is it possible to add a test suite for an external WB to the Test WB (without editing the Python code in the Test WB of the installed FreeCAD) or should the usual test runners from a testing framework be used?

3. Should the TemplatePyMod way of writing tests be considered as the example to follow to write tests (since the FreeCAD/Workbench-Starterkit repo does not seem to mention tests)?

Maybe this is all explained somewhere but I could not find the info.

Thanks in advance to anyone who may help with that.
User avatar
DeepSOIC
Posts: 6965
Joined: Fri Aug 29, 2014 12:45 am
Location: Saint-Petersburg, Russia

Re: External full Python WB Unit Tests

Postby DeepSOIC » Thu Aug 23, 2018 5:06 pm

I hope to be proven wrong... but it looks like, you can't. The test modules are explicitly hard-coded here:
https://github.com/FreeCAD/FreeCAD/blob ... App.py#L56

Code: Select all

def All():
    # Base system tests
    tests = [ "UnicodeTests",
              "Document",
              "UnitTests",
              "BaseTests" ]

    # Base system gui test
    if (FreeCAD.GuiUp == 1):
        tests += [ "Workbench",
                   "Menu" ]

    # add the module tests
    tests += [ "TestFem",
               "MeshTestsApp",
               "TestSketcherApp",
               "TestPartApp",
               "TestPartDesignApp",
               "TestSpreadsheet",
               "TestTechDrawApp",
               "TestPathApp" ]

    # gui tests of modules
    if (FreeCAD.GuiUp == 1):
        tests += [ "TestSketcherGui",
                   "TestPartGui",
                   "TestPartDesignGui",
                   "TestDraft",
                   "TestArch" ]

    suite = unittest.TestSuite()

    for test in tests:
        suite.addTest(tryLoadingTest(test))

    return suite
I guess, you can hack in by overwriting the function All() to inject your own tests, but that doesn't sound great.


EDIT: even worse, it's hard-coded yet again here:
https://github.com/FreeCAD/FreeCAD/blob ... Gui.py#L43
User avatar
gflorent
Posts: 49
Joined: Fri Aug 10, 2018 10:05 am
Location: France
Contact:

Re: External full Python WB Unit Tests

Postby gflorent » Mon Aug 27, 2018 4:09 pm

There is a nice solution in the InitGui.py of https://github.com/microelly2/freecad-nurbs near line 140. The same dialog as the one used in the 'official' Test WB appears (when clicking a 'Test' command) with a drop-down list of unit tests scripts.

Here is the snippet extracted from InitGui.py

Code: Select all

class MyTestCmd2:
    """Opens a Qt dialog with all inserted unit tests"""

    def Activated(self):
        import QtUnitGui
        QtUnitGui.addTest("nurbswb.TestNurbsGui")
        QtUnitGui.addTest("nurbswb.TestNurbs")
        QtUnitGui.addTest("nurbswb.TestMeinAll.Col1")
        QtUnitGui.addTest("nurbswb.TestMeinAll.Col2")
        QtUnitGui.addTest("TestMeinAll.Col2")

    def GetResources(self):
        return {'MenuText': 'Test-test...',
                'ToolTip': 'Runs the self-test for the workbench'}


FreeCADGui.addCommand('My_Test2', MyTestCmd2())
But, unfortunately, I could not find the actual tests scripts in microelly2's repo because of the following line in the .gitignore

Code: Select all

Test*py
@microelly2: I like your testing solution in the Nurbs WB, and I am sure many others will, would you mind sharing the test files?
User avatar
DeepSOIC
Posts: 6965
Joined: Fri Aug 29, 2014 12:45 am
Location: Saint-Petersburg, Russia

Re: External full Python WB Unit Tests

Postby DeepSOIC » Mon Aug 27, 2018 4:50 pm

I have a different plan. The plan is to make a specialized tester document object, which is to remember and verify the result of a sequence of features. Then I am to make a test project, where I use each feature, visually verify the correctness, and attach a tester object to remember it. And the test is then to load the project, and force-recompute it.

This will probably work OK for Lattice2, which is very feature-centric. But it won't work on part-o-magic, as it is more of a gui/interactivity workbench.
User avatar
gflorent
Posts: 49
Joined: Fri Aug 10, 2018 10:05 am
Location: France
Contact:

Re: External full Python WB Unit Tests

Postby gflorent » Mon Aug 27, 2018 7:02 pm

Maybe we can combine both. The tests ran from the test runner could either check automatically that placement and other properties are as expected or display a dialog where you can click OK or NOT OK based on what you see and controlling the test result. What do you think?
User avatar
DeepSOIC
Posts: 6965
Joined: Fri Aug 29, 2014 12:45 am
Location: Saint-Petersburg, Russia

Re: External full Python WB Unit Tests

Postby DeepSOIC » Mon Aug 27, 2018 7:05 pm

gflorent wrote:
Mon Aug 27, 2018 7:02 pm
display a dialog where you can click OK or NOT OK based on what you see and controlling the test result.
If I see it's NOT OK, there's no point clicking the button. I can go straight ahead and debug/fix it.
User avatar
sgrogan
Posts: 5330
Joined: Wed Oct 22, 2014 5:02 pm

Re: External full Python WB Unit Tests

Postby sgrogan » Mon Aug 27, 2018 9:20 pm

gflorent wrote:
Mon Aug 27, 2018 7:02 pm
Maybe we can combine both. The tests ran from the test runner could either check automatically that placement and other properties are as expected or display a dialog where you can click OK or NOT OK based on what you see and controlling the test result. What do you think?
FreeCAD must run without the gui. So these type of tests should only run when the gui is up.
wmayer
Site Admin
Posts: 14782
Joined: Thu Feb 19, 2009 10:32 am

Re: External full Python WB Unit Tests

Postby wmayer » Tue Aug 28, 2018 9:43 am

1. Is any code organisation and testing framework ok (as long as it works) or should an organisation and framework (unittest I guess) be preferred for potential integration with the Test WB?
If everything works with your own testing framework then I think it's fine, too.
3. Should the TemplatePyMod way of writing tests be considered as the example to follow to write tests (since the FreeCAD/Workbench-Starterkit repo does not seem to mention tests)?
The file Tests.py shows how to basically write a unit test, i.e. inherit from unittest.TestCase, offer a setUp and tearDown method and offer methods starting with "test"
2. Is it possible to add a test suite for an external WB to the Test WB (without editing the Python code in the Test WB of the installed FreeCAD) or should the usual test runners from a testing framework be used?
Yes, you can add your tests to the Test WB without changing its source. However, a few things must be polished a bit to improve the integration. Assuming you have a module called MyTest which resides in the Mod directory then you can do the following things:
  • Create a file __init__.py
  • Create a directory Tests with another __init__.py and e.g. TestCase.py
  • The content of Tests/__init__.py can e.g. be:

    Code: Select all

    import QtUnitGui
    QtUnitGui.addTest("MyTest.Tests.TestCase")
    
  • and TestCase.py

    Code: Select all

    import unittest
    
    class MyTestCases(unittest.TestCase):
      def setUp(self):
        pass
    
      def testNothing(self):
        self.assertTrue(True)
    
      def tearDown(self):
        pass
    
If you now import MyTest.Tests then the tests will be registered and the dialog comes up to run the unit test.

At the moment I haven't found a way to register a test (preferably at startup) without invoking the test dialog. I think this can be done by providing a simple list in the FreeCAD and FreeCADGui modules which stores the module names which will be considered in TestApp.py and TestGui.py. This way you could add tests directly in your Init.py or InitGui.py files.
wmayer
Site Admin
Posts: 14782
Joined: Thu Feb 19, 2009 10:32 am

Re: External full Python WB Unit Tests

Postby wmayer » Tue Aug 28, 2018 11:19 am

With git commit 67b5ee0931 now there are no lists in TestApp.py and TestGui.py but instead each module registers its unit tests via Init.py or InitGui.py. So, with external module it has become much easier to register tests. All what you have to do is adding this line to Init.py or InitGui.py:

Code: Select all

FreeCAD.__unit_test__ += [ "Name_of_my_test_module" ]
User avatar
DeepSOIC
Posts: 6965
Joined: Fri Aug 29, 2014 12:45 am
Location: Saint-Petersburg, Russia

Re: External full Python WB Unit Tests

Postby DeepSOIC » Tue Aug 28, 2018 12:28 pm

wmayer wrote:
Tue Aug 28, 2018 11:19 am
With git commit 67b5ee0931 now there are no lists in TestApp.py and TestGui.py but instead each module registers its unit tests via Init.py or InitGui.py. So, with external module it has become much easier to register tests. All what you have to do is adding this line to Init.py or InitGui.py:

Code: Select all

FreeCAD.__unit_test__ += [ "Name_of_my_test_module" ]
Oh, wow, thanks!