Test suite for Path Workbench

[chrisb]

I think it is sensible to have some kind of a test suite for the Path WB. This is meant as a help for the developers by moving some work to the community, and users can contribute before becoming PathWB experts.
I am thinking of a set of mostly simple objects with mostly simple paths like a cube with a square pocket, a cube with a circular pocket, the same with pads and so on.

All test cases should use the same toolset. Additional tools can be added, they must not replace existing tools.
Every test case should have:
- a FreeCAD model, (this is reusable as long as Part Design Next doesn't basically change)
- a description on how to involve Path WB
- a description of the expected result, preferably with a picture if available.

I would opt for a set of very simple models because it is easier to find a fault and additionally they can easily be rebuilt from scratch. More complicated models should be involved only if otherwise not testable.
Who will join? The developers should make requests, what kind of test cases they would like to have first.

As an important extra benefit we would get some kind of a documentation for the path WB.

[sliptonic]

I really like the idea.
Markus and I have had some discussion via IRC about testing recently. I know very little about Python unit testing but he has a lot of experience and has started writing unit tests for his dressups. Those are good examples that I, and other developers, should be able to follow going forward.

I think the most useful test shapes would be those that can be created in script. Then we can use that script in the setup() method for the unit test cases.

If there are more complex shapes that we can't build in script, I don't think we want to be using a complete FreeCAD document. We should be testing whether we can generate a valid path from a shape not whether a pre-created Job works the way it used to. That might be useful testing but we're not that stable yet. Path Jobs and operations often break between versions right now and compatibility might also break for reasons outside Path.

Also, I think we should keep test data as small as possible and a whole document is probably larger than needed. Based on some crude testing I did, I think the objects should be in .brep format. (Objects exported in .step / .iges files were quite large.)

[sgrogan]

Also, I think we should keep test data as small as possible and a whole document is probably larger than needed. Based on some crude testing I did, I think the objects should be in .brep format. (Objects exported in .step / .iges files were quite large.)

There is the possibility for a hybrid approach. The FEM workbench, in addition to the unit tests run as apart of the CI process also has an example project in the StartPage. This can be a full .FCstd document that need not be tested as part of the automated unit tests, but provides a common starting point for bug triage. This file need not always be up to date. When a potential problem is discovered, everyone works from the same base line. "Load the example file delete this add that etc." It works very well in FEM in combination with the regression tests. I think the two workbenches share a common development arc, very fast development, more developers becoming involved, and a user base that my not have programming skills but are eager testers.

[mlampert]

The current plan is to add a new entry into FC's Test WB, so anybody will be able to run the tests.

What we're struggling with is a concept of how to test UI issues. We can write tests for the logic and algorithms but ... if anyone has experience in writing tests for the UI and how to detect regressions there .... any pointers appreciated

[chrisb]

I see the advantage of automated tests, as they are immediately available to the developers and do not depend on additional mistakes made by the tester. However, I cannot contribute (yet). So my approach remains as a pure educational one where eager users may contribute. Herbert, you want to jump in?

[bill]

Glad to see this Idea resurface; it would be very useful. I suggested the concept a while back ( viewtopic.php?f=15&t=14511 ) but no one grabbed on to it.

Generating the machining testing environment by script, very, very nice!

[mlampert]

I see the advantage of automated tests, as they are immediately available to the developers and do not depend on additional mistakes made by the tester. However, I cannot contribute (yet). So my approach remains as a pure educational one where eager users may contribute. Herbert, you want to jump in?

Oops. it seems the intent came out wrong - sorry about that! I did not intent/hope or suggest to replace you or your expertise with automated tests - in fact I don't think that's even possible. The hope is that automated testing detects regressions and prevents me from introducing bugs we've already fixed, or breaking things that did work before. They cannot be a substitute for experienced users though

[herbk]

Hi,
some automatet testings could not be bad to find bugs in programming work, like mlampert (Markus?) said. My skills in programming are, maybe level -2 (ore something like that), and i'm very happy for each automatet improving (for my little arduino sketches) i can get...
In my mind one "problem" is, that the guys with the experience in programming not can have the experience in machining, for both the day hase not enough hours..
I have a little bit experience in optimizing workflows (thats what i make a part of my money with) so i can help (i hope so) to show of what i notice.

[sliptonic]

I have a little bit experience in optimizing workflows (thats what i make a part of my money with) so i can help (i hope so) to show of what i notice.

Thanks herbk, I appreciate your input. 'Optimizing workflows' in software application, especially one like FreeCAD is a slippery subject full of trade-offs.

For example, We could put a lot of effort into optimizing the user interface to be very fast at creating jobs and operations. However, this might come at a cost of more errors. Conversely, if the application requires many things from the user to avoid those errors, it could be cumbersome. Obviously there are better and worse ways to build the UI and we try to balance both needs but you see my point. In the case of CAM, I think it is more important to be accurate than fast. Mistakes in a tool path can be both expensive and dangerous.

Another example: Should the CAM workbench be optimized as part of an overall CAD/CAM/CAE tool set in FreeCAD or should we optimize with only the CAM user in mind? If a person only uses FreeCAD for the CAM workbench he might think some things are cumbersome but if he also does design and analysis he might think it is very efficient.

One more: Should we design assuming the user will be using the GUI or should we build the system to also work 'headless' as part of a scripted toolchain? Working headless is very important for some but might mean that compromises must be made.

My personal preference is to focus first on building:
1) accuracy
2) robustness
3) flexibility
4) speed (computation)

With a solid foundation, we can add features and GUI improvements forever.

[herbk]

Good morning sliptonic,

i read you post several times now, and a lot you sayd is so "of course", that i fore some points not realy understand what you want to explain.

In my mind CAE is the "whole thing" and CAD, FEM and CAM are parts of it, and each part is the specialist for what it should do. But from the spezialist i'm expecting that he takes the attention at that, what he is the spezialist for, first.
That the whole CAE process needs a common base, and for that needs compromises, is a "of course" ...

Should we design assuming the user will be using the GUI or should we build the system to also work 'headless'

I do not know what you mean with "headless", but for the first part of the question: I guess about 99,8723% of the USERS like to have a well done GUI with an intuitively recognizable workflow.
And on this "guessing" a important point i'm missing in your preverences list: The usability/workflow. (But that's thomething what i'm (not only i) missing in the hole FreeCAD)
For me to set to point 2 of the list, the rest i agree with.