Making projects resistant to breaking changes

[ceremcem]

Problem Description

We need to protect our projects/works from getting broken by breaking changes, even after our projects are archived. If the breaking change is inevitable, we need to be notified at the time of introduction of the breaking change, in order to reduce the refactoring costs by early intervention.

A complex project such as FreeCAD will introduce breaking changes sooner or later during the development.

Proposal

Based on Original conversation and Asm3 discussion.

TL;DR;

To address such an issue, there should be a way to turn our projects into "user defined tests". Here goes my proposal:

1. Start/resume recording a macro during project modifications (macro name: ${filename}.logs.FCMacro).
2. Create/edit project as usual.
3. Pause the macro recording.
4. Create a Spreadsheet named Tests that will hold the expected values (lengths, angles etc.). It should contain "Expected" and "Observed" columns. Expected column should be hard coded and Observed column will be calculated by an assembly measurement constraint.
5. Use a simple "Test.FCMacro" which will simply "Compare the Expected and Observed column values in the Tests Spreadsheet".

The ${filename}.logs.FCMacro actually becomes the "source code" of the current output (FCStd files). Only format I can think of that can not be broken in time (?) is "the verbal description of the model". (Hello DeclaraCAD!)

Selling Point

According to my experiences, people want to stay away from a software in production environments that gives no guarantees for (or even tends to have frequent) "breaking change" cases. I have those reservations as well.

However, by adopting the "tests per project" approach, we can use/test any FreeCAD commit in production without fear. That will also speed up FreeCAD and Workbench developments because more test cases will be created by the users by using FreeCAD more by using FreeCAD in production.

Moreover, the end users will benefit from their "project tests" during/between the development cycles of their very own work, because any new product feature might interfere with early constraints of the product description in the first place.

These "logs" can also be used while creating the model manually from scratch in case of a disaster situation (like switching to a totally different software). Having such an option will make the company officials feel safe while considering trying the FreeCAD in a small project.

RFC
How does it sound? What would be the complications?

[suzanne.soy]

I think that having some sort of Continuous Integration process that checks whether a bunch of projects are still "working" in FreeCAD+workbenches is a very good idea. It's even possible to do that as a separate repository (or many separate repositories): just download the .AppImage release when there is a new one, run the tests, and log all errors.

FreeCAD's macro recording doesn't seem reliable enough to record the creation of an entire document, there are sometimes small steps which are not recorded, any missed one would break the reconstruction of a model. Also, that would keep track of all the temporary features, history undo etc. The real "source code" for a FCStd file is… the FCStd itself, it's just an .xml file that describes the model tree + properties (and contents of sketches or non-parametric geometries) that are visible in FreeCAD's combo view.

So I think getting a full macro trace on how a model was created probably won't be possible (not now, and not for a while; macros can just poke at the model without having to record anything in the macro trace, for now it's just "good practice" to include the main steps to replay the change), but loading a document, calling recompute, maybe performing a couple of recorded steps, and checking the dimensions and coordinates of some elements should be very possible.

I just wonder if there's some sort of feed from which to get the latest FreeCAD release in order to trigger CI jobs on GitHub / GitLab / …

[Kunda1]

I like where this thread is going!

[suzanne.soy]

I just wonder if there's some sort of feed from which to get the latest FreeCAD release in order to trigger CI jobs on GitHub / GitLab / …

These two posts from September and October 2019 saying that this is not possible for arbitrary test repos.

* https://stackoverflow.com/questions/584 ... ew-release
* https://github.community/t/triggering-b ... tory/16163

The repository-dispatch action allows the dependencty (FreeCAD) to notify one or more dependents (test repos), but these need to be hard-coded. It would work for a single centralized test repo, but it won't work if many users keep their own test repo.

Alternatively, a test repo could use a recurring (cron-style) CI job to check if there have been any new releases, and exit without further work otherwise. Should be fairly easy to set up except for saving the already seen flag, github actions don't have facilities for persisting state between jobs (solutions to circumvent include: publishing an artifact, e.g. the resulting .FCStd after opening+saving with a new version, or creating a dummy commit / tag to use git itself as a storage mechanism).

This brings the question: do we want a single central repo with lots of models and their tests (allows devs to quickly see that 0.5% of the models broke after an update and track down the issue), or do we want users to keep their own models in their own repos, and it is their responsibility to notify the devs of breakage?

[ceremcem]

FreeCAD's macro recording doesn't seem reliable enough to record the creation of an entire document, there are sometimes small steps which are not recorded, any missed one would break the reconstruction of a model.

That's actually not a problem. The "unrecorded small steps" can even be interpreted as a "feature" in this context.

There is no need to log every Python command to the Python console. If all necessary parameters are set, a simple "Gui.runCommand('SMUnfoldUnattended')" command will suffice. We wouldn't be interested in "how" the command does its job, we would only be interested in "what is done".

FreeCAD's macro recording doesn't seem reliable enough to record the creation of an entire document

That's the point. If FreeCAD officially accepts this proposal, any WB that can not recreate the model via its own macro output will be considered buggy. Actually that can not happen after the first time the WB's fixed their problem, because all other tests will be depend on this feature. A developer even can not push his/her commits while his/her WB is failing to pass the tests.

This brings the question: do we want a single central repo with lots of models and their tests (allows devs to quickly see that 0.5% of the models broke after an update and track down the issue), or do we want users to keep their own models in their own repos, and it is their responsibility to notify the devs of breakage?

The only available choice is letting a distributed test repo setup approach because users wouldn't want their proprietary models live in a foreign location. We could keep a public repository which will hold the "Reproduction of edge cases". Users will run their tests offline, prior to a FC image/commit update. The errors will be handled in a semi-automatic fashion: When a test (project) fails during an offline test, the backtrace will be mailed to the user's own e-mail with a cover letter, saying "Following traceback is collected when your project-xyz is failed to be recreated against FC-version-abc. Should we send the error to the developer?". If you approve, a mail with a tracking ID will be sent to the developer. (We'll eventually figure out something for this matter, I'm sure of it.)

Devs will test their changes against the public test cases repository before pushing to upstream.

Alternatively, a test repo could use a recurring (cron-style) CI job to check if there have been any new releases, and exit without further work otherwise. Should be fairly easy to set up except for

That's right, a polling mechanism will make everything easy as pie. For the "except for" part: I can handle it, no worries.

[ceremcem]

The real "source code" for a FCStd file is… the FCStd itself, it's just an .xml file that describes the model tree + properties (and contents of sketches or non-parametric geometries) that are visible in FreeCAD's combo view.

I intentionally named it as "source code" because it will contain more data than the XML file, such as "Selected face" for creating a SheetMetal bend. (Currently Sheet Metal WB doesn't store the selected face for unfold operations. However, that might change in the future. For example, I'm currently storing that information in Assembly3 containers in my projects.)

Another information where the XML file can't contain (in current fashion) is the starting state of an assembly solving operation, see this one.

My point is that a model creation can not be "replayed" by looking at the FCStd file.

[suzanne.soy]

A developer even can not push his/her commits while his/her WB is failing to pass the tests.

WB developers are pushing to their own repo on GitHub or elsewhere, there's no way to impose that (I'm not saying it's not a good practice in general, just that it is still possible to break compatibility, and I expect it to happen sometimes for good reasons (e.g. conversion to the new format requires human intervention to disambiguate when the past specification was imprecise)).

The only available choice is letting a distributed test repo setup approach because users wouldn't want their proprietary models live in a foreign location.

This also prevents these tests from causing build failures when a WB developer, macro developer or core developer pushes an update (as you said, the errors would be sent manually by e-mail at a later date, so "can not push their commits while their WB is failing to pass the tests" won't apply for these private tests, which makes sense).

Overall your suggestions seem cool, though I expect implementing the full pipeline (reproducible macro-like traces, with automatic e-mails and auto-send of reports to developers) would require quite a lot of work, with a lot of attention to detail. It would be more achievable in the short term to start a simpler goals (poll for updates, load a document, recompute, check dimensions in spreadsheet, fail the build if something goes wrong) and improve from there.

I'm all for exact macro traces, they're also helpful when writing our own macros. Recently however, for my own projects, I've grown more fond of a fully-parametric approach: use the appropriate tricks to avoid or circumvent topological naming issues, and use only operations which create a parametric node in the document tree. That way, the document tree *is* an accurate description of the history of changes through the document, and modifying earlier stages of that history affect the later ones. Talking about the sheet metal WB, I'm tempted to modify the unfold to get a parametric unfold, and add a couple of operations to have a more declarative model (start with an unfolded sketch, mark some edges as "fold edges" with an angle, mark pairs of edges sharing a corner as "glue together", and it should fold according to the instructions whenever the sketch gets updated).

[realthunder]

Yes, we need to have something that checks for model breakage on FreeCAD source code changes. But we have to keep our expectation realistic here. I don't think it is possible or even necessary to demand that FreeCAD should be able to repeat user's steps of modeling to reproduce the model. It will put too much restriction on development.

I think we can have several levels of test requirements here,

1) The old model can still be loaded, and gives the same shape on restore.
2) Doing a full recompute, and the old model still gives the same shape.
3) Modify some model parameters and the model still gives expected result.

Maybe we can take a look at what OCCT is doing for consistency testing. They must have something similar. In fact, we have to rely on their consistency to even be possible to test level one requirement here, and this is where the complexity is, i.e. OCCT changes may break our test especially level 2 & 3, making it hard to judge whether it is the test or FreeCAD that's broken. And once it is determined that OCCT causes the breakage, mostly like we have to rebuild our test cases. We don't know how often does this kind of OCCT change happens. You know with upstream FreeCAD, this type of problems are largely hidden, because user seldom perform a full recompute on old model loaded using newer OCCT.

Another complexity here is that we have many different type of objects in different workbenches, and may have very different test requirement.

[thomas-neemann]

Does it make sense to create a long-term export format in addition to or as a replacement for this project, e.g. the bodies are exported as a step and the complete sketches are also written into the file?
this would mean that most of the working hours would have to be protected for years.

[realthunder]

Does it make sense to create a long-term export format in addition to or as a replacement for this project, e.g. the bodies are exported as a step and the complete sketches are also written into the file?
this would mean that most of the working hours would have to be protected for years.

But STEP file does not capture design intentions. All those parameters weill be lost. May be the GD&T section of the STEP format can store some information as parameters. Right now FreeCAD does not support GD&T yet. I do plan to add the support one day, but it's at a lower priority.