[Ubuntu Daily PPA] Transitioning to OCCT7, VTK7...

[sgrogan]

OK I will try to start with the history of the new dependencies and OOC7 support. Unfortunately the time line is not quite linear, so I will endeavour to not get too convoluted. I know the thread is specific to the Ubuntu PPA but I will mention some issues encountered on other platforms/Distros that may be relevant.

Shortly after 0.16 release VTK post-processing was introduced to the FEM WB. This created a soft dependency to libvtk6-dev. v6.2 or greater was required. There was a USE_VTK cmake switch that activated this feature. This was supported on Windows with VTK 7.0 being added to the libpack. The PPA never supported this, mainly because the Trusty VTK6-dev package was v6.0 and a backport was necessary. I did locally compile a minimal VTK7 and built FreeCAD successfully build FreeCAD with the FEM post-processing on Trusty.

Next vejmarie began FreeCAD's port to SMESH 7.7.1. SMESH is 3rd party code included in FreeCAD master. This created a hard dependency to OCCT7 and libmedC-dev. This work wasn't accepted in master at this point because it would have required self-compilation of OCCT7 and netgen on all platforms. At this point ickby joined vejmarie and they began patching the SMESH sources to support older versions of OCC.

At this point wmayer ported FreeCAD to OCCT7. This was supported on Windows (including the FEM post-processing using the new VTK7 in the libpack). Netgen support on Windows was broken at this point and still is. I'm not sure about OSX. This was not supported on the PPA but did point out a weakness in FreeCAD's cmake when trying to link against OCCT7 when there was a system installed version of OCE. I hack into FreeCAD's cmake make was required to link against OCCT7. I don't know if there is the same weakness if there is no system installed OCE. I know looo has looked at this, with a pull request that was later retracted. I've also looked at this, and discovered some unexpected behaviour even with different versions of OCE installed.

At this point ickby and vejmarie were successful with their patches, ianrees and bblacey joined in to make it work on OSX, and their work was commited to master. This created a dependency to libvtk-dev >= 6.0 (>=6.2 to activate the FEM post processing) and libmedc-dev >= 3.0.6. OSX supported this immediately with "bottles" ,self-compiled OCCT7-dev, VTK7-dev, vejmaries patched Netgen 5.3 and libmedc-dev v 3.1.0. The PPA supported this by adding libvtk6-dev and libmedc-dev and the FreeCAD maintainers OCE package. This revealed a problem with the Xenial libVTK6-dev package (there is an unmet dependency to libproj-dev) Trusty which has libVTK6-dev v6.0 has FEM post-processing disabled automatically by FreeCAD's cmake.
EDIT: here the dependency is drawn in by SMESH and is needed by MESH_PART and FEM.

I hope this is not too confusing. A key point is the inter-relation between the versions/packages of OCCT/OCE, NetGen, VTK, MedC, and hdf5 pulled in by MedC is important.

[sgrogan]

And to answer some of NormandC's questions more specifically:

Open CASCADE now relies on VTK (this is not needed as FreeCAD uses VTK independently from OCC), FreeCAD does as well, we may need to create a VTK7 package first

I think VTK can be done as a second stage if necessary. OCCT7 can be compiled without VTK, it causes problems and we don't use it.

The CMakeLists.txt file sets cmake 2.8.12 as the minimum required version, trusty barely complies with the same version. This means it won't build on Ubuntu 12.04 LTS, but I believe we stopped supporting it anyway.

The support on precise goes deeper it requires non package manager versions of gcc, cmake, libboost-dev, libmedc-dev, lib-vtk-dev I believe wandererfan is still using 12.04 but he's a developer and I'm sure has heavily modified his version to make this happen.

Should we create a git repo on Launchpad for the debian packaging? I guess it would make sense.
Should we also clone the OCC git repo to Launchpad? Or simply put everything, the upstream sources and the debian folder into the same repo? Kind of like the Debian Science guys are doing: https://anonscm.debian.org/cgit/debian- ... /Regarding the package naming, I'm really not fond of how the liboce runtime packages get an arbitrary number bumped for each OCE release (liboce-foundation10, liboce-modeling10, etc.). I propose we add the real OCC version number to the package name: libopencascade7 (without minor version and maintenance version digits since they are zero). When OCC is updated to 7.1, we can name the new packages libopencascade7.1. Minor maintenance version digits (i.e. 7.0.1) should not change the package name, we'd just provide an updated package. But I doubt this will happen, OCCT does not seem to publish maintenance versions.

Cloning (or mirroring) the OCCT7 repo requires permissions (signing up to OCCT, accepting the developer agreement, etc) We can download a snapshot, initialize as a local git repo, and push to Launchpad. Also OCCT has switched to a cmake build system, a quick look indicates that the default cmake install pre-fixes shows differences between OCCT and OCE. Probably not insurmountable, but another issue. I haven't looked closely at this.

I seem to recall that vejmarie needed to patch netgen 5.1 to get it working. Is it still the case? (Found this patch from looo, what about wmayer's?)

vejmarie is patching Netgen 5.3. Probably our best bet (the OSX guys are using this). I haven't been able to make any of these patches work on Windows. looo's work is intersting because it uses Netgen 6.x that adds cmake support, this is desirable for long term support.

What about VTK7? Does it need to be patched too?

VTK7 doesn't need to be patched, but I haven't been able to locally compile FreeCAD with VTK7 and OCCT7 on Trusty. I believe it's related to libmedc-dev. vejmarie compiled libmedc-dev v 3.1.0 to make this work. With libmedc-dev v 3.0.6 there are mpi related linking errors. I've seen similar on Debian Testing and Ubuntu Yakkety.

Adding support for TBB parallel processing (libtbb2?) along with vejmarie's patch to improve read time on STEP files (topic: Re: STEP file read might be crazy long ...)
TBB might also be useful for Boolean operations: OCCT Forum / Current Progress - Parallelization of Boolean Operations Algorithm

Yes! There are big claims that multi-thread support is greatly improve in OCCT7. DeepSOIC has indicated that some changes in FreeCAD code are required to activate the features. He described them as "straight forward" I found some tbb pre-compiled binaries for the correct compiler on Win but I haven't tried to re-compile OCCT yet. I think if it isn't a show stopper it's worth having the capability there.

Thanks Norm for kick starting this! It seems my skills are still not what they need to be (Win or Ubuntu):oops:

[NormandC]

Thanks for the very detailed summary, sgrogan.

Frankly, it's worse than I thought... And to be blunt, the only word that comes to my mind to describe the whole FEM situation starts with cluster and ends with f...

We can download a snapshot, initialize as a local git repo, and push to Launchpad.

It seems silly to me that we can't clone the OCCT repo but we can download a snapshot and push it to a public repo.

looo's work is intersting because it uses Netgen 6.x that adds cmake support

Doesn't netgen 6.x require python3? That's the impression I had when I looked at it a few months ago.

vejmarie compiled libmedc-dev v 3.1.0 to make this work. With libmedc-dev v 3.0.6 there are mpi related linking errors.

But didn't you write earlier that libmedc-dev 3.1.0 needed to be patched as well?

vejmaries patched Netgen 5.3 and libmedc-dev v 3.1.0.

A key point is the inter-relation between the versions/packages of OCCT/OCE, NetGen, VTK, MedC, and hdf5 pulled in by MedC is important.

At this point I would like to know what exact version of each of those libraries is known to work and activate the full FEM functionality. Can some of them be used straight from the Xenial repos, apart from libmedc? Can we simply create a libmedc 3.1.0 package linked to the regular hdf5 from the Ubuntu repo?

Determining that, we could attempt to backport those versions to trusty. Right now trusty and xenial are the only two supported Ubuntu versions until Yakkety comes out next month. Support for wily ended July 28th so we don't need to bother with it.

Thanks Norm for kick starting this!

Don't thank me yet...

[wandererfan]

The support on precise goes deeper it requires non package manager versions of gcc, cmake, libboost-dev, libmedc-dev, lib-vtk-dev I believe wandererfan is still using 12.04 but he's a developer and I'm sure has heavily modified his version to make this happen.

My recipe for 12.04:

gcc 4.9.4 ppa http://ppa.launchpad.net/ubuntu-toolcha ... st/ubuntu/
cmake 3.2.3 ppa http://ppa.launchpad.net/george-edison5 ... ts/ubuntu/
boost 1.55 ppa http://ppa.launchpad.net/boost-latest/ppa/ubuntu/
vtk 6.3 local compile
OCCT 7.0 local compile
OCE-0.17 local compile (but I think this is the one in the Ubuntu repository)
libmedc/hdf5 etc is from the Ubuntu repository.

16.04 box is sitting on the desk, but still in configuration stage.

wf

[looo]

Doesn't netgen 6.x require python3? That's the impression I had when I looked at it a few months ago.

smesh doesn't need the netgen python stuff. So you can disable python with cmake. This is how I have done for conda packages: https://github.com/looooo/FreeCAD_Conda ... n/build.sh (Theoretically it could also work with enabling the python3 bindings.)

It seems silly to me that we can't clone the OCCT repo but we can download a snapshot and push it to a public repo.

It's silly, but I think it is not a bad idea to have a occt copy inside the FreeCAD-githup project. (patching, testing, better access to the source)

regarding libmedc:

With conda I use this one:
source:
git_url: https://github.com/vejmarie/libMED.git
git_tag: master

I think this is the patched version. Compilation didn't made any problems.

[sgrogan]

source:
git_url: https://github.com/vejmarie/libMED.git
git_tag: master

I think this is the patched version. Compilation didn't made any problems.

I don't think this is patched. I believe vejmarie created this to use with his vagrant script. There is no MEDC repo either and the zipped sources need to be down loaded from the Salome-Meca site.

I think Xenial should work with a combination of FreeCAD PPA and stock packages from the repo. I suggest we try to build OCCT7 and Netgen only and test it out. I recently acquired a decommissioned XP computer that I will try to find some time to install Xenial on. No good for compiling but it will offer a nice clean install to test on.

[NormandC]

@ wandererfan & looo, thanks for the additional info.

I think it is not a bad idea to have a occt copy inside the FreeCAD-githup project.

Just to clarify, we are not talking about the FreeCAD Github here, but about creating a Launchpad git repo associated with the FreeCAD Maintainers team, and specifically to package OCCT for Ubuntu. I'm not sure it's a good idea to include OCCT in FreeCAD sources. If we go there, do we also include the sources from netgen, vtk, medc, hdf5? Were does it end? Also this would really add bloat to the repo...

I suggest we try to build OCCT7 and Netgen only and test it out.

Okay, sounds like a plan. I'm only interested in OCCT7 myself anyway.

Not sure if I'll be able to this week, I'll start from the current OCE debian folder and modify it as I mentioned. I guess it would be a good idea for me to compile OCCT first just to see how it works...

[sgrogan]

Not sure if I'll be able to this week, I'll start from the current OCE debian folder and modify it as I mentioned. I guess it would be a good idea for me to compile OCCT first just to see how it works...

Thanks for the help. Don't worry about time. "It's done when it's done"
My work has a surge and were working 12 hour days.
Do you think the OCCT git repo should be posted on the FreeCAD Maintainers? I don't fully understand the implications with the oce packages?
Thanks again Norm for jumping in. Please don't stress. We have a lot of time until the next release. I do really appreciate your time. I learn a lot from for direction. Sorry if I seem a little slow on the uptake. Your modesty aside, you are THE FreeCAD packaging expert

[looo]

Just to clarify, we are not talking about the FreeCAD Github here, but about creating a Launchpad git repo associated with the FreeCAD Maintainers team, and specifically to package OCCT for Ubuntu. I'm not sure it's a good idea to include OCCT in FreeCAD sources. If we go there, do we also include the sources from netgen, vtk, medc, hdf5? Were does it end? Also this would really add bloat to the repo...

No, not include occt with the FreeCAD source. My post was about creating a occt repository here: https://github.com/FreeCAD.
Having it on launchpad is also ok, but gathering the sources at one place would make more sense.

[NormandC]

Having it on launchpad is also ok, but gathering the sources at one place would make more sense.

Unfortunately, to build a Ubuntu package on the PPA using recipes, the bzr or git repo has to be hosted on Launchpad.

Edit: a Launchpad (LP) bzr repo can be mirrored from an external git branch, but there is a bug on signed Git commits that forced sgrogan and I to switch to git on LP for FreeCAD, and AFAIK the LP git mirror needs to be synchronized manually. So, the OCCT LP git could be mirrored from a FreeCAD OCCT github repo.