ana(conda) windows packaging

[kkremitzki]

and got this spaghetti:

I feel your pain. I wish vtk would split the package upstream, on Ubuntu, at least, python-vtk6 is a separate package.
Maybe this gives some clues: https://github.com/conda-forge/vtk-feedstock/issues/46?

Based on some posts I've seen recently in the debian-devel mailing list, I think Kitware, the company behind VTK, ParaView, and CMake, has dedicated a person to improving the state of their software on Debian, so I think things will be improving in the future in Debian & Ubuntu.

[looo]

for now please pin tbb to version "2018.0.5". The problem here is really that the dependencies are not yet ready for the latest tbb. The package was split in tbb, tbb-devel and tbb4py. So every recipe using tbb in the build-section now also must specify the tbb-devel as additional dependency. I did this change for the occt-recipe, but somehow this didn't solve the issue for freecad.
So I think more packages have to be updated. As this is not going to happen until tomorrow please use the workaround to simple stay with tbb=2018.0.5. To update with pinned tbb please use something like this:

Code: Select all

conda update --all tbb=2018.0.5 occt=7.3.*

[looo]

@UR_ I had another look at your problem-description. To me this problem seems to be directly related to the switch to a spereated tbb / tbb-devel. So installing tbb-devel should fix this issue. But I am not sure if the build will work (binary incompatibility).

[triplus]

Yes if you add the channel freecad/label/dev the latest build should be used where the version info works.

OK therefore in short all this channels are needed for now: conda-forge, freecad and freecad/label/dev? Will things stay like this in the (foreseeable) future or the freecad/label/dev channel is only a temporary thing?

After some further testing it seems pip is fully working with the appimage.

Good to hear that.

Btw.: I do not have the problem on windows because there I do not build occt with tbb enabled. Does anyone know what the advantages of tbb are?

In the past TBB was needed to enable parallelization routines (meshing related, boolean operations (with multiple arguments) ...) but now (by using newer OCCT) using TBB is not mandatory anymore to achieve that (multi-threading):

[*] Intel TBB, optional for several modules.
It is NOT mandatory for activating parallel algorithms in OCCT,
but can be more efficient for this task then built-in OCCT parallelization routines (OSD_Parallel).

But subprocess commands are a bit more difficult to handle... Maybe you can add my pip module to the appimage to get it tested a bit more:
https://github.com/looooo/freecad_conda ... age.sh#L14

Is this ready yet to be exposed to (regular) end users? If yes i can add it.

P.S. Will do some additional tests tomorrow and read the additional answers. And hopefully enable all what was discussed on Travis.

[UR_]

So installing tbb-devel should fix this issue. But I am not sure if the build will work

Just,
- unpinned tbb
- added tbb-devel
- updated env
- build FreeCAD
- tested some vtk stuff with FEM-WB

and everything looks fine

[looo]

OK therefore in short all this channels are needed for now: conda-forge, freecad and freecad/label/dev? Will things stay like this in the (foreseeable) future or the freecad/label/dev channel is only a temporary thing?

I will upload releases to conda-forge and weekly builds to freecad/label/dev. It would be also possible to upload to freecad directly. I used the dev-label only because some other projects do it the same way. (The freecad-channel is not yet used, but I will use it if the dependency-tree at some point breaks again, to have tempary fixed/updated packages available. This is necessary because conda-forge is a rolling-release -> unstable (see the current tbb-prolem))

In the past TBB was needed to enable parallelization routines (meshing related, boolean operations (with multiple arguments) ...) but now (by using newer OCCT) using TBB is not mandatory anymore to achieve that (multi-threading):

gkv311 wrote: ↑
Mon Oct 24, 2016 9:48 am
[*] Intel TBB, optional for several modules.
It is NOT mandatory for activating parallel algorithms in OCCT,
but can be more efficient for this task then built-in OCCT parallelization routines (OSD_Parallel).

I will try to get all packages build with tbb enabled. If this doesn't work out of the box, I will disable it for all plattforms. I guess the most difficult package will be vtk. I expect to see multiple different problems here: https://github.com/conda-forge/vtk-feedstock/pull/67

The question is also if all occt/vtk-dependent packages have to be updated. (netgen, smesh)

As a workaround I will use an old tbb for the next freecad-builds (Done by making freecad dependent on tbb directly). Locally this worked already on linux. I hope this works now as I am already a bit late for the sunday-build https://github.com/looooo/freecad-feeds ... 9230149247

Just,
- unpinned tbb
- added tbb-devel
- updated env
- build FreeCAD
- tested some vtk stuff with FEM-WB

and everything looks fine

so no binary incompatibilities? Are you compiling occt also by yourself? And what about vtk.

Is this ready yet to be exposed to (regular) end users? If yes i can add it.

Hmm, difficult question. I haven't found a solution to install packages to a custom directory. Currently a directory shared with the system python is used to install all packages installed via the pipintegration tool. But I don't think I can come up with a solution very soon (see the packaging troubles, which need to be solved first)

[UR_]

Are you compiling occt also by yourself?

I'm just using these precompiled conda packages

Code: Select all

(freecad_py3) C:\Users\aio>conda list
# packages in environment at C:\Users\aio\Miniconda3\envs\freecad_py3:
#
# Name                    Version                   Build  Channel
blas                      1.0                         mkl
boost-cpp                 1.67.0                   vc14_0  [vc14]  conda-forge
ca-certificates           2018.8.24            ha4d7672_0    conda-forge
certifi                   2018.8.24             py36_1001    conda-forge
clangdev                  5.0.0                   flang_3  [flang vc14]  conda-forge
coin3d                    4.0.0                h037026a_4    conda-forge
curl                      7.60.0                   vc14_0  [vc14]  conda-forge
cycler                    0.10.0                     py_1    conda-forge
eigen                     3.3.5             he980bc4_1001    conda-forge
expat                     2.2.5                    vc14_0  [vc14]  conda-forge
freecad                   0.18b0           py36ha567227_4    conda-forge
freetype                  2.8.1                    vc14_0  [vc14]  conda-forge
future                    0.16.0                   py36_2    conda-forge
gitdb2                    2.0.4                      py_0    conda-forge
gitpython                 2.1.11                     py_0    conda-forge
graphviz                  2.38.0               h6538335_9    conda-forge
hdf4                      4.2.13                   vc14_0  [vc14]  conda-forge
hdf5                      1.10.1                   vc14_2  [vc14]  conda-forge
icc_rt                    2017.0.4             h97af966_0
icu                       58.2                     vc14_0  [vc14]  conda-forge
intel-openmp              2019.0                      118
jpeg                      9b                       vc14_2  [vc14]  conda-forge
jsoncpp                   1.8.1                    vc14_0  [vc14]  conda-forge
kiwisolver                1.0.1            py36he980bc4_2    conda-forge
krb5                      1.14.6                   vc14_0  [vc14]  conda-forge
libflang                  5.0.0             vc14_20180207  [vc14]  conda-forge
libiconv                  1.14                     vc14_4  [vc14]  conda-forge
libnetcdf                 4.4.1.1                 vc14_10  [vc14]  conda-forge
libpng                    1.6.34                   vc14_0  [vc14]  conda-forge
libssh2                   1.8.0                    vc14_2  [vc14]  conda-forge
libtiff                   4.0.9                    vc14_0  [vc14]  conda-forge
libxml2                   2.9.5                    vc14_1  [vc14]  conda-forge
libxslt                   1.1.32                   vc14_0  [vc14]  conda-forge
llvm-meta                 6.0.1                         0    conda-forge
llvmdev                   4.0.0                 default_0  [vc14]  conda-forge
lz4-c                     1.8.2                    vc14_0  [vc14]  conda-forge
matplotlib                2.2.2                    py36_1    conda-forge
mkl                       2019.0                      118
mkl_fft                   1.0.6                    py36_0    conda-forge
mkl_random                1.0.1                    py36_0    conda-forge
netgen                    6.2.1804         py36he980bc4_3    conda-forge
numpy                     1.15.1           py36ha559c80_0
numpy-base                1.15.1           py36h8128ebf_0
occt                      7.3.0                h39117bb_0    conda-forge
openblas                  0.2.20                   vc14_8  [vc14]  conda-forge
openmp                    5.0.0                    vc14_0  [vc14]  conda-forge
openssl                   1.0.2p            hfa6e2cd_1000    conda-forge
pip                       18.0                     py36_1    conda-forge
pivy                      0.6.4b2             py36_vc14_0  [vc14]  conda-forge
pthreads-win32            2.9.1                    vc14_1  [vc14]  conda-forge
pybind11                  2.2.4           py36he980bc4_1000    conda-forge
pyparsing                 2.2.1                      py_0    conda-forge
pyqt                      5.6.0            py36h764d66f_7    conda-forge
pyside2                   5.6.0a1             py36_vc14_2  [vc14]  conda-forge
python                    3.6.6                he025d50_0    conda-forge
python-dateutil           2.7.3                      py_0    conda-forge
pytz                      2018.5                     py_0    conda-forge
qt                        5.6.2                    vc14_1  [vc14]  conda-forge
setuptools                40.4.0                py36_1000    conda-forge
sip                       4.18.1           py36h6538335_0    conda-forge
six                       1.11.0                   py36_1    conda-forge
smesh                     8.3.0b0          py36h7731f85_9    conda-forge
smmap2                    2.0.4                      py_0    conda-forge
soqt                      1.6.0a               he980bc4_0    conda-forge
sqlite                    3.22.0                   vc14_0  [vc14]  conda-forge
swig                      3.0.12                        2    conda-forge
tbb                       2019.0               he980bc4_0    conda-forge
tbb-devel                 2019.0               he980bc4_0    conda-forge
tornado                   5.1.1            py36hfa6e2cd_0    conda-forge
vc                        14                            0    conda-forge
vs2015_runtime            14.0.25420                    0    conda-forge
vtk                       8.1.1           py36_vc14h99a48a5_200  [vc14]  conda-forge
wheel                     0.31.1                py36_1001    conda-forge
wincertstore              0.2                   py36_1002    conda-forge
xerces-c                  3.2.0                    vc14_0  [vc14]  conda-forge
zlib                      1.2.11                   vc14_0  [vc14]  conda-forge

FreeCAD's self tests are working as expected (not 100% error free, as usual)

[looo]

I'm just using these precompiled conda packages

so it seems the new tbb is compatible with the old one (At least for freecad on windows). But anyway I will try to get vtk and occt updated before I release the pinning of the freecad-feedstock again.

FreeCAD's self tests are working as expected (not 100% error free, as usual)

Running tests with the GUI I see some failing tests which are not reported. Maybe someone can report them so we can work on these issues.

@triplus:
the new build is available. Creating an Appimage with my script works fine, but still the size is about 280Mb. And finally:

Bildschirmfoto von 2018-09-25 09-25-18.png (188.08 KiB) Viewed 1452 times

[sgrogan]

Based on some posts I've seen recently in the debian-devel mailing list, I think Kitware, the company behind VTK, ParaView, and CMake, has dedicated a person to improving the state of their software on Debian, so I think things will be improving in the future in Debian & Ubuntu.

This would be welcome, also for Win on conda. For the PY2 builds I package only a small sub-set of VTK that FreeCAD needs, but for conda such a stripped down package is limited. Splitting the package would go a long way. Some CMake help would help on all OS's.

Do you have a link to the mailing list? I might sign up and express my opinion.

[sgrogan]

Running tests with the GUI I see some failing tests which are not reported. Maybe someone can report them so we can work on these issues.

I only get one

Code: Select all

Traceback (most recent call last):
  File "C:\Users\Chris\GitHub\FreeCAD_0.18.14802_Conda_Py3QT5-WinVS2015_x64\bin\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\Users\Chris\GitHub\FreeCAD_0.18.14802_Conda_Py3QT5-WinVS2015_x64\bin\lib\unittest\case.py", line 605, in run
    testMethod()
  File "C:\Users\Chris\GitHub\FreeCAD_0.18.14802_Conda_Py3QT5-WinVS2015_x64\Mod\Test\TestPythonSyntax.py", line 44, in testAll
    test_python_syntax(mod_dir, self.whitelist)
  File "C:\Users\Chris\GitHub\FreeCAD_0.18.14802_Conda_Py3QT5-WinVS2015_x64\Mod\Test\TestPythonSyntax.py", line 19, in test_python_syntax
    ast.parse(py_file.read())
  File "C:\Users\Chris\GitHub\FreeCAD_0.18.14802_Conda_Py3QT5-WinVS2015_x64\bin\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 30763: character maps to <undefined>

I broke my self compiled PY3 build on Ubuntu, so if someone can verify with conda or otherwise, I'll make a ticket.

OS: Windows 7
Word size of OS: 64-bit
Word size of FreeCAD: 64-bit
Version: 0.18.14802 (Git)
Build type: Release
Branch: master
Hash: dfd3d86b227099bab3649624507fcb205cdd93a2
Python version: 3.6.6
Qt version: 5.6.2
Coin version: 4.0.0a
OCC version: 7.3.0
Locale: English/UnitedStates (en_US)

This is from the portable builds, and I DO NOT regularly update the conda env I package them from. I will test with the new env. conda is very good for me in this respect.