Packaging solution: (ana)conda

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
looo
Posts: 2948
Joined: Mon Nov 11, 2013 5:29 pm

Packaging solution: (ana)conda

Postby looo » Sat Apr 09, 2016 5:03 pm

So we have this long time ongoing discussion about plungin-managers, package-manager,...

Currently I wanted to install the python-occ package and saw that they have switched to the conda installer. This package manager is not like pip only for python packages. With conda it's possible to install packages written in any language. So I gave it a try and installed anaconda. First time I installed as admin and somehow anaconda disabled the installed python. After some time I tried it again, but this time without sudo, and this worked quite good.

Conda is part of the anaconda-system, which is a full python enviroment shipping many frequently used packages (also packages we use in freecad: matplotlib, numpy, scipy, ...).

As I understand, Conda was created because the std python install tools are not suited for non-python installations. Conda installs compiled packages which are hosted in repositories. At the anconda cloud it's possible for everyone to host his packages.

Conda gives you also the possebility to create conda packages from pypi. This is the simplest way how packages can be created :

Code: Select all

conda skeleton pypi <packagename>          # creates a so called recipe which consists of a meta.yaml and a build.sh 

# building the package:
conda build <packagename>
the code to upload the package will be printed at the end of the build-process
so this is really simple and it shows how a conda recipe should look like. More complex packages like (boost, coin, ...) can be created with costum recipe.

All in all I have the feeling ana/conda could be the right thing for freecad-packages and for maintaining freecad it self. It supports all platforms. Many packages are already available. It is simple to create new packages. FreeCAD addons could be added very easily (even with c++ libraries)...

At the moment I don't see any show-stoppers, but I would like to hear any opinions on this. Especially I would like to know how conda compares to apt and other package manager. Is it possible to have one package-manager for all platforms??? Or will this bring in dependencies issues?


There is much documentation for conda. For building recipes this site was helpful for me.
conda build tutorials
please help with my conda-packaging efforts: https://liberapay.com/looooo/
looo
Posts: 2948
Joined: Mon Nov 11, 2013 5:29 pm

Re: Packaging solution: (ana)conda

Postby looo » Mon Apr 11, 2016 10:06 am

Here is a first build of freecad with python3: https://anaconda.org/looooo/freecad. It works on my two ubuntu machines. On other linux distribution there are maybe errors, because I haven't added all needed libraries as dependencies. The package also doesn't contain every module... and still has some problems with qrc-files...
There occurred also problems with boost. The boost-builds found on anaconda weren't compatible with my freecad-build. (I think because of gcc / clang incompatibility) So I have also uploaded a boost build.

To install:
(only linux)
download anaconda for python3.5
install anaconda with "bash <anaconda_download_file>"
create env: "conda create -n <name> python=3.5"
activate env: "source activate <name>"
add necessary channels:
" conda config --add channels https://anaconda.org/looooo" # boost, coin, pivy
" conda config --add channels https://conda.anaconda.org/omnia" # for pytools
install freecad: conda install freecad
please help with my conda-packaging efforts: https://liberapay.com/looooo/
triplus
Posts: 8786
Joined: Mon Dec 12, 2011 4:45 pm

Re: Packaging solution: (ana)conda

Postby triplus » Mon Apr 11, 2016 12:41 pm

ATM installing Python based workbenches from GitHub and improving that experience makes the most sense to me. As for the future sure why not if it could be done and somebody is willing to do it.

That is to have possibility to go beyond supporting the installation of Python based FreeCAD modules with ease.
looo
Posts: 2948
Joined: Mon Nov 11, 2013 5:29 pm

Re: Packaging solution: (ana)conda

Postby looo » Mon Apr 11, 2016 2:20 pm

ATM installing Python based workbenches from GitHub and improving that experience makes the most sense to me. As for the future sure why not if it could be done and somebody is willing to do it.
After my first experiments with conda, I have the feeling that it isn't that suited as alternative to apt/... . For example it won't be possible to install packages into the FreeCAD-home directory. The only way is to have everything inside of the anaconda directory.
My impression is, that conda is a tool for developers: You can create many envs with different kind of freecad-versions. So if a developer want to get something tested, not every tester has to compile the code. Creating an env and installing the freecad-version is all one has to do.
please help with my conda-packaging efforts: https://liberapay.com/looooo/
looo
Posts: 2948
Joined: Mon Nov 11, 2013 5:29 pm

Re: Packaging solution: (ana)conda

Postby looo » Sun May 01, 2016 10:47 am

Here is a short instruction how to install a python3 version of freecad with conda: (by now only a linux version is available, maybe someone want to make a conda paykage of freecad for windows and mac)

INSTALL MINICONDA:
  1. first get miniconda: http://conda.pydata.org/miniconda.html choose python3 (it's not necessary, you could also choose python2, but then it will download python3 stuff in the "create-env-step")
  2. install miniconda: bash <miniconda-file>.sh (not as root!!!)
  3. at the end of the install it will ask you if you want to add the anaconda-dir to the $PATH, say yes.
  4. if you do not want anaconda to be the default python open the ~/.bashrc and edit the new line:
    -from this: export PATH="path_to_anaconda/bin:$PATH"
    -to this: alias initConda='export PATH="path_to_anaconda/bin:$PATH" '
    : this way anaconda isn't perpended by default. As soon as you call "initConda" python will be the anaconda version.
CREATE A NEW ENV
  1. type in terminal: initConda (now the "conda" command should be available)
  2. create an env: conda create -n <env-name> python=3.5 # with <env-name> is the name of the env, eg. fc_test
    (it will install some packages, but I think most of them are available within miniconda, so it will only link this packages to the env)
  3. at the end of this process a short statement is printed how to activate the new env.: source activate <env-name>
  4. now you have a minimal setup of python (only packages in this env are available)
INSTALL FREECAD
  1. first we have to add some channels to get all the necesarry packages:
    - you can add them one by one with: conda config --add channels <name>
    - or you can open the ~/.condarc and add them directly to this file
    at the end this file should look like this:

    Code: Select all

    channels:
      - spyder-ide
      - dlr-sc
      - looooo
      - defaults
    
    the channels hosting this libraries:
    looooo: freecad, coin, pivy, boost,...
    dlr-sc: oce
    spyder-ide: libuci
  2. install freecad: conda install freecad
  3. start freecad: FreeCAD
    If you had freecad previously on your machine this will maybe work. otherwise there will be errors about missing libraries. Please report this and search if the missing packages are available on anaconda.org.
Last edited by looo on Sun May 01, 2016 2:54 pm, edited 3 times in total.
please help with my conda-packaging efforts: https://liberapay.com/looooo/
triplus
Posts: 8786
Joined: Mon Dec 12, 2011 4:45 pm

Re: Packaging solution: (ana)conda

Postby triplus » Sun May 01, 2016 1:35 pm

I guess i will try this out when i have a bit more time.

P.S. But it looks complex? ;)
looo
Posts: 2948
Joined: Mon Nov 11, 2013 5:29 pm

Re: Packaging solution: (ana)conda

Postby looo » Sun May 01, 2016 2:50 pm

I think it isn't too difficult to get used to it. Creating a env is not really necessary, but doing this will let you try out different versions of packages for different version of python...

With conda it is also possible to build the packages with eg. travis and directly upload to the anaconda cloud. So daily packages should be possible. The only problem I see is the limitation to 3GB.
please help with my conda-packaging efforts: https://liberapay.com/looooo/
triplus
Posts: 8786
Joined: Mon Dec 12, 2011 4:45 pm

Re: Packaging solution: (ana)conda

Postby triplus » Sun May 01, 2016 8:30 pm

I wanted to try this out but the first obstacle was getting Miniconda. That is 2 hours of download time and it didn't finish (30MB). I cancelled it after.

I assume your packages are of much bigger size and the download speed is not fast? I was afraid of this "bandwidth issue" before trying it out and now after actually trying it out i feel if we wouldn't solve the bandwidth issue this could probably never work.
looo
Posts: 2948
Joined: Mon Nov 11, 2013 5:29 pm

Re: Packaging solution: (ana)conda

Postby looo » Sun May 01, 2016 9:41 pm

No idea. I never had any problems with Download speeds.
please help with my conda-packaging efforts: https://liberapay.com/looooo/
triplus
Posts: 8786
Joined: Mon Dec 12, 2011 4:45 pm

Re: Packaging solution: (ana)conda

Postby triplus » Sun May 01, 2016 10:08 pm

looo wrote:No idea. I never had any problems with Download speeds.
OK i will try again tomorrow. As currently in quick test the speed improved and it's around 100Kb/s. :)