can we use conda for building/debuging localy modified freecad files?

[danreb]

Looo you probably have an idea on what's explained below?

in the work dir of my build env I can't see the latest git commit (i.e. those from today).

Indeed git log on master branch in the /work dir of my conda debug/build env shows the latest commit dating from the 7th of october, while on Freecad/Freecad github the latest commits are from today the 8th of october?


What I understand is that by running the

Code: Select all

conda debug . -m .ci_support/osx_python3.7.yaml 

a git_cache has been created in my debug/build conda env but in the state the github repository was at that time. This cache is then used as my remote/origine repository for my conda debug/build env (cf. the work dir). Hence I am not aware of more recent commits on github.

What would be the recommended way for updating the my debug/build conda env git_cache? i.e. how tho resync my cache with the actual state of the github repo?

• Can I do it with a git command
• should I use a conda command like "conda update ..."
• do we have to rebuild the all debug env and then rebuild freecad (which solution would be quite time consuming)

What would be the recommended way of pulling these recent commits in my conda debug env since git status gives to folowing output i.e. it looks that I am

[looo]

You are right. You can simple switch the remote repo:

Code: Select all

git remote set-url origin https://github.com/FreeCAD/FreeCAD

Also note that some extra files are added in the directory, so be careful with "git add" to not include these files in a PR...

It seems like there are always some manual steps necessary to setup the development environment...

What would be the recommended way of pulling these recent commits in my conda debug env since git status gives to folowing output i.e. it looks that I am

I guess there is no recommendation. This is quite experimental.

[danreb]

OK maybe it's best to consider our conda debug/build git repo (i.e. the one that is in the git-cached folder) as a kind of "fork" of the original and try to keep it in sync with the github/Freecad original repo.

This could apparently be done by using for example the git upstream repo functionality (cf. https://medium.com/@_oleksii_/how-to-sy ... 3b78892901). Well I don't really know but I'll have to test this.....

The idee is that we can keep the master branch of our "FORK" repo in sync with the upstream master branch (fetch and pull/commit from github on the fork). It means never change anything on the master branch except from the upstream sync, And make our development on a dedicated new branch in our "Fork" repo.

maybe it's even easier to propose a pull request at the end of a developpment cycle?

anyway just thinking ...

[looo]

I guess the inconvenience we are seeing here are due to feedstocks are not really designed for the creation of a development environment. Also conda debug is not meant to address this purpose. I guess this tool should provide maintainers the ability to run a recipe several times without going through the whole process of gathering source and dependencies.

So some inconvenience are expected, but in my eyes "conda debug" is pretty close to the things we want to achieve here.

OK maybe it's best to consider our conda debug/build git repo (i.e. the one that is in the git-cached folder) as a kind of "fork" of the original and try to keep it in sync with the github/Freecad original repo.

This could apparently be done by using for example the git upstream repo functionality (cf. https://medium.com/@_oleksii_/how-to-sy ... 3b78892901). Well I don't really know but I'll have to test this.....

The idee is that we can keep the master branch of our "FORK" repo in sync with the upstream master branch (fetch and pull/commit from github on the fork). It means never change anything on the master branch except from the upstream sync, And make our development on a dedicated new branch in our "Fork" repo.

maybe it's even easier to propose a pull request at the end of a developpment cycle?

There are several options how to solve this. One might be to specify source in another directory which can be done by commenting out the source section in the meta.yaml and set the directory in the build script. We could add these changes to a fork/branch of the freecad-feedstock, but I learned in the past, having different variants of the repo will increase maintenance work and pretty soon we see things getting out of date and so on.

So in my eyes, for now, the simplest way to solve this is switching the remote to point to whatever branch you want to follow.

[looo]

Please search for conda-build or conda in general in this forum. This topic is about OCCT7.4.

I asked you because I could not find a step-by-step instruction (I searched of course already the Forum and the Wiki) and I see that you are familiar with Conda. Thus it would be nice if you could give me a pointer to an instruction?

moving this discussion from occt7.4 topic to this topic.

Today I tested conda-debug on windows and it seems to work the same way as it does on linux. So if you have visual studio installed (1) you can follow pretty much the same instructions as shown for linux in this video [1]. This is:

* install miniconda (if you haven't done already, this is not done in the video)
* conda install conda-build (if you haven't done yet)
* git clone https://github.com/looooo/freecad-feedstock (# this includes the recipe used for the weekly builds)
* cd freecad-feedstock
* conda debug . -m .ci_support\win....yaml

This will gather all the things necesarry to build freecad (deps, sources via git) and will need quite some time (depending on your internet-connection). Once done there will be a output on how to proceed (changing into the work directory and activating the build-env on top of the host-env). As @danreb has figured out there is one additional step necessary to pull sources from the freecad-repo once the currently used sources are out of date. So the following:

* git remote set-url origin https://github.com/FreeCAD/FreeCAD
* git fetch --all
* git reset origin/master --hard

(If you are used to git, cmake and know how it works you can also do any other workaround, for example you can modify the conda_build.bat file to look for the yource in another directory)

Once this is done you simple call the build script:

* conda_build.bat

and wait until the build has finished. Once done you can start freecad by this command:

* FreeCAD

Now you can create a script which does all the stuff at once, eg. this is how it looks for me:

Code: Select all

cd C:\Users\lo\Miniconda3\conda-bld\debug_1571046021538\work
call C:\Users\lo\Miniconda3\conda-bld\debug_1571046021538\work\build_env_setup.bat
git pull
call conda_build.bat
FreeCAD

Now you simple have to doubleklick this script and latest freecad gets pulled and compiled. I hope this helps.

[1] https://vimeo.com/364817445

(1) I guess 2015 / 2017 / 2019 should work. Microsoft changes it's strategy pretty fast regarding the compilers and the download page also changes quite often, so I can't make a 1., 2., 3. suggestion on how to install visual studio compilers. In the long run I hope conda packages for windows will switch toolchain to a more openly distributed set of compilers (clang, flang, ...)

edit: not yet sure if the script approach really works. Somehow I see a crash during linking but I am not sure what the reason for this is.

edit1: currently we use a patch for pyside which is necessary for windows. This patch must be applied otherwise the build is abborted. This was the reson for the aborted build which I mentioned in the first edit. Therefor I applied the script and switch to another bbranch and use rebase instead off pull to get the latest source.

[UR_]

I guess (VS) 2015 / 2017 / 2019 should work.

Just modifyed CMakeCache.txt from:

Code: Select all

//Name of generator.
CMAKE_GENERATOR:INTERNAL=Visual Studio 14 2015 Win64
//Generator instance identifier.
CMAKE_GENERATOR_INSTANCE:INTERNAL=C:/Program Files (x86)/Microsoft Visual Studio/2015/Community

to

Code: Select all

//Name of generator.
CMAKE_GENERATOR:INTERNAL=Visual Studio 15 2017 Win64
//Generator instance identifier.
CMAKE_GENERATOR_INSTANCE:INTERNAL=C:/Program Files (x86)/Microsoft Visual Studio/2017/Community

worked flawlessly.

[uwestoehr]

So if you have visual studio installed (1) you can follow pretty much the same instructions as shown for linux in this video [1]. This is:...

Many thanks looo!
I will test this out and create a Wiki page when I succeeded to document it for further usage.

[uwestoehr]

* install miniconda (if you haven't done already, this is not done in the video)
* conda install conda-build (if you haven't done yet)
* git clone https://github.com/looooo/freecad-feedstock (# this includes the recipe used for the weekly builds)
* cd freecad-feedstock
* conda debug . -m .ci_support\win....yaml

I tried it and are stuck. When calling

Code: Select all

>conda debug . -m .ci_support\win_cxx_compilervs2015python3.7.yaml

I get:

Code: Select all

(base) D:\FreeCAD-feedstock>conda debug . -m .ci_support\win_cxx_compilervs2015p
ython3.7.yaml
No numpy version specified in conda_build_config.yaml.  Falling back to default
numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.ya
ml.  Falling back to default numpy value of 1.11
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from D:\FreeCAD-feedstock\recipe\conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from D:\FreeCAD-feedstock\recipe\co
nda_build_config.yaml
Adding in variants from .ci_support\win_cxx_compilervs2015python3.7.yaml
INFO:conda_build.variants:Adding in variants from .ci_support\win_cxx_compilervs
2015python3.7.yaml
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
Traceback (most recent call last):
  File "C:\Miniconda\lib\site-packages\conda_build\environ.py", line 757, in get
_install_actions
    actions = install_actions(prefix, index, specs, force=True)
  File "C:\Miniconda\lib\site-packages\conda\common\io.py", line 88, in decorate
d
    return f(*args, **kwds)
  File "C:\Miniconda\lib\site-packages\conda\plan.py", line 474, in install_acti
ons
    txn = solver.solve_for_transaction(prune=prune, ignore_pinned=not pinned)
  File "C:\Miniconda\lib\site-packages\conda\core\solve.py", line 117, in solve_
for_transaction
    should_retry_solve)
  File "C:\Miniconda\lib\site-packages\conda\core\solve.py", line 158, in solve_
for_diff
    force_remove, should_retry_solve)
  File "C:\Miniconda\lib\site-packages\conda\core\solve.py", line 275, in solve_
final_state
    ssc = self._add_specs(ssc)
  File "C:\Miniconda\lib\site-packages\conda\core\solve.py", line 555, in _add_s
pecs
    explicit_pool = ssc.r._get_package_pool(self.specs_to_add)
  File "C:\Miniconda\lib\site-packages\conda\resolve.py", line 523, in _get_pack
age_pool
    pool = self.get_reduced_index(specs)
  File "C:\Miniconda\lib\site-packages\conda\common\io.py", line 88, in decorate
d
    return f(*args, **kwds)
  File "C:\Miniconda\lib\site-packages\conda\resolve.py", line 544, in get_reduc
ed_index
    explicit_specs, features = self.verify_specs(explicit_specs)
  File "C:\Miniconda\lib\site-packages\conda\resolve.py", line 280, in verify_sp
ecs
    raise ResolvePackageNotFound(bad_deps)
conda.exceptions.ResolvePackageNotFound:
  - qt=5.12
  - doxygen
  - netgen
  - coin3d
  - hdf5=1.10.5
  - boost-cpp=1.70.0
  - blas=[build=openblas]
  - occt=7.3
  - smesh
  - pivy
  - pyside2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Miniconda\Scripts\conda-debug-script.py", line 10, in <module>
    sys.exit(main())
  File "C:\Miniconda\lib\site-packages\conda_build\cli\main_debug.py", line 94,
in main
    return execute(sys.argv[1:])
  File "C:\Miniconda\lib\site-packages\conda_build\cli\main_debug.py", line 70,
in execute
    activation_string = api.debug(thing_to_debug, verbose=(not _args.activate_st
ring_only), **_args.__dict__)
  File "C:\Miniconda\lib\site-packages\conda_build\api.py", line 451, in debug
    metadata_tuples = render(recipe_or_package_path_or_metadata_tuples, config=c
onfig, **kwargs)
  File "C:\Miniconda\lib\site-packages\conda_build\api.py", line 45, in render
    permit_unsatisfiable_variants=permit_unsatisfiable_variants)
  File "C:\Miniconda\lib\site-packages\conda_build\render.py", line 819, in rend
er_recipe
    allow_no_other_outputs=True, bypass_env_check=bypass_env_check)
  File "C:\Miniconda\lib\site-packages\conda_build\render.py", line 725, in dist
ribute_variants
    bypass_env_check=bypass_env_check)
  File "C:\Miniconda\lib\site-packages\conda_build\metadata.py", line 1018, in p
arse_until_resolved
    bypass_env_check=bypass_env_check)
  File "C:\Miniconda\lib\site-packages\conda_build\metadata.py", line 940, in pa
rse_again
    bypass_env_check=bypass_env_check),
  File "C:\Miniconda\lib\site-packages\conda_build\metadata.py", line 1528, in _
get_contents
    rendered = template.render(environment=env)
  File "C:\Miniconda\lib\site-packages\jinja2\asyncsupport.py", line 76, in rend
er
    return original_render(self, *args, **kwargs)
  File "C:\Miniconda\lib\site-packages\jinja2\environment.py", line 1008, in ren
der
    return self.environment.handle_exception(exc_info, True)
  File "C:\Miniconda\lib\site-packages\jinja2\environment.py", line 780, in hand
le_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Miniconda\lib\site-packages\jinja2\_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "D:\FreeCAD-feedstock\recipe\meta.yaml", line 43, in top-level template c
ode
    - eigen
  File "C:\Miniconda\lib\site-packages\conda_build\jinja_context.py", line 230,
in pin_compatible
    pins, _, _ = get_env_dependencies(m, 'host', m.config.variant)
  File "C:\Miniconda\lib\site-packages\conda_build\render.py", line 154, in get_
env_dependencies
    channel_urls=tuple(m.config.channel_urls))
  File "C:\Miniconda\lib\site-packages\conda_build\environ.py", line 759, in get
_install_actions
    raise DependencyNeedsBuildingError(exc, subdir=subdir)
conda_build.exceptions.DependencyNeedsBuildingError: Unsatisfiable dependencies
for platform win-64: {'occt=7.3', 'boost-cpp=1.70.0', 'smesh', 'blas=[build=open
blas]', 'pyside2', 'hdf5=1.10.5', 'netgen', 'qt=5.12', 'coin3d', 'pivy', 'doxyge
n'}

I could install miniconda successfully in advance. I have MSVC 2017 installed and can successfully use it to build FC using the LibPack.

[looo]

* install miniconda (if you haven't done already, this is not done in the video)
* conda install conda-build (if you haven't done yet)
* git clone https://github.com/looooo/freecad-feedstock (# this includes the recipe used for the weekly builds)
* cd freecad-feedstock
* conda debug . -m .ci_support\win....yaml

I tried it and are stuck. When calling

Code: Select all

>conda debug . -m .ci_support\win_cxx_compilervs2015python3.7.yaml

I get:

Code: Select all

(base) D:\FreeCAD-feedstock>conda debug . -m .ci_support\win_cxx_compilervs2015p
ython3.7.yaml
No numpy version specified in conda_build_config.yaml.  Falling back to default
numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.ya
ml.  Falling back to default numpy value of 1.11
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from D:\FreeCAD-feedstock\recipe\conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from D:\FreeCAD-feedstock\recipe\co
nda_build_config.yaml
Adding in variants from .ci_support\win_cxx_compilervs2015python3.7.yaml
INFO:conda_build.variants:Adding in variants from .ci_support\win_cxx_compilervs
2015python3.7.yaml
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
Traceback (most recent call last):
  File "C:\Miniconda\lib\site-packages\conda_build\environ.py", line 757, in get
_install_actions
    actions = install_actions(prefix, index, specs, force=True)
  File "C:\Miniconda\lib\site-packages\conda\common\io.py", line 88, in decorate
d
    return f(*args, **kwds)
  File "C:\Miniconda\lib\site-packages\conda\plan.py", line 474, in install_acti
ons
    txn = solver.solve_for_transaction(prune=prune, ignore_pinned=not pinned)
  File "C:\Miniconda\lib\site-packages\conda\core\solve.py", line 117, in solve_
for_transaction
    should_retry_solve)
  File "C:\Miniconda\lib\site-packages\conda\core\solve.py", line 158, in solve_
for_diff
    force_remove, should_retry_solve)
  File "C:\Miniconda\lib\site-packages\conda\core\solve.py", line 275, in solve_
final_state
    ssc = self._add_specs(ssc)
  File "C:\Miniconda\lib\site-packages\conda\core\solve.py", line 555, in _add_s
pecs
    explicit_pool = ssc.r._get_package_pool(self.specs_to_add)
  File "C:\Miniconda\lib\site-packages\conda\resolve.py", line 523, in _get_pack
age_pool
    pool = self.get_reduced_index(specs)
  File "C:\Miniconda\lib\site-packages\conda\common\io.py", line 88, in decorate
d
    return f(*args, **kwds)
  File "C:\Miniconda\lib\site-packages\conda\resolve.py", line 544, in get_reduc
ed_index
    explicit_specs, features = self.verify_specs(explicit_specs)
  File "C:\Miniconda\lib\site-packages\conda\resolve.py", line 280, in verify_sp
ecs
    raise ResolvePackageNotFound(bad_deps)
conda.exceptions.ResolvePackageNotFound:
  - qt=5.12
  - doxygen
  - netgen
  - coin3d
  - hdf5=1.10.5
  - boost-cpp=1.70.0
  - blas=[build=openblas]
  - occt=7.3
  - smesh
  - pivy
  - pyside2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Miniconda\Scripts\conda-debug-script.py", line 10, in <module>
    sys.exit(main())
  File "C:\Miniconda\lib\site-packages\conda_build\cli\main_debug.py", line 94,
in main
    return execute(sys.argv[1:])
  File "C:\Miniconda\lib\site-packages\conda_build\cli\main_debug.py", line 70,
in execute
    activation_string = api.debug(thing_to_debug, verbose=(not _args.activate_st
ring_only), **_args.__dict__)
  File "C:\Miniconda\lib\site-packages\conda_build\api.py", line 451, in debug
    metadata_tuples = render(recipe_or_package_path_or_metadata_tuples, config=c
onfig, **kwargs)
  File "C:\Miniconda\lib\site-packages\conda_build\api.py", line 45, in render
    permit_unsatisfiable_variants=permit_unsatisfiable_variants)
  File "C:\Miniconda\lib\site-packages\conda_build\render.py", line 819, in rend
er_recipe
    allow_no_other_outputs=True, bypass_env_check=bypass_env_check)
  File "C:\Miniconda\lib\site-packages\conda_build\render.py", line 725, in dist
ribute_variants
    bypass_env_check=bypass_env_check)
  File "C:\Miniconda\lib\site-packages\conda_build\metadata.py", line 1018, in p
arse_until_resolved
    bypass_env_check=bypass_env_check)
  File "C:\Miniconda\lib\site-packages\conda_build\metadata.py", line 940, in pa
rse_again
    bypass_env_check=bypass_env_check),
  File "C:\Miniconda\lib\site-packages\conda_build\metadata.py", line 1528, in _
get_contents
    rendered = template.render(environment=env)
  File "C:\Miniconda\lib\site-packages\jinja2\asyncsupport.py", line 76, in rend
er
    return original_render(self, *args, **kwargs)
  File "C:\Miniconda\lib\site-packages\jinja2\environment.py", line 1008, in ren
der
    return self.environment.handle_exception(exc_info, True)
  File "C:\Miniconda\lib\site-packages\jinja2\environment.py", line 780, in hand
le_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Miniconda\lib\site-packages\jinja2\_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "D:\FreeCAD-feedstock\recipe\meta.yaml", line 43, in top-level template c
ode
    - eigen
  File "C:\Miniconda\lib\site-packages\conda_build\jinja_context.py", line 230,
in pin_compatible
    pins, _, _ = get_env_dependencies(m, 'host', m.config.variant)
  File "C:\Miniconda\lib\site-packages\conda_build\render.py", line 154, in get_
env_dependencies
    channel_urls=tuple(m.config.channel_urls))
  File "C:\Miniconda\lib\site-packages\conda_build\environ.py", line 759, in get
_install_actions
    raise DependencyNeedsBuildingError(exc, subdir=subdir)
conda_build.exceptions.DependencyNeedsBuildingError: Unsatisfiable dependencies
for platform win-64: {'occt=7.3', 'boost-cpp=1.70.0', 'smesh', 'blas=[build=open
blas]', 'pyside2', 'hdf5=1.10.5', 'netgen', 'qt=5.12', 'coin3d', 'pivy', 'doxyge
n'}

I could install miniconda successfully in advance. I have MSVC 2017 installed and can successfully use it to build FC using the LibPack.

You need to add the conda-forge channel:

Code: Select all

conda config --add channel conda-forge

[uwestoehr]

You need to add the conda-forge channel:

Code: Select all

conda config --add channel conda-forge

This fails here:

Code: Select all

>conda config --add channel conda-forge
CondaValueError: Key 'channel' is not a known sequence parameter.