I have not had much time to do actual modifications lately.
Clcache in AppVeyor
I have enabled clcache in AppVeyor, and I ran into the problem described by saso: Appveyor by default does not record the atime, this is the access time of the file, for performance reasons. Clcache relies on atime to determine the old cache items, which it "auto-cleans". I found a solution from the bitcoin guys to enable atime during compilation.
I have not found documentation of the auto-clean feature, but mere references to the fact there is one. But it seems to be working.
Code: Select all
clcache statistics:
current cache dir : Disk cache at C:\Users\appveyor\clcache
cache size : 1,040,653,217 bytes
maximum cache size : 1,073,741,824 bytes
cache entries : 1462
cache hits : 1136
cache misses
total : 1
evicted : 0
header changed : 1
source changed : 0
passed to real compiler
called w/ invalid argument : 0
called for preprocessing : 0
called for linking : 0
called for external debug : 0
called w/o source : 0
called w/ multiple sources : 0
called w/ PCH : 18
In AppVeyor with have 1 worker, 60 minutes limit and 1 GB cache (the result above gets compressed and we are somehow fitting the 1 GB even though we build for both 32 and 64 bits). The cache is local to the datacenter. This is ok for a full compilation taking about 57 minutes with MSVC2013. With the clcache improvements, most compilations (it depends on the changes) go down to around
21 minutes (2 minutes downloading libraries and clcache, 1 minute cmake, 17 minutes msbuild, 1 minute running tests). But this requires a full compilation first (or that timeout compilations save the cache, which might be possible).
I would like to move from MSVC2013 to MSVC2017 to have full c++11 support. However, MSVC2017 between 2-2.5 times slower than MSVC2013 (according to Travis tests).
I have found this thread:
https://github.com/appveyor/ci/issues/517
People are asking here for extended time in AppVeyor and getting successful at getting at least 90 minutes. I am not sure if "Shrek's cat's eyes" could get sufficient build time for a full build.
AppVeyor is fast, but having only one worker and making 32 and 64 bit builds with little changes still means waiting around
40 minutes, as the 32 bits build starts after the 64 bit build is finished. This is just a note to consider.
PCH
For reducing the build time, we should continue PCH. I am not sure how much it is possible to optimize with PCH, but I think we should aim to project-wide PCH.
Travis Windows
In Travis we have many workers (concurrent jobs) and 2h 30 minutes of build time. Travis is treating us like kings. Cache is located outside the datacenter.
We are running it with clcache too:
Code: Select all
clcache statistics:
current cache dir : Disk cache at C:\Users\travis\clcache
cache size : 718,772,447 bytes
maximum cache size : 1,073,741,824 bytes
cache entries : 1915
cache hits : 1120
cache misses
total : 26
evicted : 0
header changed : 1
source changed : 25
passed to real compiler
called w/ invalid argument : 0
called for preprocessing : 0
called for linking : 0
called for external debug : 0
called w/o source : 0
called w/ multiple sources : 0
called w/ PCH : 18
Running one typical MSVC2017 build (x64) with clcache takes about
1 hour. The first compilation took about 2 hours 20 minutes,
before PCH was introduced.
There is still a big problem to solve: Tests are not completely run and I am not sure if it would return "failure" if a test fails.
I will be working on this problem next.
Travis build times
Linux gcc builds of typical PRs take around
7 minutes to complete.
Linux clang builds sometimes take a little bit more, sometimes the same as gcc. Not sure what is the reason.
OSX builds take around
2 hours to complete. Not sure if something like caching can be done here.
The "Python 3.7" build is
systematically failing. I do not know if something may be done about this. I am not sure exactly what this build is:
https://travis-ci.org/FreeCAD/FreeCAD/builds/529083852
