GitHub CodeQL (formerly Semmle LGTM)

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
keithsloan52
Veteran
Posts: 2764
Joined: Mon Feb 27, 2012 5:31 pm

Re: LGTM (split from 'Code quality checker' thread)

Post by keithsloan52 »

Joel_graff wrote: Sat Oct 19, 2019 3:28 pm I could possibly try to write a Wiki enttry or something.

This also goes with my attempt at building a better workbench starterkit (which I've been neglecting)...
Falling at the first hurdle (Getting too old for this game).

Having read message from saso and this thread, decide it might be a good idea to get some of my efforts analysed. (bit wary of the wake up call it may give on the code quality).

Tried to follow the wiki, assume I am suppose to login to LGTM, which I did using my GitHub account. It lists various projects, so assume I have to create a project? Is that right? Tried on GitHub to start making a Project and a bit confused, don't want to muck up things on my Github, so bottled out and have come here for advice as I did not find the LGTM getting started very useful.
User avatar
saso
Veteran
Posts: 1924
Joined: Fri May 16, 2014 1:14 pm
Contact:

Re: LGTM (split from 'Code quality checker' thread)

Post by saso »

keithsloan52 wrote: Sun Sep 06, 2020 9:55 am Tried to follow the wiki, assume I am suppose to login to LGTM, which I did using my GitHub account. It lists various projects, so assume I have to create a project? Is that right? Tried on GitHub to start making a Project and a bit confused, don't want to muck up things on my Github, so bottled out and have come here for advice as I did not find the LGTM getting started very useful.
What repo do you want to have analysed? In most cases you don't have to do anything in your code or in the github. When you are logged in to LGTM, go to the "Project list" and just give it the link to the repo you want it to analyse and click "Follow". It will try to detect and run everything automatically, so just give it some time (from few minutes to few hours). If it will finish successfully you are done, it will continue to automatically analyse your repo once a day. If it will not be able to automatically detect and finish the first run, we can have a look at the logs and see if we can change some things to make it work. The problem with the cpp code from FreeCAD for example is that it is just to big to run it automatically on their servers, it would run for 10 or more hours, so I am building it manually...

Update: A note about the shut down of LGTM.com
https://github.blog/2022-08-15-the-next ... -scanning/

lgtm.png
lgtm.png (198.98 KiB) Viewed 2742 times
Last edited by saso on Wed Sep 07, 2022 11:46 am, edited 3 times in total.
User avatar
saso
Veteran
Posts: 1924
Joined: Fri May 16, 2014 1:14 pm
Contact:

Re: LGTM (Semmle CodeQL)

Post by saso »

"GitHub code scanning is a developer-first, GitHub-native approach to easily find security vulnerabilities before they reach production..."
https://github.blog/2020-09-30-code-sca ... available/
User avatar
saso
Veteran
Posts: 1924
Joined: Fri May 16, 2014 1:14 pm
Contact:

Re: LGTM (Semmle CodeQL)

Post by saso »

I am sharing my current build recipe for the LGTM (CodeQL) build on Ubuntu 16.04. Mostly it is basically just a normal build with a few commands to set up and run the scanner. I will continue to make the builds as I did in the past, so this is mostly for the purpose if it can maybe be of some use and help for some other projects... Check also https://help.semmle.com/codeql/codeql-c ... arted.html

sudo add-apt-repository ppa:freecad-maintainers/freecad-daily

sudo apt-get update -qq

sudo apt-get install -y libboost-date-time-dev libboost-dev libboost-filesystem-dev libboost-graph-dev libboost-iostreams-dev libboost-program-options-dev libboost-regex-dev libboost-serialization-dev libboost-signals-dev libboost-thread-dev libboost-python-dev libcoin-dev libdouble-conversion-dev liblz4-dev libglew-dev libopencv-dev libeigen3-dev libgts-bin libgts-dev libkdtree++-dev libmedc-dev libmetis-dev libocct-data-exchange-dev libocct-ocaf-dev libocct-visualization-dev libproj-dev libpyside-dev libqt4-dev libqt4-opengl-dev libqtwebkit-dev libshiboken-dev libspnav-dev libvtk6-dev libx11-dev libxerces-c-dev libzipios++-dev lsb-release netgen netgen-headers occt-draw pybind11-dev pyside-tools python3-pyside.qtcore python3-pyside.qtgui python3-pyside.qtsvg python3-pyside.qtuitools python3-pyside.qtxml python3-dev python3-ply swig doxygen

sudo apt-get install -y git curl cmake

sudo apt-get clean

git clone --depth=1600 --branch=master https://github.com/FreeCAD/FreeCAD.git FreeCAD

cd FreeCAD/

git checkout

cd ..

mkdir codeql-home

cd codeql-home/

git clone https://github.com/github/codeql.git codeql-repo

download and unpack codeql-cli from https://github.com/github/codeql-cli-binaries/releases

export PATH=$PATH:/home/user/codeql-home/codeql

echo $PATH

codeql resolve languages

codeql resolve qlpacks

cd ..

cd FreeCAD/

codeql database create codeql-cpp --language=cpp

codeql database analyze codeql-cpp cpp-lgtm.qls --format=csv --output=cpp-results.csv --threads=4

codeql database analyze codeql-cpp cpp-lgtm.qls --format=sarif-latest --output=cpp-results.sarif --sarif-add-file-contents

codeql database cleanup codeql-cpp

codeql dataset cleanup codeql-cpp

A small note on the commands that run the scan, CodeQL supports the analysis of both C++ and python code (and others). In above example the commands that run the scan and analysis are setup for the C++ code, since scanning of the python code we normally do automatically over the LGTM website. For running the scanner manually on the python code the last few commands from the above example would be...

cd ..

cd FreeCAD/

cd src/

codeql database create codeql-python --language=python

codeql database analyze codeql-python python-lgtm.qls --format=csv --output=python-results.csv --threads=4

codeql database analyze codeql-python python-lgtm.qls --format=sarif-latest --output=python-results.sarif --sarif-add-file-contents

codeql database cleanup codeql-python

codeql dataset cleanup codeql-python
User avatar
chennes
Veteran
Posts: 3910
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: LGTM (Semmle CodeQL)

Post by chennes »

As noted up-topic, running the entire C++ build through LGTM times out. But this morning a new PR was accepted that enabled LGTM for the core of our C++ codebase. No analysis of the GUI, and with many of the modules disabled:

Code: Select all

"cmake ./ -DPYTHON_EXECUTABLE=/usr/bin/python3 -DBUILD_GUI=OFF -DBUILD_ARCH=OFF\
        \ -DBUILD_DRAWING=OFF -DBUILD_IMAGE=OFF -DBUILD_INSPECTION=OFF -DBUILD_OPENSCAD=OFF\
        \ -DBUILD_RAYTRACING=OFF -DBUILD_REVERSEENGINEERING=OFF -DBUILD_SURFACE=OFF -DBUILD_START=OFF\
        \ -DBUILD_ROBOT=OFF -DBUILD_PATH=OFF -DBUILD_FEM=OFF"
Alas, this will probably affect our "A+" current overall ranking (kudos to all the Python devs writing such clean code!), but I imagine we'll get it back over time. I don't know how long it takes for LGTM to start displaying results from a new language, but hopefully it shows up in the next day or two.
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
User avatar
saso
Veteran
Posts: 1924
Joined: Fri May 16, 2014 1:14 pm
Contact:

Re: LGTM (Semmle CodeQL)

Post by saso »

Ok, interesting... I just hope it will not mess up everything since LGTM is so thoroughly tracking all the files and commits over time :| lets wait and see. As a side note, it is not that we did not have the results of the C++ analysis from LGTM (CodeQL), we have just been doing it manually.
User avatar
saso
Veteran
Posts: 1924
Joined: Fri May 16, 2014 1:14 pm
Contact:

Re: LGTM (Semmle CodeQL)

Post by saso »

New report from build 0.19.23754 for the c++ code has been created...
User avatar
saso
Veteran
Posts: 1924
Joined: Fri May 16, 2014 1:14 pm
Contact:

Re: LGTM (Semmle CodeQL)

Post by saso »

chennes wrote: Sat Jan 30, 2021 8:50 pm ...
By the way Chris, did it hurt? Hopefully no broken bones... :)
Attachments
droop.png
droop.png (10.62 KiB) Viewed 2246 times
User avatar
chennes
Veteran
Posts: 3910
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: LGTM (Semmle CodeQL)

Post by chennes »

saso wrote: Fri Feb 12, 2021 11:33 am did it hurt? Hopefully no broken bones... :)
:lol: During the 0.19 feature freeze several devs have turned their attention to the static analysis tools to help find bugs... nice to see that sharp drop there! Thanks for keeping up with these. I'm working on a tool to help prune the SARIF files to see how to best make use of those other results, too.
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
User avatar
Kunda1
Veteran
Posts: 13434
Joined: Thu Jan 05, 2017 9:03 pm

Re: LGTM (Semmle CodeQL)

Post by Kunda1 »

Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
Post Reply