Feature #2738 Provide Hash source installation

Having trouble installing or compiling FreeCAD? Get help here.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Post Reply
User avatar
Kunda1
Veteran
Posts: 13434
Joined: Thu Jan 05, 2017 9:03 pm

Feature #2738 Provide Hash source installation

Post by Kunda1 »

In issue #2738 OP makes a good point:
Please, provide Hash information of the binaries in all versions and OS of FreeCAD on the download page product.
In deed, this information is important to be sure that integrity installation source is ok and can be trust.
is the team thinking about this?
I remember that sha256 is what homebrew uses (from my OSX days)
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
wmayer
Founder
Posts: 20243
Joined: Thu Feb 19, 2009 10:32 am
Contact:

Re: Feature #2738 Provide Hash source installation

Post by wmayer »

I think this is a point that becomes more and more important these days. Currently sgrogan does most of the work to upload installers and developer snapshots so it's best to assign it to him.
User avatar
sgrogan
Veteran
Posts: 6499
Joined: Wed Oct 22, 2014 5:02 pm

Re: Feature #2738 Provide Hash source installation

Post by sgrogan »

Kunda1 wrote:is the team thinking about this?
I have this on my list.
Current plan is provide the hash for future release assets and eventually to add it to the dailys in the Travis/Appveyor scripts.
It can be assigned to me.
EDIT: I see that wmayer and I are of the same mind.
"fight the good fight"
User avatar
sgrogan
Veteran
Posts: 6499
Joined: Wed Oct 22, 2014 5:02 pm

Re: Feature #2738 Provide Hash source installation

Post by sgrogan »

With all the help with packaging lately, Thanks everyone :) , I think I have some time to address this.
I'm looking for opinions/suggestions how it should work.
"fight the good fight"
User avatar
Kunda1
Veteran
Posts: 13434
Joined: Thu Jan 05, 2017 9:03 pm

Re: Feature #2738 Provide Hash source installation

Post by Kunda1 »

What type of Hash should we use?
IIRC, I think Homebrew uses SHA256

Do we want to use GPG?
This would be a nice extra step.
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
rolandog
Posts: 3
Joined: Fri Apr 20, 2018 3:33 pm

Re: Feature #2738 Provide Hash source installation

Post by rolandog »

Hello, @sgrogan

I just contributed a script (and added it as a note to the issue at the bug tracker) for the automated creation of digests of SHA-256 checksums in a directory. Here's the code in case anyone wants to try it as well:

Code: Select all

# -*- coding: utf-8 -*-
"""
Script to create SHA-256 digest of all files in the current directory

Based on an answer by Richard Neumann on Code Review
https://codereview.stackexchange.com/a/147191

Based on recommendation that file digests are created nowadays
https://en.wikipedia.org/wiki/File_verification#File_formats

Created on Fri Apr 20 12:07:41 2018

@author: rolandog
"""


from os import getcwd, listdir
from os.path import join, isfile
from time import strftime
from hashlib import sha256


def list_files(basedir=None):
    """List only files within the respective directory"""

    if basedir is None:
        basedir = getcwd()

    for item in listdir(basedir):
        path = join(basedir, item)

        # skip listing a hash of our hash digest
        if "sha256-digest" in item:
            continue

        if isfile(path):
            # changed so that we get the path and the filename
            yield (path, item)


def sha256sum(file_name, block_size=None):
    """Returns the sha256 checksum of the respective file"""

    if block_size is None:
        block_size = 4096

    checksum = sha256()

    with open(file_name, "rb") as file_handle:
        block = file_handle.read(block_size)

        while block:
            checksum.update(block)
            block = file_handle.read(block_size)

    return checksum.hexdigest()


def sha256sums(basedir=None, block_size=None):
    """Yields (<sha256sum>, <file_name>) tuples
    for files within the basedir.
    """

    for file_path, file_name in list_files(basedir=basedir):
        yield (sha256sum(file_path, block_size=block_size), file_name)


def create_sha256_digest(basedir=None, block_size=None, outputdir=None):
    """Creates de sha256-digest file with a timestamp"""

    hash_file_name = strftime("sha256-digest_%Y%m%d-%H%M%S")

    if outputdir is None:
        outputdir = getcwd()

    hash_file_path = join(outputdir, hash_file_name)

    with open(hash_file_path, "w") as file_handle:
        for file_hash in sha256sums(basedir, block_size):
            file_handle.write(" *".join(file_hash) + "\n")


if __name__ == "__main__":
    import argparse
    PARSER = argparse.ArgumentParser()
    PARSER.add_argument("-d",
                        "--directory",
                        default=None,
                        type=str,
                        required=False,
                        help="Path, str, to the directory of the files")
    PARSER.add_argument("-b",
                        "--blocksize",
                        default=None,
                        type=int,
                        required=False,
                        help="Block size, int, in bytes to read from files")
    PARSER.add_argument("-o",
                        "--outputdir",
                        default=None,
                        type=str,
                        required=False,
                        help="Output directory, str, for sha256 digest")
    ARGUMENTS = PARSER.parse_args()
    create_sha256_digest(basedir=ARGUMENTS.directory,
                         block_size=ARGUMENTS.blocksize,
                         outputdir=ARGUMENTS.outputdir)

Would this be the proper place to confirm the latest bugfix release (FreeCAD-0.17.13515.e17b340-WIN-x64-installer.exe) or should I create a new thread? Currently, the output for my script is
29653a667977724b38da6f6e7f7cea4bbd5e1a3b8df1b641c671e6a029d2f6b9 *FreeCAD-0.17.13515.e17b340-WIN-x64-installer.exe
rolandog
Posts: 3
Joined: Fri Apr 20, 2018 3:33 pm

Re: Feature #2738 Provide Hash source installation

Post by rolandog »

I updated my script so that it has a permissive license and that it can also output individual sha256 files (if needed), though it defaults to a digest of checksums and filenames:

Code: Select all

# -*- coding: utf-8 -*-
"""
Purpose
=======
Creates a SHA-256 digest of files in a directory


Attributions
============
Based on an answer by Richard Neumann on Code Review
https://codereview.stackexchange.com/a/147191


Based on statement that file digests are considered best-practice as of 2012
https://en.wikipedia.org/wiki/File_verification#File_formats


Creation and Modification Times
===============================
Created on Fri Apr 20 12:07:41 2018
Last Modified on Mon Apr 23 09:50:21 2018


License
=======
License: Creative Commons CC0
License URL: https://creativecommons.org/publicdomain/zero/1.0/legalcode


Changelog
=========
* 0.0.2
    - Added option to request individual files instead of digest
    - Don't output hash of currently running script
    - Added changelog
    - Added last modified date
    - Added example usage by developer and end-user
    - Added Creative Commons CC0 License
* 0.0.1
    - Initial release.


@author: rolandog
@version: 0.0.2
"""


from os import getcwd, listdir
from os.path import join, isfile, basename
from time import strftime
from hashlib import sha256
from textwrap import dedent


def list_files(basedir=None):
    """List only files within the respective directory"""

    if basedir is None:
        basedir = getcwd()

    # gets the name of the running script
    script_name = basename(__file__)

    for item in listdir(basedir):
        path = join(basedir, item)

        # don't make a hash of a hash file or current file
        if "sha256-digest" in item or ".sha256" in item or script_name in item:
            continue

        if isfile(path):
            # changed so that we get the path and the filename
            yield (path, item)


def sha256sum(file_name, block_size=None):
    """Returns the sha256 checksum of the respective file"""

    if block_size is None:
        block_size = 4096

    checksum = sha256()

    with open(file_name, "rb") as file_handle:
        block = file_handle.read(block_size)

        while block:
            checksum.update(block)
            block = file_handle.read(block_size)

    return checksum.hexdigest()


def sha256sums(basedir=None, block_size=None):
    """Yields (<sha256sum>, <file_name>) tuples
    for files within the basedir.
    """

    for file_path, file_name in list_files(basedir=basedir):
        yield (sha256sum(file_path, block_size=block_size), file_name)


def create_sha256_digest(basedir=None,
                         block_size=None,
                         outputdir=None,
                         individual=False):
    """Creates de sha256-digest file with a timestamp"""

    if outputdir is None:
        outputdir = getcwd()

    if individual is False:
        hash_file_name = strftime("sha256-digest_%Y%m%d-%H%M%S")
        hash_file_path = join(outputdir, hash_file_name)

        with open(hash_file_path, "w") as file_handle:
            for file_hash in sha256sums(basedir, block_size):
                file_handle.write(" *".join(file_hash) + "\n")
    else:
        for checksum, file_name in sha256sums(basedir, block_size):
            hash_file_name = file_name + ".sha256"
            hash_file_path = join(outputdir, hash_file_name)
            with open(hash_file_path, "w") as file_handle:
                file_handle.write(" *".join((checksum, file_name)) + "\n")


if __name__ == "__main__":
    from argparse import ArgumentParser, RawDescriptionHelpFormatter
    DESCRIPTION = "Creates a SHA-256 digest of files in a directory"
    EPILOG = """\
        example usage by developer
        --------------------------
        ls
            hello-world.txt  sha256digest.py

        cat hello-world.txt
            Hello, World!

        python sha256digest.py -i

        ls
            hello-world.txt  hello-world.txt.sha256  sha256digest.py

        cat hello-world.txt.sha256
            c98c24b677eff44860afea6f493bbaec5bb1c4cbb209c6fc2bbb47f66ff2ad31 *hello-world.txt


        example usage by end-user
        -------------------------
        ls
            hello-world.txt  hello-world.txt.sha256

        sha256sum -c hello-world.txt.sha256
            hello-world.txt: OK
        """
    PARSER = ArgumentParser(prog="sha256digest.py",
                            description=DESCRIPTION,
                            formatter_class=RawDescriptionHelpFormatter,
                            epilog=dedent(EPILOG))
    PARSER.add_argument("-i",
                        "--individual",
                        default=False,
                        action="store_true",
                        required=False,
                        help="outputs one hash file per file in folder")
    PARSER.add_argument("-d",
                        "--directory",
                        default=None,
                        type=str,
                        required=False,
                        help="path to the folder containing the files")
    PARSER.add_argument("-b",
                        "--blocksize",
                        default=None,
                        type=int,
                        required=False,
                        help="read files in chunks less than BLOCKSIZE bytes")
    PARSER.add_argument("-o",
                        "--outputdir",
                        default=None,
                        type=str,
                        required=False,
                        help="output directory for sha256 digest or files")
    ARGUMENTS = PARSER.parse_args()
    create_sha256_digest(basedir=ARGUMENTS.directory,
                         block_size=ARGUMENTS.blocksize,
                         outputdir=ARGUMENTS.outputdir,
                         individual=ARGUMENTS.individual)

Hopefully it is of some use during the build process.
User avatar
sgrogan
Veteran
Posts: 6499
Joined: Wed Oct 22, 2014 5:02 pm

Re: Feature #2738 Provide Hash source installation

Post by sgrogan »

rolandog wrote: Fri Apr 20, 2018 7:13 pm Would this be the proper place to confirm the latest bugfix release (FreeCAD-0.17.13515.e17b340-WIN-x64-installer.exe) or should I create a new thread? Currently, the output for my script is
I use certutil

Code: Select all

SHA256 hash of file FreeCAD-0.17.13515.e17b340-WIN-x64-installer.exe:
29 65 3a 66 79 77 72 4b 38 da 6f 6e 7f 7c ea 4b bd 5e 1a 3b 8d f1 b6 41 c6 71 e6
 a0 29 d2 f6 b9
Thanks for the script, I will look into incorporating this into the build process!
"fight the good fight"
rolandog
Posts: 3
Joined: Fri Apr 20, 2018 3:33 pm

Re: Feature #2738 Provide Hash source installation

Post by rolandog »

sgrogan wrote: Mon Apr 23, 2018 8:20 pm

Code: Select all

SHA256 hash of file FreeCAD-0.17.13515.e17b340-WIN-x64-installer.exe:
29 65 3a 66 79 77 72 4b 38 da 6f 6e 7f 7c ea 4b bd 5e 1a 3b 8d f1 b6 41 c6 71 e6
 a0 29 d2 f6 b9
Thanks for the checksum!

sgrogan wrote: Mon Apr 23, 2018 8:20 pm I use certutil
At first I thought that maybe I wasn't properly informed and that maybe there was a 'verb' that I didn't know about that could automate the verification of signatures; then I read the docs but couldn't find anything.

I thought of cooking up a (batch script) that would use Certutil, but output hashfiles in a more standard format (like the ones used to automate the checking of lists of files).

But then I found that David Benham (dbenham), over at DosTips had already made a pretty awesome script that is a sort of swiss-army knife for hashes in Windows. Maybe that could be of help for FreeCAD development on Windows, if you plan to stick to Certutil.

Also, on Windows, there's the Git Bash command-line shell that brings a lot of the cool features from Bash; you'd still need a small bash script, but it'd be much more compact. I'm basing this off of Phil Hollenback's answer at Serverfault:

Code: Select all

find . -type f -a \! -name SHA256SUMS -print0 | xargs -0 sha256sum >> SHA256SUMS
I just ran this on my Win10 machine, on the Git Bash shell. Then all I did to verify signatures was:

Code: Select all

sha256sum -c SHA256SUMS
./FreeCAD-0.17.13519.1a8b868-WIN-x64-installer.exe: OK
sgrogan wrote: Mon Apr 23, 2018 8:20 pm Thanks for the script, I will look into incorporating this into the build process!
No problem! The goal was to automate the generation of checksums for files so that end-users could also easily check hashes if they have either GPG4Win or Git for Windows installed.
Post Reply