Storing FreeCAD files in a git repo

[shoogen]

The idea behind this is, instead of creating a number of backups to check the File into a git repo. Checking in a compressed file renders the whole deduplication and delta compression useless.
In my concept every FreeCAD file has its own branch. It uses the python library dulwich, which implements the handling of git objects. The source is available at https://gist.github.com/5263/0f1554decb1cd329f4ee

Code: Select all

import zip2git
#create a new file and save it once using the gui to set the path
>>> App.getDocument("Unnamed2").saveAs("/home/username/freecad/newfile1.fcstd")
>>> zip2git.saveandcommit(FreeCAD.ActiveDocument,'../freecad')
#../freecad is the path to an existing (empty) git repo

Code: Select all

$ git log --pretty=fuller --stat newfile1 
commit 42a602c2724028a66eb2ff929d274d7bea403aec
Author:     unkown <unknown>
AuthorDate: Wed Dec 3 18:46:10 2014 +0000
Commit:     shoogen <shoogen@forum.freecadweb.org>
CommitDate: Wed Dec 3 17:46:10 2014 +0000

    New FreeCAD Document newfile1
    
    UUID: ed7d9934-eb2d-4b80-aeca-98be58cc6bb2
    
    CreationDate: Wed Dec  3 18:43:54 2014
    FileName: /home/username/freecad/newfile1.fcstd
    Label: newfile1
    LastModifiedDate: Wed Dec  3 18:46:10 2014
    License: CC-BY 3.0
    LicenseURL: http://creativecommons.org/licenses/by/3.0/

 DiffuseColor    |  Bin 0 -> 8 bytes
 Document.xml    |   79 +++++++++++++++
 GuiDocument.xml |   64 +++++++++++++
 PartShape.brp   |  287 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 430 insertions(+)

The author would be set to the "LastModifiedBy" value. But i didn't set before saving the file in FreeCAD. The Commiter date, is the actual time of the check-in (currently using UTC). The Author time is taken from the LastModifiedDate Property. (But it obviously has the wrong timezone. It should be +0100. But this information is only available if the file was last saved on the machine that runs this python script)

Code: Select all

>>> zip2git.saveandcommit(FreeCAD.ActiveDocument,'../freecad')

Code: Select all

commit af3187c35e728a48dd9782847bb349c4eee9fa1a
Author:     shoogen <unknown>
AuthorDate: Wed Dec 3 19:00:57 2014 +0000
Commit:     shoogen <shoogen@forum.freecadweb.org>
CommitDate: Wed Dec 3 18:00:57 2014 +0000

    Updated FreeCAD Document newfile1
    
    UUID: ed7d9934-eb2d-4b80-aeca-98be58cc6bb2
    
    Comment: it's a box 10mm x 10mm x 10 mm
    Company: ACME Inc.
    CreatedBy: shoogen
    CreationDate: Wed Dec  3 18:43:54 2014
    FileName: /home/username/freecad/newfile1.fcstd
    Label: newfile1
    LastModifiedBy: shoogen
    LastModifiedDate: Wed Dec  3 19:00:57 2014
    License: MIT License
    LicenseURL: http://opensource.org/licenses/MIT

 Document.xml |   14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

commit 42a602c2724028a66eb2ff929d274d7bea403aec
[...]

Obviously i changed some properties, but the shapes and colors remained the same.
Next i just changed the placement of the first PartFeature

Code: Select all

$ git show --pretty=short --stat newfile1 
commit 22596a24022a487df8d3c859c2eb7a75bc3d5c77
Author: shoogen <unknown>

    Updated FreeCAD Document newfile1

 Document.xml  |    4 ++--
 PartShape.brp |   18 +++++++++---------
 2 files changed, 11 insertions(+), 11 deletions(-)

This changes the Location and the Vertices in the respective Shape as well.

To load the latest version form the repo use

Code: Select all

>>> zip2git.checkoutandload('newfile1','../freecad')

to retrieve an older version you currently need to pass the full hash. The easiest way to get it is to use "git rev-parse" on the commandline or use some graphical fronted.
Of course the files could be extracted by any git tool and assembled to a FCStd file using the ProjectUtility in FreeCAD.

[saso]

Yes! I like this and have been thinking about it for different implementations now for some time... Also see some ideas for gui and ux on top of it here http://www.ht-timchen.com/research/nonl ... or-images/ (video and paper).

It is interesting to also think about how it can be done for different working environments like single user on one computer; single user on multiple computers (ex. different locations, home - office); multiple users in single office with no server; multiple users in single office with server; multiple users in multiple locations with own servers; multiple users in multiple locations with hosted servers (ex. amazon); possible integration with file hosting services (ex. github, dropbox, google drive,...); and more ?!

[shoogen]

One thing that i learned about recently is "git bundle". This allows you transfer git data even with means like google drive or dropbox. If you want to transfer data really regularly you would surely need some kind of git server. If it's dumb or smart is less important. But you want it to be continuously reachable.

[yorik]

Excellent shoogen.. I've been thinking a lot about this too but your implementation goes far further than that

One thing would be interesting to automate, is the linked git folder of a file... It would be cool if it was saved with the file somehow, so you could simply load the file, do some work and press a button to commit.

Maybe a new field in the document properties? It doesn't even need to be editable via the GUI...

Another idea would be to use an automatic folder, à la unix, for example a "myfile.fcstd.d" that would be at the same location as "myfile.fcstd"

[rockn]

Hi Shoogen,

I failed to use your script. And I am not sure how it really work.

What I do :
Create a repo on Github
Git clone this repo on my computer. (ex : /home/jo/Documents/FreeCAD/Projets-FC )
Make an object with FreeCAD.
Save it, where I want ??? (ex : /home/jo/Documents/FreeCAD/Projects01/Part01.fcstd)
Import zip2git in FC python console it's ok. (zip2git.py is in my macros folder)
I failed at saveandcommit

Code: Select all

>>> import zip2git
>>> zip2git.saveandcommit(FreeCAD.ActiveDocument,'/home/jo/Documents/FreeCAD/Projets-FC')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/jo/.FreeCAD/Macros/zip2git.py", line 277, in saveandcommit
    return commitfile(filename,repopath=repodir)
  File "/home/jo/.FreeCAD/Macros/zip2git.py", line 186, in commitfile
    repo=dulwich.repo.Repo(repopath)
AttributeError: 'module' object has no attribute 'repo'
>>> 

I have copy/paste the latest gist code at this moment.

What I am doing wrong or don't doing ?

edit :
OS: Ubuntu 14.04.1 LTS
Word size of OS: 64-bit
Word size of FreeCAD: 64-bit
Version: 0.15.4252 (Git)
Branch: master
Hash: c02590c666c6dd7d33bf3cb795497311b6badadb
Python version: 2.7.6
Qt version: 4.8.6
Coin version: 4.0.0a
OCC version: 6.7.0

[shoogen]

I failed to use your script. And I am not sure how it really work.

I suspect that you are using a different version of dulwich. Probably a more recent one. So far I've only used the dulwich wich is included in debian wheezy. Parhaps you can try to add "import dulwich.repo" after the "import dulwich" line.
I updated the gist to include this change.
You might need to do this for other submodules as well.

I have copy/paste the latest gist code at this moment.

pasting python code always brings the risk of screwing up indentation. But this seem to be fine in this case.

Git clone this repo on my computer. (ex : /home/jo/Documents/FreeCAD/Projets-FC )
[...]zip2git.saveandcommit(FreeCAD.ActiveDocument,'/home/jo/Documents/FreeCAD/Projets-FC')

Thats fine for a repo that could save more that one FreeCAD file.
If you just want to use it as backup system for a single file you could call
"zip2git.saveandcommit()" without additional parameters. This will create a Part01.fcstd.git directory containing a "bare" git repository. This use it effectively i created a FCMacro that allows me to assign a toolbar icon and a key binding.

The functions to checkout a FreeCAD file are not complete yet.
One problem is that changing the name/path of a file changes the respective properties in the document.xml. But those are used to determine the name of the branch. This means that the files present in the directory would need to be overwritten. On the other hand this should not happen quietly if the version persent as a file is not available in the repo.
The next big step you be to offer a GUI to choose the repo path and the repo branch.
But I'm not really keen in putting together a shiny GUI, by myself, by now. I'd rather see more progress on the Resource_framework_project
So there will consistent behavior among the interfaces for CADinet and the FreeCAD Parts library on github.

[rockn]

Thanks Shoogen, the fix did the tricks.

I could help for gui stuff, I have to fully understand the script before

[shoogen]

I added a Macro to open a file from repo using the gui. The Line edit on the top is the path to the repo. On the left are the refs (branches). It possible to select multiple at once. On the right are the commits of the selected branches.

checkoutgui1.png (72.73 KiB) Viewed 8433 times

[eukreign]

Has anyone looked at possibly making FreeCAD open/save mechanism not require a zip file?

Seems like if FreeCAD can be made to work with a directory instead of a zip file then it'll be significantly easier to use version control without having to use a script to zip/unzip the project file between FreeCAD and repository.

[shoogen]

Has anyone looked at possibly making FreeCAD open/save mechanism not require a zip file?
Seems like if FreeCAD can be made to work with a directory instead of a zip file[...]

I thought about it. But I came to the conclusion to avoid working with multiple files in a directory (for now). IMHO either FreeCAD should directly work on the backend (File, PLM interface, version control) or output a single file.
If FreeCAD would output multiple files, it raises a whole new class of problems. Would it create a new directory every time the proect is saved. How can the version control check if the last save completed and the file is correct. If the directory would be recreated it can't be used to store the repo (.git) inside.