We Need Text-based File Format

Have some feature requests, feedback, cool stuff to share, or want to know where FreeCAD is going? This is the place.
Forum rules
Be nice to others! Read the FreeCAD code of conduct!
dxdt
Posts: 4
Joined: Thu Jan 14, 2021 12:47 am

We Need Text-based File Format

Post by dxdt »

I'm writing this because I think this is not getting enough attention: WE NEED TEXT-BASED FILE FORMAT

It is not a good idea to track compressed files with VCS. Compressed files have high entropy, making it difficult for VCS to optimise repo. I know FCStd files are basically zip files and I can always decompress them into a directory before making a commit and compress them back again but it's just too much manual work. Something can definitely go wrong in the process.
When I write commit logs, it's basically like swimming in the dark because `git diff` shows "binary file". I keep some sort of "work log" on my notepad so I know what to write on commit logs. I find this rather inefficient.
I don't know why they decided to zip everything from the get-go. I know there's no point in saying "I would've done differently", but doesn't it make sense to build things from bottom(putting everything in a directory) to top(bundled file format)? Well, I guess you can say FreeCAD worried about prior-art competitors like Adobe and AutoDesk. People are used to the single file concept. But there's projects like LibreOffice that supports .fods, so you can't say it's not possible.

What's the hold up? Why didn't anyone implement this 7 years ago? Is anyone already working on it? Or has anyone worked on it and the pull request is still open for 7 years? Please, I'd like to know because I'm about to go implement it myself.

So, to sum it up: there's problems when it comes to using VCS for FreeCAD files.
  • Compressed files are not good with VCS
  • It's not possible to diff FCstd files without a special tool
I think merging is a different problem. It's a good start.

Here's the list of forum posts I found:
chrisb
Veteran
Posts: 53920
Joined: Tue Mar 17, 2015 9:14 am

Re: We Need Text-based File Format

Post by chrisb »

Hi and welcome to the forum!

Thanks for the offer to work on this!
dxdt wrote: Thu Jan 14, 2021 2:21 am Why didn't anyone implement this 7 years ago?
Choose one of the following to that recurring question
- there weren't enough developers around
- there weren't appropiate developers around
- there were more important things to do
- nobody was paid for it
- (insert anything you want)
Is anyone already working on it?
You found already realthunder's topic. Did you check if it is implemented in his branch?
A Sketcher Lecture with in-depth information is available in English, auf Deutsch, en français, en español.
User avatar
yorik
Founder
Posts: 13640
Joined: Tue Feb 17, 2009 9:16 pm
Location: Brussels
Contact:

Re: We Need Text-based File Format

Post by yorik »

It's not so simple... While indeed 95% of what you have in an .FCStd file is indeed text, and the remaining 5% could be converted to text, there would be very little advantage. Most part of these 95% are brep files, which are text, but that you can barely call human-readable. Opening a FreeCAD file and changing just a couple of things would change a lot of stuff all around the brep structure, that would have little or no meaning in a diff, it would be mostly numbers and tabs...

In practice, 95% of the text file would probably be changed with each commit. Unreadable by a human, and very little storage gain with vcs (maybe even negative, if you consider the compression offered by zip).

That said, it's not such a complicated thing to do. I'd say the largest part is to define how to efficiently format that format... It could easily be implemented as a python script, then, if useful, integrated better in FreeCAD.
User avatar
chennes
Veteran
Posts: 3878
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: We Need Text-based File Format

Post by chennes »

dxdt wrote: Thu Jan 14, 2021 2:21 am When I write commit logs, it's basically like swimming in the dark because `git diff` shows "binary file". I keep some sort of "work log" on my notepad so I know what to write on commit logs. I find this rather inefficient.
What would you expect a diff to show you if working with text files? It's not like it's going to say

Code: Select all

- Line from A to B
+ Arc from C to D
As @yorik points out, converting to text doesn't actually solve the problem that it looks like you are trying to solve. So it might be more fruitful to discuss first exactly what that is, and second how it can be addressed. I'm interested in versioning, but don't see text-based systems as a viable way forward. I think of it more like a backup problem, where you want named/labelled snapshots in time that you can distinguish from one another and go back to if necessary. FreeCAD already has the ability to have thumbnails embedded in its files (which can help you distinguish one from the other) as well as a ton of other metadata. A "simple" first cut might be to make FCStd files self-embeddable and add a "take snapshot"-type feature. The naive implementation would result in file sizes getting quite large (I imagine) but there is room for refinement using the exact same strategy that incremental backups use.
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
HardRock
Posts: 133
Joined: Tue Mar 13, 2018 5:50 am
Location: Russian Federation

Re: We Need Text-based File Format

Post by HardRock »

Text files is a bad idea, this wont help to solve your "problem" becase 3d model is not a source file.
Even with source code filese there are number of situations where you need to merge commits manually.

I keeps my FreeCAD projects int GIT, and this is not a problem. Commits stores whole file, in commit message you may describe your changes.
drmacro
Veteran
Posts: 8865
Joined: Sun Mar 02, 2014 4:35 pm

Re: We Need Text-based File Format

Post by drmacro »

HardRock wrote: Thu Jan 14, 2021 5:40 pm Text files is a bad idea, this wont help to solve your "problem" becase 3d model is not a source file.
Even with source code filese there are number of situations where you need to merge commits manually.
Indeed!, we need a thumbs up emoji. :mrgreen:
I keeps my FreeCAD projects int GIT, and this is not a problem. Commits stores whole file, in commit message you may describe your changes.
Indeed again!
Star Trek II: The Wrath of Khan: Spock: "...His pattern indicates two-dimensional thinking."
chrisb
Veteran
Posts: 53920
Joined: Tue Mar 17, 2015 9:14 am

Re: We Need Text-based File Format

Post by chrisb »

drmacro wrote: Thu Jan 14, 2021 7:08 pm we need a thumbs up emoji.
:+1:
A Sketcher Lecture with in-depth information is available in English, auf Deutsch, en français, en español.
openBrain
Veteran
Posts: 9034
Joined: Fri Nov 09, 2018 5:38 pm
Contact:

Re: We Need Text-based File Format

Post by openBrain »

dxdt wrote: Thu Jan 14, 2021 2:21 am Compressed files are not good with VCS

Or maybe VCS are not good with compressed files...

It's not possible to diff FCstd files without a special tool

Yes but that's possible. We could even imagine developing and integrating a tool that would perform it. Did you (or someone else) ever tried to different 2 sldprt files?
drmacro
Veteran
Posts: 8865
Joined: Sun Mar 02, 2014 4:35 pm

Re: We Need Text-based File Format

Post by drmacro »

chrisb wrote: Thu Jan 14, 2021 9:16 pm
drmacro wrote: Thu Jan 14, 2021 7:08 pm we need a thumbs up emoji.
:+1:
interesting, I didn't know that one...but I was really referring to the cluster labeled Smilies in the text entry window of phpbb reply.
;)
Star Trek II: The Wrath of Khan: Spock: "...His pattern indicates two-dimensional thinking."
lukasubo
Posts: 45
Joined: Sun Jun 05, 2016 5:41 pm

Re: We Need Text-based File Format

Post by lukasubo »

git can be configured to use a custom command for diffs anyways - some do it already for FCStd files, a utility for showing some file info is included with FreeCAD.

https://github.com/FreeCAD/FreeCAD/blob ... ols/fcinfo

The source file linked above includes usage instructions.

CAD files aren't code, so it doesn't make sense to try to mash them through VCS designed for code. It just ends up being one big compromise on all fronts. Not to mention FCStd files are absolutely massive uncompressed IIRC.

Ideally, for CAD, you'd want some basic metrics about the file (bodies, volume, etc.) along with a model compare. Below is a screenshot of NX's model compare, which works fairly well.
maxresdefault.jpg
maxresdefault.jpg (124.77 KiB) Viewed 1178 times
The below video shows a fairly recent version of that same tool. You usually have to tweak the display options a bit to get exactly what you need, but setting old and new to show as different coloured wireframes is my preference, with unchanged edges shown in gray.

https://www.youtube.com/watch?v=20D4ObvqOaA

Another way I usually compare CAD file revisions is by superimposing the two technical drawings, like you might on a light table. Often engineering part revisions have no actual geometric changes and are just drawing corrections or tolerance adjustments.
Post Reply