Assembly4: XLink absolute filepath problem

Discussion about the development of the Assembly workbench.
project4
Posts: 192
Joined: Fri Jul 12, 2013 12:53 pm

Re: Assembly4: XLink absolute filepath problem

Postby project4 » Thu Aug 27, 2020 1:27 pm

aapo wrote:
Thu Aug 27, 2020 1:16 pm
I'm not re-inventing the wheel, just saying that realthunder's current relative-path solution already fulfills my needs quite nicely, thanks. And, I believe absolute file paths would introduce more problems than they solve.
Everything is still relative.
The problem with current code (without the fix) is that it checks physical path to find the relativity instead of using the logical file path.
The fix will solve the issue with symbolic links in the path and keep it in the logical way.

Thinking about it while writing...
Even with the physical location check as it was implemented now (before the fix), someone could symlink the file itself in the working directory, so there will be 2 different file names pointing to the same file and those will still be overridden without any warning.
Don't know why would someone do that, symlinking directories is more common case for what we're talking here.
aapo
Posts: 204
Joined: Mon Oct 29, 2018 6:41 pm

Re: Assembly4: XLink absolute filepath problem

Postby aapo » Thu Aug 27, 2020 1:43 pm

project4 wrote:
Thu Aug 27, 2020 1:27 pm
aapo wrote:
Thu Aug 27, 2020 1:16 pm
I'm not re-inventing the wheel, just saying that realthunder's current relative-path solution already fulfills my needs quite nicely, thanks. And, I believe absolute file paths would introduce more problems than they solve.
Everything is still relative.
The problem with current code (without the fix) is that it checks physical path to find the relativity instead of using the logical file path.
The fix will solve the issue with symbolic links in the path and keep it in the logical way.

Thinking about it while writing...
Even with the physical location check as it was implemented now (before the fix), someone could symlink the file itself in the working directory, so there will be 2 different file names pointing to the same file and those will still be overridden without any warning.
Don't know why would someone do that, symlinking directories is more common case for what we're talking here.
I know, but Zolko is proposing an option to also make it possible to use absolute file paths when saving documents, and I'm thinking this is not a good idea because of interoperability reasons. This would make it easy to load/save projects utilizing external part libraries; but it seems to me that it would be impossible to load such projects with different operating systems, because the part library paths would be absolute. I could be wrong, though, if there actually exists some kind of "C:\" conversion mechanism within FreeCAD.

Also, realthunder's second patch fix already implements a warning when saving a document with 2 different file names pointing to the same file. This, however, complicates part library use in some scenarios; hence the discussion.
Last edited by aapo on Thu Aug 27, 2020 4:42 pm, edited 1 time in total.
User avatar
Zolko
Posts: 1097
Joined: Mon Dec 17, 2018 10:02 am

Re: Assembly4: XLink absolute filepath problem

Postby Zolko » Thu Aug 27, 2020 2:01 pm

realthunder wrote:
Thu Aug 27, 2020 10:09 am
The files are still loaded as separate files. When linked file is modified and the user wants to save it, another check will be done, and warn the user with this dialog.
If there are 100 such files, the user will get 100 times that warning ?
try the Assembly4 workbench for FreCAD v0.19
install with Tools > Addon Manager > Assembly4 — tutorials here and here
project4
Posts: 192
Joined: Fri Jul 12, 2013 12:53 pm

Re: Assembly4: XLink absolute filepath problem

Postby project4 » Thu Aug 27, 2020 6:26 pm

What other CAD programs do in that case?
aapo
Posts: 204
Joined: Mon Oct 29, 2018 6:41 pm

Re: Assembly4: XLink absolute filepath problem

Postby aapo » Thu Aug 27, 2020 10:08 pm

project4 wrote:
Thu Aug 27, 2020 6:26 pm
What other CAD programs do in that case?
Zolko already explained it in a very detailed post, one page back in this very thread: https://forum.freecadweb.org/viewtopic. ... 30#p426692.

However, the way other (commercial) CAD programs have solved this admittedly quite a difficult problem might not be the right solution for FreeCAD. I argued against option 1, as it may limit the transfer of a project to a different kind of computer system, even if the user has all the required files. If I understood correctly, Zolko would want all the three options with user-selectable preferences. What I'd like to see would be that realthunder would fix the relative-path option to also support Zolko's use case through filesystem links, so that opening a file through different linkpaths would actually open the file just once, and just point the XLinks to the same target (so that editing or saving the file through either link would always affect the same file, and there would not be two separate memory instances). But, apparently, there is some technical obstacle to do just that. Thus, Zolko may well turn out to be quite right, if it won't be technically possible to solve the two-links-to-the-same-file problem without duplicate memory representations, which risk accidental overwrites of the real file.

So, admittedly, my arguments could well be wrong, because a) it would be possible to write a conversion utility/workbench/macro for file transfer between different operating systems, and b) such a conversion could be included into FreeCAD internally (and maybe it already is through Qt functions, I think there is such automatic conversion stuff there at least for '/' and '\' already). In the end, my fundamental argument could probably be just written as "I just don't like hardcoded stuff!" :D
realthunder
Posts: 1809
Joined: Tue Jan 03, 2017 10:55 am

Re: Assembly4: XLink absolute filepath problem

Postby realthunder » Thu Aug 27, 2020 11:47 pm

Zolko wrote:
Thu Aug 27, 2020 12:32 pm
Please: don't re-invent the wheel, this problem has been solved before !!! Just do it the proper way : absolute-path, relative-path, library-path.
Absolute-path is useful in 'some' cases, but very limited. Library-path is good, and it can do everything absolute-path can, I can add that in the form of a application environment variable type of thing, so that XLink can recognize path like ${LIBRARY_PATH}.

Zolko wrote:
Thu Aug 27, 2020 2:01 pm
If there are 100 such files, the user will get 100 times that warning ?
No. When the user clicks save button, the check will be performed on all 'modified' files that are going to be written, and report all conflict link path in the report view. The dialog will only appear once, and list at most two conflict instances, and show 'check report view for more', if there are more instances.

aapo wrote:
Thu Aug 27, 2020 10:08 pm
What I'd like to see would be that realthunder would fix the relative-path option to also support Zolko's use case through filesystem links, so that opening a file through different linkpaths would actually open the file just once, and just point the XLinks to the same target (so that editing or saving the file through either link would always affect the same file, and there would not be two separate memory instances). But, apparently, there is some technical obstacle to do just that. Thus, Zolko may well turn out to be quite right, if it won't be technically possible to solve the two-links-to-the-same-file problem without duplicate memory representations, which risk accidental overwrites of the real file.
The two-links-to-the-same-file problem is caused by the user outside of FreeCAD's control. There are valid use cases. I think we should keep our software from being 'too smart', and only warn user if there is real danger.
Try Assembly3 (latest version 0.11) along with my custom build of FreeCAD at here.
And if you'd like to show your support, you can donate through patreon, liberapay, or paypal
project4
Posts: 192
Joined: Fri Jul 12, 2013 12:53 pm

Re: Assembly4: XLink absolute filepath problem

Postby project4 » Fri Aug 28, 2020 5:43 am

aapo wrote:
Thu Aug 27, 2020 10:08 pm
Zolko already explained it in a very detailed post, one page back in this very thread: https://forum.freecadweb.org/viewtopic. ... 30#p426692.
There are other CAD programs as well :)
I thought it was zolko's proposal for the solution, does NX really let the user select if the path is relative or absolute?

realthunder,
what if you will keep both the absoluteFilePath and canonicalFilePath returned values of open files?
On next open event it will be possible to run through all already open files and check if that physical location of the file (canonicalFilePath) was already open?
So the warning will pop earlier.
The linked parts could be marked as error/warning, similar to the case where the linked file can't be open or there are other errors (red in tree view, exclamation mark).
realthunder
Posts: 1809
Joined: Tue Jan 03, 2017 10:55 am

Re: Assembly4: XLink absolute filepath problem

Postby realthunder » Fri Aug 28, 2020 11:26 pm

project4 wrote:
Fri Aug 28, 2020 5:43 am
realthunder,
what if you will keep both the absoluteFilePath and canonicalFilePath returned values of open files?
On next open event it will be possible to run through all already open files and check if that physical location of the file (canonicalFilePath) was already open?
So the warning will pop earlier.
The warning will be printed in report view whenever a new file is loaded from an identical physical path of some existing document. The dialog warning is only popped for saving.
The linked parts could be marked as error/warning, similar to the case where the linked file can't be open or there are other errors (red in tree view, exclamation mark).
There is mechanism to mark object as error, but I don't think loading such file is serious enough to be considered as error. Unfortunately, there is no mechanism to mark object for warning.
Try Assembly3 (latest version 0.11) along with my custom build of FreeCAD at here.
And if you'd like to show your support, you can donate through patreon, liberapay, or paypal
User avatar
Zolko
Posts: 1097
Joined: Mon Dec 17, 2018 10:02 am

Re: Assembly4: XLink absolute filepath problem

Postby Zolko » Sun Aug 30, 2020 1:40 pm

project4 wrote:
Fri Aug 28, 2020 5:43 am
does NX really let the user select if the path is relative or absolute?
Yes. What NX actually does is to set this option globally, but also per-directory. So if you start NX and then open a file, you'll have the global settings, and when you open an NX file, which then starts NX, you'll have the directory settings (if available).

CATIA, if I remember correctly, tries first with the absolute path, and if the file is not available, tries the relative path. Which is a horrible behaviour, because if you duplicate a directory containing an assembly, to try some new design for example, you'll open the new assembly but which references the old parts, so you may modify the old parts instead of the new ones inadvertently. They do this to force to use their PLM (Ennovia for v5 I think, and built-in with v6).
try the Assembly4 workbench for FreCAD v0.19
install with Tools > Addon Manager > Assembly4 — tutorials here and here
wmayer
Site Admin
Posts: 16648
Joined: Thu Feb 19, 2009 10:32 am

Re: Assembly4: XLink absolute filepath problem

Postby wmayer » Mon Aug 31, 2020 12:01 pm

My physical path include symbolic links and on some PCs also a NFS mount, that's probably that cause the problem along the way.
In the following thread the offered solution is to replace QFileInfo::canonicalFilePath() with QFileInfo::absoluteFilePath().

However, it has never become really clear why exactly the current implementation fails. So, to examine the issues I did the following steps:
  1. Create the directory /tmp/test/real_dir/asm
  2. In /tmp/test create a soft link called link_dir to real_dir
  3. Copy the assembly project from here to /tmp/test/real_dir/asm
Now if I open the file test asm4.FCStd via /tmp/test/real_dir/asm everything works as expected. If I open it instead via /tmp/test/link_dir/asm the problems start. The error message is:
11:58:17 8.7e-08 <App> Document.cpp(2842): test_asm4#_x25mm_round_shaft.LinkedObject: Link not restored
Linked object: Model
Linked file: 6x25mm round shaft.FCStd
11:58:17 0.000474066 <App> Document.cpp(2842): test_asm4#_mm_lighweight_hub.LinkedObject: Link not restored
Linked object: Model
Linked file: 6mm lighweight hub.FCStd
And the loaded test asm4 document is marked as touched.

What happened is:
When loading a project then inside App::DocInfo::get() an entry to a map is added with information about the document. As key the absolute file path with the soft link is used.

When trying to search for the further assembly files while loading the project the function App::DocInfo::restoreDocument() is used. In this function the path was created with QFileInfo::canonicalFilePath() and thus the soft link has been resolved to the real path.
This path is used to search in the map but because it has no matching key the second file won't be loaded.

This explains why the whole assembly project cannot be loaded completely.
I don't understand how the file location is handled by freecad and ASM4, but it looks like the path is not saved as "/home/alex/FreeCAD/test/filename.FCStd" as the user sees is when saving the file, but something like "../../../../../mnt/storage/FreeCAD/test/filename.FCStd"
The reason for this mess is caused by computing a relative path from two absolute paths with a different directory hierarchy. You can easily test it with:

Code: Select all

from PySide2 import QtCore
docDir = QtCore.QDir("/home/alex/FreeCAD/test")
docDir.relativeFilePath("/mnt/storage/FreeCAD/test")
# gives: '../../../../mnt/storage/FreeCAD/test'
The function call QDir::relativeFilePath() is used in App::DocInfo::getDocPath() and it will we assigned to the filePath member of the PropertyXLink.
You will break your model as soon as you save the document because it then writes this broken path into the Document.xml

Btw:
Uncompressed the file, in Document.xml I can see the problematic paths, such as:
<XLink file="../../../../../media/storage/home/alex/Workspace/FreeCAD/Mahsanatron/v-wheel set 13mm.FCStd"
Since it's guaranteed that the very first file of a valid FCStd project is Document.xml you can directly read its content with zless test\ asm4.FCStd
Thinking about it while writing...
Even with the physical location check as it was implemented now (before the fix), someone could symlink the file itself in the working directory, so there will be 2 different file names pointing to the same file and those will still be overridden without any warning.
Yes and no. It's mainly an issue when you create symlinks to directories. However, when you create soft links or hard links to the actual file the situation is different:

The default behaviour of FreeCAD when overwriting an existing file is that it creates a separate file with a GUID as suffix and if the write process was successful it renames or removes (depends on the number of backup files) the original file and removes the GUID from the new file.
This means you can load the actual file, a soft-linked file or a hard-linked file without worrying that you overwrite files each other. What will happen is that you will end up with three independent files.

Now you can switch off this behaviour by adding the boolean parameter User parameter:BaseApp/Preferences/Document/BackupPolicy and set it to false. In this case when saving a document the existing file will be used directly to write the content to.

And with the PR applied there is still a certain risk that you overwrite a file by accident:
it happens when you create a hard link to the original project and if BackupPolicy is set to false. With the Qt API there is no way to see that both files are identical in the file system and thus the check to warn the user will fail.