Hitting some limitations with Cloud workbench / App::Link

[vejmarie]

Hi,

While implementing support for Cloud based import/export of Documents using App::Link I am hitting a limitation, that I liked to discuss with the community.

Currently everything is implemented within the Cloud Module (user account setup, Reader / Writer class expansion, and python interface).
As to support restore of Document which contains Cloud based entry which replace filename by a URI (aka something like https://localhost:443/<document_name>), the App::Link object needs to know how to read such "file". Right now it relies on App objects standard reader, which knows only local file storage. The URI approach is known by the Cloud::Reader object which is sitting into a module, that I can't use, as it is a cycling dependancy.

I have 2 options (maybe more, but this is why I would like to open the discussion).

- Either keep moving forward with the Cloud workbench as it is, but in the case of a link save/restore operation, everything will be done by the Cloud module. I believe I can manage it, by a basic "hack" which will parse the Document.xml file to identify all Links XML entry, and open the missing Documents locally before sending the Document.xml file to the interpreter and traditionnal App::Link workflow. In that case, as all pointed Document will be soon opened, it shall work. The save phase will be easier, as App::Link do not save attached Documents. This approach has a strong limitation it doesn't allow to link local and remote storage document, or to save the Document which contains the links locally. Everything will be remote.

- The other option is to integrate the Cloud object, into App. This could allow to mix everything, and handle local store operation. The App::Link will handle everything regarding opening associated Document, and could learn how to handle URI file name.

Any other option you might be envisioning ?

Option 1 seems easier for me to implement short term. Option 2 requires to integrate part of Cloud workbench into FreeCAD core, and move user credentials management into it.

Let me know what you think. I am currently thinking at moving forward with option 1 assuming that the limitation is not going to be a showstopper, but still might be perceived as a weird behaviour. When option 1 will be done, and stable enough, I might be moving to option 2 of 0.20/0.21 ?

vejmarie

[Kunda1]

@vejmarie perhaps you need to ping the relevant devs to comment on your post? The forum is bursting these days so your post may not get the required exposure it needs.

[yorik]

I think the whole topic of allowing App::Links to fetch something from the internet is something sensitive to be discussed very well before implementing. Certainly we'd need @Realthunder's input here because it's his baby and he will probably have ideas on the matter already.

[aapo]

I think the whole topic of allowing App::Links to fetch something from the internet is something sensitive to be discussed very well before implementing. Certainly we'd need @Realthunder's input here because it's his baby and he will probably have ideas on the matter already.

That is certainly true. I'm just a FreeCAD user, not a developer, but I was thinking along the same lines. @vejmarie 's first option would definitely be much safer, as the security implications would be limited to just this one workbench. The second option would, of course, be much more better and nicer, but it'd involve rather big additions to the FreeCAD core, as some kind of .FCStd file signing-off and/or keyring systems would be needed in order to avoid automated downloading of malicious files from the Internet. And designing a good security solution is never easy or fast...

[Kunda1]

[vejmarie]

I don't think the security is going to be a massive challenge with this option. All the required technologies to make it bulletproof exist today, and it will be as safe as opening a local FreeCAD file.

By the way, currently and during the next couple of months (year), the feature is activated only at compile time with preprocessor switch. Our default built doesn't integrate it except some that I generate.

I will work on option 1 first. It is easy to do.

The addition to freeCAD core is not that big, by the way and will be "easy" after option 1 will be done. The thing is that i will have to rewrite some code, as the data structure will be addressed differently

[realthunder]

App::Link does not handle files directly. It just saves the file name as a simple string and pass it to the core for loading. Although it does have a few use of QFilePath/QDir to manipulate the file path, but it is easy to make it work for URL as well. So if you are going to do this, most of the changes required does not really affect Link.

That being said. I really do not recommend adding cloud and networking stuff directly into the core. Like I have suggested in the beginning when you started this Cloud workbench, it will be much better to handle it in DocumentObserver and do the network syncing asynchronously, and the core should always deal with native files without latency.

[vejmarie]

App::Link does not handle files directly. It just saves the file name as a simple string and pass it to the core for loading. Although it does have a few use of QFilePath/QDir to manipulate the file path, but it is easy to make it work for URL as well. So if you are going to do this, most of the changes required does not really affect Link.

Yes I have made such modifications (with relevant pre processor defines), and it works. I agree it doesn't change anything related to the Link structure.

That being said. I really do not recommend adding cloud and networking stuff directly into the core. Like I have suggested in the beginning when you started this Cloud workbench, it will be much better to handle it in DocumentObserver and do the network syncing asynchronously, and the core should always deal with native files without latency.

I made an implementation with asynchronous save/restore, and got a couple of users complaining about the asynchronous approach of the "file" operation. Most of them prefer to wait and being sure that everything went smoothly.

I bet one day we will be ending up with such code into core, we are not ready yet, but I bet

Any idea on how to overload the file opening operation for XLink ? Roughly PropertyLinks is used to restore missing Documents when an XLink element is detected into Document.xml. I need to catch up that call before it happens as to ensure that I read it remotely instead of locally when an URI is detected within the filename. I can't do it by calling back the module, so I need to attach the code in some places, but do not find where, and how to be sure that my code is executed instead of yours when such node is detected.

My current strategy due to that sucks. I am reading Document.xml and before restoring it into FreeCAD Document, I am parsing it through a DOM XML parser instead of a SAX2 as to extract all XLink elements, and check if they are soon opened or not, and if they are an URI. if that is not the case, I am reading the associated Document through the Cloud interface. Everything is done into the Cloud module in such case.

[realthunder]

Any idea on how to overload the file opening operation for XLink ? Roughly PropertyLinks is used to restore missing Documents when an XLink element is detected into Document.xml. I need to catch up that call before it happens as to ensure that I read it remotely instead of locally when an URI is detected within the filename. I can't do it by calling back the module, so I need to attach the code in some places, but do not find where, and how to be sure that my code is executed instead of yours when such node is detected.

App::Link calls App::Application::addPendingDocument() to queue the file name, which gets processed in App::Application::openDocuments(), which calls App::Application::openDocumentPrivate() to create an empty new document, and calls App::Document::restore() to actually opens a physical file and reads it.

[vejmarie]

Any idea on how to overload the file opening operation for XLink ? Roughly PropertyLinks is used to restore missing Documents when an XLink element is detected into Document.xml. I need to catch up that call before it happens as to ensure that I read it remotely instead of locally when an URI is detected within the filename. I can't do it by calling back the module, so I need to attach the code in some places, but do not find where, and how to be sure that my code is executed instead of yours when such node is detected.

App::Link calls App::Application::addPendingDocument() to queue the file name, which gets processed in App::Application::openDocuments(), which calls App::Application::openDocumentPrivate() to create an empty new document, and calls App::Document::restore() to actually opens a physical file and reads it.

I finally created a new Document Signal which is called within LinkXsetValue, the place where potential new Document check is perform when an XLink is detected within Document.xml. I attach the Cloud module as a listener to that new signal, and if the filename is a URL based format, the Cloud module take care about checking if the document exist or not. If not, it does create a new Doc, and load the related content into it before handling over the control back to LinkXsetValue which detect that the Document is open, and do not have to open the Document as it is soon open.

Please not that all of this is activated only if the cloud module is built. (through -DBUILD_CLOUD). The good news is that everything works, and I will have a look now how to create assembly through the ongoing work that you perform, and other community members.