Hitting some limitations with Cloud workbench / App::Link

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
User avatar
vejmarie
Posts: 623
Joined: Mon Jan 04, 2016 4:52 pm
Location: Somewhere between France, USA and Taiwan
Contact:

Hitting some limitations with Cloud workbench / App::Link

Postby vejmarie » Tue Jul 07, 2020 1:07 am

Hi,

While implementing support for Cloud based import/export of Documents using App::Link I am hitting a limitation, that I liked to discuss with the community.

Currently everything is implemented within the Cloud Module (user account setup, Reader / Writer class expansion, and python interface).
As to support restore of Document which contains Cloud based entry which replace filename by a URI (aka something like https://localhost:443/<document_name>), the App::Link object needs to know how to read such "file". Right now it relies on App objects standard reader, which knows only local file storage. The URI approach is known by the Cloud::Reader object which is sitting into a module, that I can't use, as it is a cycling dependancy.

I have 2 options (maybe more, but this is why I would like to open the discussion).

- Either keep moving forward with the Cloud workbench as it is, but in the case of a link save/restore operation, everything will be done by the Cloud module. I believe I can manage it, by a basic "hack" which will parse the Document.xml file to identify all Links XML entry, and open the missing Documents locally before sending the Document.xml file to the interpreter and traditionnal App::Link workflow. In that case, as all pointed Document will be soon opened, it shall work. The save phase will be easier, as App::Link do not save attached Documents. This approach has a strong limitation it doesn't allow to link local and remote storage document, or to save the Document which contains the links locally. Everything will be remote.

- The other option is to integrate the Cloud object, into App. This could allow to mix everything, and handle local store operation. The App::Link will handle everything regarding opening associated Document, and could learn how to handle URI file name.

Any other option you might be envisioning ?

Option 1 seems easier for me to implement short term. Option 2 requires to integrate part of Cloud workbench into FreeCAD core, and move user credentials management into it.

Let me know what you think. I am currently thinking at moving forward with option 1 assuming that the limitation is not going to be a showstopper, but still might be perceived as a weird behaviour. When option 1 will be done, and stable enough, I might be moving to option 2 of 0.20/0.21 ?

vejmarie
User avatar
Kunda1
Posts: 7990
Joined: Thu Jan 05, 2017 9:03 pm

Re: Hitting some limitations with Cloud workbench / App::Link

Postby Kunda1 » Tue Jul 07, 2020 1:03 pm

@vejmarie perhaps you need to ping the relevant devs to comment on your post? The forum is bursting these days so your post may not get the required exposure it needs.
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
User avatar
yorik
Site Admin
Posts: 12011
Joined: Tue Feb 17, 2009 9:16 pm
Location: Brussels, Belgium
Contact:

Re: Hitting some limitations with Cloud workbench / App::Link

Postby yorik » Tue Jul 07, 2020 1:40 pm

I think the whole topic of allowing App::Links to fetch something from the internet is something sensitive to be discussed very well before implementing. Certainly we'd need @Realthunder's input here because it's his baby and he will probably have ideas on the matter already.
aapo
Posts: 164
Joined: Mon Oct 29, 2018 6:41 pm

Re: Hitting some limitations with Cloud workbench / App::Link

Postby aapo » Tue Jul 07, 2020 3:30 pm

yorik wrote:
Tue Jul 07, 2020 1:40 pm
I think the whole topic of allowing App::Links to fetch something from the internet is something sensitive to be discussed very well before implementing. Certainly we'd need @Realthunder's input here because it's his baby and he will probably have ideas on the matter already.
That is certainly true. I'm just a FreeCAD user, not a developer, but I was thinking along the same lines. @vejmarie 's first option would definitely be much safer, as the security implications would be limited to just this one workbench. The second option would, of course, be much more better and nicer, but it'd involve rather big additions to the FreeCAD core, as some kind of .FCStd file signing-off and/or keyring systems would be needed in order to avoid automated downloading of malicious files from the Internet. And designing a good security solution is never easy or fast...
User avatar
Kunda1
Posts: 7990
Joined: Thu Jan 05, 2017 9:03 pm

Re: Hitting some limitations with Cloud workbench / App::Link

Postby Kunda1 » Tue Jul 07, 2020 3:35 pm

Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
User avatar
vejmarie
Posts: 623
Joined: Mon Jan 04, 2016 4:52 pm
Location: Somewhere between France, USA and Taiwan
Contact:

Re: Hitting some limitations with Cloud workbench / App::Link

Postby vejmarie » Wed Jul 08, 2020 12:10 am

I don't think the security is going to be a massive challenge with this option. All the required technologies to make it bulletproof exist today, and it will be as safe as opening a local FreeCAD file.

By the way, currently and during the next couple of months (year), the feature is activated only at compile time with preprocessor switch. Our default built doesn't integrate it except some that I generate.

I will work on option 1 first. It is easy to do.

The addition to freeCAD core is not that big, by the way and will be "easy" after option 1 will be done. The thing is that i will have to rewrite some code, as the data structure will be addressed differently
realthunder
Posts: 1715
Joined: Tue Jan 03, 2017 10:55 am

Re: Hitting some limitations with Cloud workbench / App::Link

Postby realthunder » Wed Jul 08, 2020 12:31 am

App::Link does not handle files directly. It just saves the file name as a simple string and pass it to the core for loading. Although it does have a few use of QFilePath/QDir to manipulate the file path, but it is easy to make it work for URL as well. So if you are going to do this, most of the changes required does not really affect Link.

That being said. I really do not recommend adding cloud and networking stuff directly into the core. Like I have suggested in the beginning when you started this Cloud workbench, it will be much better to handle it in DocumentObserver and do the network syncing asynchronously, and the core should always deal with native files without latency.
Try Assembly3 (latest version 0.11) along with my custom build of FreeCAD at here.
And if you'd like to show your support, you can donate through patreon, liberapay, or paypal
User avatar
vejmarie
Posts: 623
Joined: Mon Jan 04, 2016 4:52 pm
Location: Somewhere between France, USA and Taiwan
Contact:

Re: Hitting some limitations with Cloud workbench / App::Link

Postby vejmarie » Wed Jul 08, 2020 2:23 am

realthunder wrote:
Wed Jul 08, 2020 12:31 am
App::Link does not handle files directly. It just saves the file name as a simple string and pass it to the core for loading. Although it does have a few use of QFilePath/QDir to manipulate the file path, but it is easy to make it work for URL as well. So if you are going to do this, most of the changes required does not really affect Link.
Yes I have made such modifications (with relevant pre processor defines), and it works. I agree it doesn't change anything related to the Link structure.
realthunder wrote:
Wed Jul 08, 2020 12:31 am
That being said. I really do not recommend adding cloud and networking stuff directly into the core. Like I have suggested in the beginning when you started this Cloud workbench, it will be much better to handle it in DocumentObserver and do the network syncing asynchronously, and the core should always deal with native files without latency.
I made an implementation with asynchronous save/restore, and got a couple of users complaining about the asynchronous approach of the "file" operation. Most of them prefer to wait and being sure that everything went smoothly.

I bet one day we will be ending up with such code into core, we are not ready yet, but I bet ;)

Any idea on how to overload the file opening operation for XLink ? Roughly PropertyLinks is used to restore missing Documents when an XLink element is detected into Document.xml. I need to catch up that call before it happens as to ensure that I read it remotely instead of locally when an URI is detected within the filename. I can't do it by calling back the module, so I need to attach the code in some places, but do not find where, and how to be sure that my code is executed instead of yours when such node is detected.

My current strategy due to that sucks. I am reading Document.xml and before restoring it into FreeCAD Document, I am parsing it through a DOM XML parser instead of a SAX2 as to extract all XLink elements, and check if they are soon opened or not, and if they are an URI. if that is not the case, I am reading the associated Document through the Cloud interface. Everything is done into the Cloud module in such case.
realthunder
Posts: 1715
Joined: Tue Jan 03, 2017 10:55 am

Re: Hitting some limitations with Cloud workbench / App::Link

Postby realthunder » Thu Jul 09, 2020 12:02 am

vejmarie wrote:
Wed Jul 08, 2020 2:23 am
Any idea on how to overload the file opening operation for XLink ? Roughly PropertyLinks is used to restore missing Documents when an XLink element is detected into Document.xml. I need to catch up that call before it happens as to ensure that I read it remotely instead of locally when an URI is detected within the filename. I can't do it by calling back the module, so I need to attach the code in some places, but do not find where, and how to be sure that my code is executed instead of yours when such node is detected.
App::Link calls App::Application::addPendingDocument() to queue the file name, which gets processed in App::Application::openDocuments(), which calls App::Application::openDocumentPrivate() to create an empty new document, and calls App::Document::restore() to actually opens a physical file and reads it.
Try Assembly3 (latest version 0.11) along with my custom build of FreeCAD at here.
And if you'd like to show your support, you can donate through patreon, liberapay, or paypal
User avatar
vejmarie
Posts: 623
Joined: Mon Jan 04, 2016 4:52 pm
Location: Somewhere between France, USA and Taiwan
Contact:

Re: Hitting some limitations with Cloud workbench / App::Link

Postby vejmarie » Mon Jul 13, 2020 12:07 am

realthunder wrote:
Thu Jul 09, 2020 12:02 am
vejmarie wrote:
Wed Jul 08, 2020 2:23 am
Any idea on how to overload the file opening operation for XLink ? Roughly PropertyLinks is used to restore missing Documents when an XLink element is detected into Document.xml. I need to catch up that call before it happens as to ensure that I read it remotely instead of locally when an URI is detected within the filename. I can't do it by calling back the module, so I need to attach the code in some places, but do not find where, and how to be sure that my code is executed instead of yours when such node is detected.
App::Link calls App::Application::addPendingDocument() to queue the file name, which gets processed in App::Application::openDocuments(), which calls App::Application::openDocumentPrivate() to create an empty new document, and calls App::Document::restore() to actually opens a physical file and reads it.
I finally created a new Document Signal which is called within LinkXsetValue, the place where potential new Document check is perform when an XLink is detected within Document.xml. I attach the Cloud module as a listener to that new signal, and if the filename is a URL based format, the Cloud module take care about checking if the document exist or not. If not, it does create a new Doc, and load the related content into it before handling over the control back to LinkXsetValue which detect that the Document is open, and do not have to open the Document as it is soon open.

Please not that all of this is activated only if the cloud module is built. (through -DBUILD_CLOUD). The good news is that everything works, and I will have a look now how to create assembly through the ongoing work that you perform, and other community members.