Extending python persistence interface

Merged, abandoned or rejected pull requests are moved here to clear the main Pull Requests forum.
ickby
Posts: 2903
Joined: Wed Oct 05, 2011 7:36 am

Extending python persistence interface

Post by ickby » Wed Oct 17, 2018 6:45 pm

Hello,

with pullrequest 1730 I extend the persistence object python interface. This exposes freecads save/restore functionality on a finer level than only document and also to/from memory instead of file. It now allows to
  • store persistencec information into a bytearray (Persistence object is e.g. document, document object, Mesh, etc.)
  • restore the persistence object from a python buffer object (e.g. bytearray)
  • store/restore persistence data for properties
Open questions:
  • I used boost io streams, but i'm not sure if this is available in the libpack or guranteed to be available with current cmake boost settings?
  • Not all persistence objects iplement save/restore (e.g. TopoShape) but rely on properties for persistence. I thought about moving the implementation from property to object and let property use the object functions, but was not sure if this is too much change. What do you guys think?
  • Not sure about the naming dumpContent/restoreContent. Wanted to make it different to the save methods, and Content was already used in persistence, so this may be an option
  • Cannot test it with python 3 atm

User avatar
DeepSOIC
Posts: 6639
Joined: Fri Aug 29, 2014 12:45 am
Location: Saint-Petersburg, Russia

Re: Extending python persistence interface

Post by DeepSOIC » Wed Oct 17, 2018 6:52 pm

Sounds really interesting, but I don't quite understand.

Does it mean that I can make a py object that will store large amounts of data into FCStd file? such as a picture. Or for example, my Lattice2 objects store arrays of placements encoded into shape, but it will be far more efficient to store it as a separate file zipped into FCStd file. Is this what your interface allows to do?

ickby
Posts: 2903
Joined: Wed Oct 05, 2011 7:36 am

Re: Extending python persistence interface

Post by ickby » Wed Oct 17, 2018 6:58 pm

No, it takes the freecad object and puts its content in binary form into a byte array. With this you have a universal method of putting a document object, a property, a whole document or many other freecad objects into memory and do with it what you want. Example (from the test cases):

Code: Select all

   #test smallest level... property
    self.Doc.Label_1.Vector = (1,2,3)
    dump = self.Doc.Label_1.dumpPropertyContent('Vector', Compression = 9)
    self.Doc.Label_2.restorePropertyContent('Vector', dump)
    
    #next higher: object
    dump = self.Doc.Label_1.dumpContent()
    self.Doc.Label_3.restoreContent(dump)
    
    #highest level: document
    dump = self.Doc.dumpContent(9)
    Doc = FreeCAD.newDocument("DumpTest")
    Doc.restoreContent(dump)

User avatar
DeepSOIC
Posts: 6639
Joined: Fri Aug 29, 2014 12:45 am
Location: Saint-Petersburg, Russia

Re: Extending python persistence interface

Post by DeepSOIC » Wed Oct 17, 2018 7:03 pm

Okay, that's quite interesting too, but I'm not quite sure how it is useful.

ickby
Posts: 2903
Joined: Wed Oct 05, 2011 7:36 am

Re: Extending python persistence interface

Post by ickby » Wed Oct 17, 2018 7:13 pm

DeepSOIC wrote:
Wed Oct 17, 2018 7:03 pm
Okay, that's quite interesting too, but I'm not quite sure how it is useful.
I need it for a personal project :) For FreeCAD users in general most likely not super useful, however, it exposes a given c++ functionality to python, so gives more freedom to the users for little cost.

User avatar
sgrogan
Posts: 5201
Joined: Wed Oct 22, 2014 5:02 pm

Re: Extending python persistence interface

Post by sgrogan » Wed Oct 17, 2018 8:07 pm

ickby wrote:
Wed Oct 17, 2018 6:45 pm
I used boost io streams, but i'm not sure if this is available in the libpack or guranteed to be available with current cmake boost settings?
It's available in both the VS2013 Libpack and Conda based builds on Win. I don't think any special CMake setting is necessary.

ickby
Posts: 2903
Joined: Wed Oct 05, 2011 7:36 am

Re: Extending python persistence interface

Post by ickby » Thu Oct 18, 2018 5:55 am

sgrogan wrote:
Wed Oct 17, 2018 8:07 pm
It's available in both the VS2013 Libpack and Conda based builds on Win. I don't think any special CMake setting is necessary.
Thanks for checking, thats good to know!
Okay, that's quite interesting too, but I'm not quite sure how it is useful.
I realised a use case: saving single objects instead of whole documents. As the binary you get back from the function basically is a zip file you can simply put it into a file as "myobj.zip" (and than even open it with a zip reader) and reopen it with the restoreContent function in annother freecad instance and document (but not with the default FreeCAD open function)

wmayer
Site Admin
Posts: 14426
Joined: Thu Feb 19, 2009 10:32 am

Re: Extending python persistence interface

Post by wmayer » Thu Oct 18, 2018 9:21 am

Some comments:
Inside the UserDocu tags of the XML files you should remove the leading spaces of new lines otherwise the tool tips look quite messy.

The methods Persistence::dumpToPython and Persistence::restoreFromPython are too Python-specific. Wouldn't it be more useful to return/pass a std::string instead of a PyObject* so that these functions can be used inside C++ code, too? The handling of packing/unpacking the string to/from the PyObject can be done in a separate method or moved to the PersistencePy class.
From Python point of view nothing changes but you will gain more flexibility.
Btw, the implementation of Persistence::restoreFromPython has some Python flaws as it messes up the ref'counting of None or doesn't set an exception everywhere when returning with NULL.

I guess that a serialized byte array is not supposed to be read-in by the document's standard read method. So, to avoid any confusion it would be good to replace the Document.xml with e.g. Persistence.xml.

ickby
Posts: 2903
Joined: Wed Oct 05, 2011 7:36 am

Re: Extending python persistence interface

Post by ickby » Thu Oct 18, 2018 10:33 am

Thanks for the feedback!
Inside the UserDocu tags of the XML files you should remove the leading spaces of new lines otherwise the tool tips look quite messy.
I will update this

The methods Persistence::dumpToPython and Persistence::restoreFromPython are too Python-specific. Wouldn't it be more useful to return/pass a std::string instead of a PyObject* so that these functions can be used inside C++ code, too? The handling of packing/unpacking the string to/from the PyObject can be done in a separate method or moved to the PersistencePy class.
It would be possible to pass a stream to and from the functions (I find it important to avoid data copies, hence no strings) But actually the real reading/restoring part is the smallest code portion, most is python specific handling of buffers and objects. As I want to expose this for properties I need the functionality in PersistencePy as well as in PropertyContainerPy. To avoid douplication I moved everything to the persistence functions.
I'm going to chang ethe functions as proposed by you (with streams). Do you have any idea howto avoid the duplication?

Btw, the implementation of Persistence::restoreFromPython has some Python flaws as it messes up the ref'counting of None or doesn't set an exception everywhere when returning with NULL.
Ahh that damit python refcounting! I will check it. Do I need to increase the refcount of the new Bytearray in dumpContent too?

So, to avoid any confusion it would be good to replace the Document.xml with e.g. Persistence.xml.
Good idea!

wmayer
Site Admin
Posts: 14426
Joined: Thu Feb 19, 2009 10:32 am

Re: Extending python persistence interface

Post by wmayer » Thu Oct 18, 2018 12:35 pm

It would be possible to pass a stream to and from the functions (I find it important to avoid data copies, hence no strings) But actually the real reading/restoring part is the smallest code portion, most is python specific handling of buffers and objects. As I want to expose this for properties I need the functionality in PersistencePy as well as in PropertyContainerPy. To avoid douplication I moved everything to the persistence functions.
I'm going to chang ethe functions as proposed by you (with streams). Do you have any idea howto avoid the duplication?
I had the idea with streams too but didn't mention it. So, yes this would be the most flexible way and IMO this also helps to avoid the duplication of memory. Some years ago I implemented the class PyStreambuf that can be assigned to a std::ostream. The class PyStreambuf accepts a file-like object where you can write the data to. All what the Python object must provide is the write() method.
Ahh that damit python refcounting! I will check it. Do I need to increase the refcount of the new Bytearray in dumpContent too?
To avoid to incorrectly use the API I suggest to change the signature to: void Persistence::restoreFromPython(PyObject *buffer)
Then inside the implementation don't set the Python exceptions directly but raise C++ exceptions where needed. The calling instance then should have a try/catch block where it sets the Python exception and returns NULL. If everything went fine Py_Return should be used.

Post Reply