Scripted Object Serialization: json or pickle?

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
User avatar
gbroques
Posts: 73
Joined: Thu Jan 23, 2020 3:28 am
Location: St. Louis, Missouri

Scripted Object Serialization: json or pickle?

Postby gbroques » Fri Jul 31, 2020 11:51 am

Reading Scripted objects it says FreeCAD uses python's json module for serialization / deserialization of scripted objects, and also alludes to how FreeCAD used to use the pickle module.

There's also discussion of this on Scripted objects saving attributes.

Furthermore, there's discussion of overriding the __getstate__ and __setstate__ methods. However, my understanding is that these methods only apply when using pickle.

See pickle module reference:
https://docs.python.org/3/library/pickl ... __getstate__

There's no mention of __getstate__ and __setstate__ methods in the json module docs:
https://docs.python.org/3/library/json.html

Also, this StackOverflow comment says:
pickle supports state hooks (__getstate__ and companions), but json does not support any such helpful methods.
My questions:
1. As of FreeCAD 19, do we use json, pickle, or both for serialization / deserialization of scripted objects?
2. If we only use json, then is overriding the __getstate__ and __setstate__ methods necessary?
vocx
Posts: 4641
Joined: Thu Oct 18, 2018 9:18 pm

Re: Scripted Object Serialization: json or pickle?

Postby vocx » Fri Jul 31, 2020 2:47 pm

gbroques wrote:
Fri Jul 31, 2020 11:51 am
...
My questions:
1. As of FreeCAD 19, do we use json, pickle, or both for serialization / deserialization of scripted objects?
2. If we only use json, then is overriding the __getstate__ and __setstate__ methods necessary?
I wrote those pages, so I tried to clean up the information as much as possible. My understanding is that __getstate__ and __setstate__ are used just like the Python Pickle module says, but FreeCAD itself doesn't use Pickle. It uses something else, that is, json (maybe?). But this is done in the internal C++ code; you won't find it somewhere that says "import json" for dealing with this.

You'd need to go inside the C++ code that deals with reading and writing the documents to really know what is going on. I'm not entirely sure where this is, but maybe check the src/Base/, and App/Document files.
Always add the important information to your posts if you need help.
To support the documentation effort, and code development, your donation is appreciated: paypal.
User avatar
gbroques
Posts: 73
Joined: Thu Jan 23, 2020 3:28 am
Location: St. Louis, Missouri

Re: Scripted Object Serialization: json or pickle?

Postby gbroques » Sat Aug 01, 2020 12:16 am

vocx wrote:
Fri Jul 31, 2020 2:47 pm
You'd need to go inside the C++ code that deals with reading and writing the documents to really know what is going on. I'm not entirely sure where this is, but maybe check the src/Base/, and App/Document files.
I found the following code for serialization:
https://github.com/FreeCAD/FreeCAD/blob ... pp#L88-L95

Code: Select all

        Py::Module pickle(PyImport_ImportModule("json"),true);
        if (pickle.isNull())
            throw Py::Exception();
        Py::Callable method(pickle.getAttr(std::string("dumps")));
        Py::Object dump;
        if (this->object.hasAttr("__getstate__")) {
            Py::Tuple args;
            Py::Callable state(this->object.getAttr("__getstate__"));
            dump = state.apply(args);
        }
        else if (this->object.hasAttr("__dict__")) {
            dump = this->object.getAttr("__dict__");
        }
        else {
            dump = this->object;
        }
and the following code for de-serialization:
https://github.com/FreeCAD/FreeCAD/blob ... #L127-L146

Code: Select all

        Py::Module pickle(PyImport_ImportModule("json"),true);
        if (pickle.isNull())
            throw Py::Exception();
        Py::Callable method(pickle.getAttr(std::string("loads")));
        Py::Tuple args(1);
        args.setItem(0, Py::String(repr));
        Py::Object res = method.apply(args);

        if (this->object.hasAttr("__setstate__")) {
            Py::Tuple args(1);
            args.setItem(0, res);
            Py::Callable state(this->object.getAttr("__setstate__"));
            state.apply(args);
        }
        else if (this->object.hasAttr("__dict__")) {
            this->object.setAttr("__dict__", res);
        }
        else {
            this->object = res;
        }
And there's other references to "pickle" and "json" throughout that file.

Can you help me interpret it? Is this some kind of hybrid-approach?
vocx
Posts: 4641
Joined: Thu Oct 18, 2018 9:18 pm

Re: Scripted Object Serialization: json or pickle?

Postby vocx » Sat Aug 01, 2020 12:46 am

gbroques wrote:
Sat Aug 01, 2020 12:16 am

Code: Select all

        Py::Module pickle(PyImport_ImportModule("json"),true);
...
Can you help me interpret it? Is this some kind of hybrid-approach?
I'm not sure, but maybe from this line of code what I can see is that they define the "pickle" module but instead of it being the actual "pickle" module, it is the "json" module. To me this indicates that in the past they used the "pickle" module, but then made the decision to replace it with the "json" module. To avoid re-writing everything, they just made the "pickle" module internally the same as the "json" module. So it uses the "json" module but with the older name (maybe).

You should move this thread to the Developer's section.
Always add the important information to your posts if you need help.
To support the documentation effort, and code development, your donation is appreciated: paypal.
User avatar
gbroques
Posts: 73
Joined: Thu Jan 23, 2020 3:28 am
Location: St. Louis, Missouri

Re: Scripted Object Serialization: json or pickle?

Postby gbroques » Sat Aug 01, 2020 1:19 am

vocx wrote:
Sat Aug 01, 2020 12:46 am
I'm not sure, but maybe from this line of code what I can see is that they define the "pickle" module but instead of it being the actual "pickle" module, it is the "json" module ... (maybe).
Ok. Maybe it's a pickle-like interface to the underlying Python json module, and was done that way for backwards compatibility purposes.

vocx wrote:
Sat Aug 01, 2020 12:46 am
You should move this thread to the Developer's section.
Sure. I don't have permissions to do that, but happy if someone moves it who does.

Hopefully that can get some authoritative answer on this question.
User avatar
gbroques
Posts: 73
Joined: Thu Jan 23, 2020 3:28 am
Location: St. Louis, Missouri

Re: Scripted Object Serialization: json or pickle?

Postby gbroques » Sat Aug 01, 2020 1:27 am

Based on C++ code posted earlier in this thread, it would appear you can override the __dict__ method as an alternative to __getstate__ and __setstate__ methods.

Although, how exactly to do this in Python, and what the differences are between using __dict__ as opposed to __getstate__ and __setstate__, is yet to be seen or documented clearly.
vocx
Posts: 4641
Joined: Thu Oct 18, 2018 9:18 pm

Re: Scripted Object Serialization: json or pickle?

Postby vocx » Sat Aug 01, 2020 2:50 pm

gbroques wrote:
Sat Aug 01, 2020 1:27 am
...you can override the __dict__ method as an alternative to __getstate__ and __setstate__ methods....
Yes, by default, if you don't implement __getstate__ and __setstate__ it serializes __dict__; so you could just modify this instead.

I think this is mentioned in Scripted objects saving attributes, or do you mean something more?
Always add the important information to your posts if you need help.
To support the documentation effort, and code development, your donation is appreciated: paypal.
User avatar
gbroques
Posts: 73
Joined: Thu Jan 23, 2020 3:28 am
Location: St. Louis, Missouri

Re: Scripted Object Serialization: json or pickle?

Postby gbroques » Sat Aug 01, 2020 9:38 pm

vocx wrote:
Sat Aug 01, 2020 2:50 pm
I think this is mentioned in Scripted objects saving attributes, or do you mean something more?
Very good! I missed this the first time browsing that page.

Thanks for documenting it and bringing it to my attention.
User avatar
Kunda1
Posts: 7679
Joined: Thu Jan 05, 2017 9:03 pm

Re: Scripted Object Serialization: json or pickle?

Postby Kunda1 » Sun Aug 02, 2020 1:30 am

Mod edit: Moved thread to Developers subforum
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features