Scripted Object Serialization: json or pickle?

[gbroques]

Reading Scripted objects it says FreeCAD uses python's json module for serialization / deserialization of scripted objects, and also alludes to how FreeCAD used to use the pickle module.

There's also discussion of this on Scripted objects saving attributes.

Furthermore, there's discussion of overriding the __getstate__ and __setstate__ methods. However, my understanding is that these methods only apply when using pickle.

See pickle module reference:
https://docs.python.org/3/library/pickl ... __getstate__

There's no mention of __getstate__ and __setstate__ methods in the json module docs:
https://docs.python.org/3/library/json.html

Also, this StackOverflow comment says:

pickle supports state hooks (__getstate__ and companions), but json does not support any such helpful methods.

My questions:
1. As of FreeCAD 19, do we use json, pickle, or both for serialization / deserialization of scripted objects?
2. If we only use json, then is overriding the __getstate__ and __setstate__ methods necessary?

[vocx]

...
My questions:
1. As of FreeCAD 19, do we use json, pickle, or both for serialization / deserialization of scripted objects?
2. If we only use json, then is overriding the __getstate__ and __setstate__ methods necessary?

I wrote those pages, so I tried to clean up the information as much as possible. My understanding is that __getstate__ and __setstate__ are used just like the Python Pickle module says, but FreeCAD itself doesn't use Pickle. It uses something else, that is, json (maybe?). But this is done in the internal C++ code; you won't find it somewhere that says "import json" for dealing with this.

You'd need to go inside the C++ code that deals with reading and writing the documents to really know what is going on. I'm not entirely sure where this is, but maybe check the src/Base/, and App/Document files.

[gbroques]

You'd need to go inside the C++ code that deals with reading and writing the documents to really know what is going on. I'm not entirely sure where this is, but maybe check the src/Base/, and App/Document files.

I found the following code for serialization:
https://github.com/FreeCAD/FreeCAD/blob ... pp#L88-L95

Code: Select all

        Py::Module pickle(PyImport_ImportModule("json"),true);
        if (pickle.isNull())
            throw Py::Exception();
        Py::Callable method(pickle.getAttr(std::string("dumps")));
        Py::Object dump;
        if (this->object.hasAttr("__getstate__")) {
            Py::Tuple args;
            Py::Callable state(this->object.getAttr("__getstate__"));
            dump = state.apply(args);
        }
        else if (this->object.hasAttr("__dict__")) {
            dump = this->object.getAttr("__dict__");
        }
        else {
            dump = this->object;
        }

and the following code for de-serialization:
https://github.com/FreeCAD/FreeCAD/blob ... #L127-L146

Code: Select all

        Py::Module pickle(PyImport_ImportModule("json"),true);
        if (pickle.isNull())
            throw Py::Exception();
        Py::Callable method(pickle.getAttr(std::string("loads")));
        Py::Tuple args(1);
        args.setItem(0, Py::String(repr));
        Py::Object res = method.apply(args);

        if (this->object.hasAttr("__setstate__")) {
            Py::Tuple args(1);
            args.setItem(0, res);
            Py::Callable state(this->object.getAttr("__setstate__"));
            state.apply(args);
        }
        else if (this->object.hasAttr("__dict__")) {
            this->object.setAttr("__dict__", res);
        }
        else {
            this->object = res;
        }

And there's other references to "pickle" and "json" throughout that file.

Can you help me interpret it? Is this some kind of hybrid-approach?

[vocx]

Code: Select all

        Py::Module pickle(PyImport_ImportModule("json"),true);

...
Can you help me interpret it? Is this some kind of hybrid-approach?

I'm not sure, but maybe from this line of code what I can see is that they define the "pickle" module but instead of it being the actual "pickle" module, it is the "json" module. To me this indicates that in the past they used the "pickle" module, but then made the decision to replace it with the "json" module. To avoid re-writing everything, they just made the "pickle" module internally the same as the "json" module. So it uses the "json" module but with the older name (maybe).

You should move this thread to the Developer's section.

[gbroques]

I'm not sure, but maybe from this line of code what I can see is that they define the "pickle" module but instead of it being the actual "pickle" module, it is the "json" module ... (maybe).

Ok. Maybe it's a pickle-like interface to the underlying Python json module, and was done that way for backwards compatibility purposes.

You should move this thread to the Developer's section.

Sure. I don't have permissions to do that, but happy if someone moves it who does.

Hopefully that can get some authoritative answer on this question.

[gbroques]

Based on C++ code posted earlier in this thread, it would appear you can override the __dict__ method as an alternative to __getstate__ and __setstate__ methods.

Although, how exactly to do this in Python, and what the differences are between using __dict__ as opposed to __getstate__ and __setstate__, is yet to be seen or documented clearly.

[vocx]

...you can override the __dict__ method as an alternative to __getstate__ and __setstate__ methods....

Yes, by default, if you don't implement __getstate__ and __setstate__ it serializes __dict__; so you could just modify this instead.

I think this is mentioned in Scripted objects saving attributes, or do you mean something more?

[gbroques]

I think this is mentioned in Scripted objects saving attributes, or do you mean something more?

Very good! I missed this the first time browsing that page.

Thanks for documenting it and bringing it to my attention.

[Kunda1]

Mod edit: Moved thread to Developers subforum