Porting to python3

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
User avatar
yorik
Site Admin
Posts: 8684
Joined: Tue Feb 17, 2009 9:16 pm
Location: São Paulo, Brazil
Contact:

Porting to python3

Postby yorik » Sat Sep 19, 2015 10:14 pm

I am finally undertaking the porting to python3...

This is a pretty hard experience for a C++ newbie like me :) They changed an awful lot of things, and the migration and py2/py3 support is not too well documented, and lots of functionality has been removed without a proper equivalent yet (they are THINKING ABOUT IT)... and our python API is HUGE. Anyway, I'm half-way through the Base module now, it begins to go faster. Fortunately a lot of other projects are doing this too.

I'm using py2/py3 compatible code wherever possible, but even so I'm afraid we'll have a lot of #if PY_VERSION_MAJOR >= 3 everywhere. When I've finished with Base I'll publish a branch so we can start testing, because many changes affect python2 too, probably we'll need some fine-tuning.

Werner, prepare for some headache if you dare to review ;)
drei
Posts: 464
Joined: Sun May 11, 2014 7:47 pm
Location: Mexico
Contact:

Re: Porting to python3

Postby drei » Sat Sep 19, 2015 10:59 pm

I can only wish you good luck on your endeavor yorik.
I'll gladly help as a tester when needed, to the best of my ability.
Need help? Feel free to ask, but please read the guidelines first
User avatar
yorik
Site Admin
Posts: 8684
Joined: Tue Feb 17, 2009 9:16 pm
Location: São Paulo, Brazil
Contact:

Re: Porting to python3

Postby yorik » Sat Sep 19, 2015 11:45 pm

Question for Werner or Jürgen I think: Where do the different src/Base/swigpyrun* come from? Are they autogenerated somehow?
wmayer
Site Admin
Posts: 11355
Joined: Thu Feb 19, 2009 10:32 am

Re: Porting to python3

Postby wmayer » Sun Sep 20, 2015 7:59 am

The bad thing with swig is that for every minor release the internal structure of the Python wrappers changes and you can't mix-up different swig versions. Since over time we had to deal with some bindings (mainly pivy and python-occ) built with different swig versions we had to have built-in all the generated interfaces. At runtime we then read-in a .py file from the external Python binding to know which swig it was built with. Then inside InterpreterSingleton::createSWIGPointerObj/InterpreterSingleton::convertSWIGPointerObj we can use the correct interface.

So, for the Python 3 port this means we can actually remove all the versions from 1.3.25-1.3.40 and just leave the file swigpyrun.cpp because this creates the interface for the currently installed swig version.

In case you need it the command to create the file swigpyrun.h on the command line is:

Code: Select all

swig -python -external-runtime swigpyrun.h


So, the answer to your question is that the swigpyrun-<version>.h were all generated on the command line by hand and only swigpyrun.h gets automatically created at build time if swig is installed.
User avatar
yorik
Site Admin
Posts: 8684
Joined: Tue Feb 17, 2009 9:16 pm
Location: São Paulo, Brazil
Contact:

Re: Porting to python3

Postby yorik » Sun Sep 20, 2015 5:39 pm

Great! I added a cmake condition to not include these files when building with python3, seems to work fine. Thanks!
User avatar
yorik
Site Admin
Posts: 8684
Joined: Tue Feb 17, 2009 9:16 pm
Location: São Paulo, Brazil
Contact:

Re: Porting to python3

Postby yorik » Sun Sep 20, 2015 6:33 pm

Ok I finished src/Base already.... I'll resume the changes here (this is mostly for myself for further reference)

https://github.com/yorikvanhavre/FreeCA ... ccc4d5a770
Don't try to use that branch, because compilation will fail (the module generation templates are already updated, but the implementations of each modules not yet, so the build will fail right after Base). This is mostly for who is curious to look. So far it compiles OK with both py2 and py3.

The major changes:

- If using py3, you need minimum 3.3 (otherwise there are some useful unicode<->UTF8 functions not available yet).
- Python3 doesn't have Int and Long anymore. Everything is Long. This is safe to use in py2 too (in python, it is still called int, mind you, but it's a long).
- By far the most important change: All python strings are now unicode. So all strings-related methods are gone. Fortunately since shoogen and werner already made a great job at spreading the use of UTF8 everywhere, this doesn't give major problems, it even simplifies (no more of that question to use Latin1 or Utf8)
- PyObject->ob_type doesn't exist anymore, use Py_TYPE(PyObject)
- Class definition (PyTypeObject) has changed a lot:
- different way to initialize it (PyVarObject_HEAD_INIT instead of PyObject_HEAD_INIT)
- tp_getattr and tp_setattr slots are being deprecated. Use tp_getattro and tp_setattro instead (they take a PyObject instead of a char*)
- several numeric handlers have been removed: divide (everything is remainder now), long (everything is long now, but the handler to keep is int :? ), coerce, oct and hex.
- Inside python code:
- print "something" becomes print("something"), works in py2 too
- print("something","something else") doesn't work in py2 unless "from __future__ import print_function"
- for key, value in dict.iteritems() becomes for key,value in dict.items(), works in py2 too
- except Error,message becomes except Error(message) works in py2 too
- import submodule becomes from . import submodule
cox
Posts: 916
Joined: Wed Nov 26, 2014 11:37 pm

Re: Porting to python3

Postby cox » Sun Sep 20, 2015 7:32 pm

Are you using any scripts while porting?

I found this https://docs.python.org/2/library/2to3.html

You probably know about it already, but it seams like it wold handle many of the simpler changes you mention in your last post.
Need help? Feel free to ask, but please read the guidelines first
User avatar
yorik
Site Admin
Posts: 8684
Joined: Tue Feb 17, 2009 9:16 pm
Location: São Paulo, Brazil
Contact:

Re: Porting to python3

Postby yorik » Sun Sep 20, 2015 9:15 pm

cox wrote:You probably know about it already, but it seams like it wold handle many of the simpler changes you mention in your last post.

Yes it does, but sometimes it doesn't do what you want... Iuse it to have a quick overview of the changes to do, then I do then by hand one by one...
A very good reference I found is this: python-future.org/compatible_idioms.html
wmayer
Site Admin
Posts: 11355
Joined: Thu Feb 19, 2009 10:32 am

Re: Porting to python3

Postby wmayer » Sun Sep 20, 2015 10:33 pm

Werner, prepare for some headache if you dare to review

I made some comments to your branch. I think the best would be to port only FreeCADBase, FreeCADApp and FreeCADMainCmd for now because then we have a lightweight application to test with.
User avatar
yorik
Site Admin
Posts: 8684
Joined: Tue Feb 17, 2009 9:16 pm
Location: São Paulo, Brazil
Contact:

Re: Porting to python3

Postby yorik » Sun Sep 20, 2015 10:52 pm

Yes that was my plan too. Thanks a lot for reviewing so far!