Python Multithreaded with MULTICORE [Solved]

Need help, or want to share a macro? Post here!
User avatar
mnesarco
Posts: 153
Joined: Thu Mar 26, 2020 8:52 pm

Python Multithreaded with MULTICORE [Solved]

Postby mnesarco » Tue May 05, 2020 4:23 am

Hi Friends,

I have created a workbench to design guitars, I faced the problem that python code cannot run on multicore (without multiprocess) and it is a big problem. I have a machine with 8 cores and running long task one by one is a pain in the ass. But I managed to overcome this limitation using QT.

This is the solution i am using, it works very well. Feel free to reuse it if you find it useful:

1. The module is in my repository, it is very very small and completely independent of the rest of the code so you can just copy it into your project (keeping credits and GPL license) and use it.

https://github.com/mnesarco/MarzWorkben ... reading.py

2. Example usage

Code: Select all

import marz_threading as mzt

def taskX(a, b):
	return a+b
	
def taskY(a,b):
	return a*b
	
# Run taskX and taskY in parallel and using cores available
jobs = [
	mzt.Task.execute(taskX, 10, 15), 
	mzt.Task.execute(taskY, 30, 10)
]

# Wait for all results
x, y = mzt.Task.joinAll(jobs)

print( f" sum={x}, mul={y}" )


This is especially useful when you need to create many objects and then combine results into something. As long as each object is independent it can be created in parallel. This solution uses all cores available. If you create more tasks than cores, they will be queued and dispatched as soon as cores are available.
User avatar
Kunda1
Posts: 8026
Joined: Thu Jan 05, 2017 9:03 pm

Re: Python Multithreaded with MULTICORE [Solved]

Postby Kunda1 » Thu May 07, 2020 3:44 am

Nice! Thanks for posting :D
Want to contribute back to FC? Checkout:
#lowhangingfruit | Use the Source, Luke. | How to Help FreeCAD | How to report FC bugs and features
User avatar
freman
Posts: 1187
Joined: Tue Nov 27, 2018 10:30 pm

Re: Python Multithreaded with MULTICORE [Solved]

Postby freman » Fri Jul 17, 2020 6:43 am

Hi mnesarco,

this looks very interesting but the link at the top gets a 404 and I don't see marz_threading.py when walking around that repo.

https://github.com/mnesarco/MarzWorkben ... eecad/marz

Could you update the link please. This could have broader uses.

Thanks.
alexu
Posts: 10
Joined: Tue Nov 26, 2019 2:14 pm

Re: Python Multithreaded with MULTICORE [Solved]

Postby alexu » Fri Jul 17, 2020 12:57 pm

freman wrote:
Fri Jul 17, 2020 6:43 am
this looks very interesting but the link at the top gets a 404 and I don't see marz_threading.py when walking around that repo.
Looking through Commit "Extension layout refactoring" of May 25:
src/marz_threading.py --> freecad/marz/extension/threading.py

https://github.com/mnesarco/MarzWorkben ... reading.py
User avatar
freman
Posts: 1187
Joined: Tue Nov 27, 2018 10:30 pm

Re: Python Multithreaded with MULTICORE [Solved]

Postby freman » Fri Jul 17, 2020 2:40 pm

many thanks alexu.
Heinzah
Posts: 10
Joined: Tue Jul 14, 2020 1:14 pm

Re: Python Multithreaded with MULTICORE [Solved]

Postby Heinzah » Fri Jul 17, 2020 2:52 pm

mnesarco wrote:
Tue May 05, 2020 4:23 am
Hi Friends,

I have created a workbench to design guitars, I faced the problem that python code cannot run on multicore (without multiprocess) and it is a big problem. I have a machine with 8 cores and running long task one by one is a pain in the ass. But I managed to overcome this limitation using QT.

This is the solution i am using, it works very well. Feel free to reuse it if you find it useful:

1. The module is in my repository, it is very very small and completely independent of the rest of the code so you can just copy it into your project (keeping credits and GPL license) and use it.

https://github.com/mnesarco/MarzWorkben ... reading.py

2. Example usage

Code: Select all

import marz_threading as mzt

def taskX(a, b):
	return a+b
	
def taskY(a,b):
	return a*b
	
# Run taskX and taskY in parallel and using cores available
jobs = [
	mzt.Task.execute(taskX, 10, 15), 
	mzt.Task.execute(taskY, 30, 10)
]

# Wait for all results
x, y = mzt.Task.joinAll(jobs)

print( f" sum={x}, mul={y}" )


This is especially useful when you need to create many objects and then combine results into something. As long as each object is independent it can be created in parallel. This solution uses all cores available. If you create more tasks than cores, they will be queued and dispatched as soon as cores are available.
Thank you for sharing. There is a strong likelihood I will use this in the future.
alexu wrote:
Fri Jul 17, 2020 12:57 pm
freman wrote:
Fri Jul 17, 2020 6:43 am
this looks very interesting but the link at the top gets a 404 and I don't see marz_threading.py when walking around that repo.
Looking through Commit "Extension layout refactoring" of May 25:
src/marz_threading.py --> freecad/marz/extension/threading.py

https://github.com/mnesarco/MarzWorkben ... reading.py
Thank you alexu.
User avatar
mnesarco
Posts: 153
Joined: Thu Mar 26, 2020 8:52 pm

Re: Python Multithreaded with MULTICORE [Solved]

Postby mnesarco » Fri Jul 17, 2020 6:19 pm

etrombly
Posts: 135
Joined: Thu Dec 05, 2019 6:50 pm

Re: Python Multithreaded with MULTICORE [Solved]

Postby etrombly » Sat Jul 18, 2020 12:53 am

I was just talking about using this here https://forum.freecadweb.org/viewtopic.php?f=15&t=48376 . Were you planning on doing a PR to get it included in FreeCAD itself, rather than just your workbench?
User avatar
mnesarco
Posts: 153
Joined: Thu Mar 26, 2020 8:52 pm

Re: Python Multithreaded with MULTICORE [Solved]

Postby mnesarco » Sat Jul 18, 2020 1:46 am

etrombly wrote:
Sat Jul 18, 2020 12:53 am
I was just talking about using this here https://forum.freecadweb.org/viewtopic.php?f=15&t=48376 . Were you planning on doing a PR to get it included in FreeCAD itself, rather than just your workbench?
Hi etrombly, have you read the code? it is very small (about 100 lines of code). You can copy the threading.py file and use it in your project it is completely standalone.

About a PR to FreeCAD, it is not clear where to put the file. I am not very familiar with internal FreeCAD code structure. But I am working on an Extension framework for freecad with the aim to standardize some things, maybe it will be a good place to put it.
etrombly
Posts: 135
Joined: Thu Dec 05, 2019 6:50 pm

Re: Python Multithreaded with MULTICORE [Solved]

Postby etrombly » Sat Jul 18, 2020 2:07 am

My code is in the Path workbench, so it would need to be included there at least. Testing it out now to see if it's going to work for our needs. Due to the GIL the code actually runs slower in parallel using the threadpool. I'm mostly looking now to see if I can use it to make long running tasks cancelable.