Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increases
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Be nice to others! Respect the FreeCAD code of conduct!
Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa
This is where postprocessors are dynamically imported: https://github.com/FreeCAD/FreeCAD/blob ... sor.py#L49
I think replacing this with a more conventional setup that imports each postprocessor module only once, without the exec shenanigans, will likely avoid the memory bloat issue.
I think replacing this with a more conventional setup that imports each postprocessor module only once, without the exec shenanigans, will likely avoid the memory bloat issue.
-
- Posts: 52
- Joined: Sat Jul 25, 2020 2:36 pm
Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa
Did some further investigation.
Brought down the PostProcessing routine grbl_post.py back to it's bare minumum.
See attached grblmem_post.py
Not much left
Added the Garbage Collect and tracemalloc tools to understand what is going on.
snippet of the basis code :
In the first 7 runs i process only 1 Path Command from the Total Path Commands of 226402
In the last run I process all 226402 Path commands.
Each Path command now takes about 0.7 KiB in the code : commandparm = cmd.Parameter
This results in a Memory use which does not seem to be released of 226402 * 0.7 KiB = 158481,4 KiB per Full Run
which is different from the resulting Memory use at the end of a full run which is Total allocated size: 71371.8 KiB
Still a lot of memory which is not released.
The code from the other lines not shown in the tracemalloc Top 10 list does not increase in Memory
Why is the memory used by code : commandparm = cmd.Parameters not released ???
commands = the number of Path commands in my model generated by Adaptive Processing
Brought down the PostProcessing routine grbl_post.py back to it's bare minumum.
See attached grblmem_post.py
Not much left
Added the Garbage Collect and tracemalloc tools to understand what is going on.
snippet of the basis code :
Code: Select all
for cmd in obj.Path.Commands: # for my model : 226402 Count using Adaptive Processing greatly increases the Path Obj.
i += 1
command = cmd.Name # example : c.Name = 'G1'
commandparm = cmd.Parameters # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
In the last run I process all 226402 Path commands.
Each Path command now takes about 0.7 KiB in the code : commandparm = cmd.Parameter
This results in a Memory use which does not seem to be released of 226402 * 0.7 KiB = 158481,4 KiB per Full Run
which is different from the resulting Memory use at the end of a full run which is Total allocated size: 71371.8 KiB
Still a lot of memory which is not released.
The code from the other lines not shown in the tracemalloc Top 10 list does not increase in Memory
Why is the memory used by code : commandparm = cmd.Parameters not released ???
commands = the number of Path commands in my model generated by Adaptive Processing
Code: Select all
The output of the tracemalloc over several runs [code]FreeCAD 0.19, Libs: 0.19R21975 (Git)
© Juergen Riegel, Werner Mayer, Yorik van Havre and others 2001-2020
FreeCAD is free and open-source software licensed under the terms of LGPL2+ license.
FreeCAD wouldn't be possible without FreeCAD community.
##### #### ### ####
# # # # # #
# ## #### #### # # # # #
#### # # # # # # # ##### # #
# # #### #### # # # # #
# # # # # # # # # ## ## ##
# # #### #### ### # # #### ## ## ##
connect failed: No such file or directory
Path workbench activated
PathPost.DEBUG: about to postprocess job: Job
PathPost.DEBUG: Ordering by Fixture
PathPost.DEBUG: Appending TC: T1__Default_tool
post: grblmem(/home/ubuntu/FLsun-Prusa-PartsUpgrade/files/MemTest.gcode, )
grblmem_post: GCode postprocessor loaded.
grblmem_post: GCode postprocessor loaded.
XXXXXXXXXXXXXXXX 1st run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Number of commands : 1
Top 10 lines
#10: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 0.7 KiB
commandparm = cmd.Parameters # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
58 other: 17.7 KiB
Total allocated size: 2373.1 KiB
XXXXXXXXXXXXXXXX 2nd run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Number of commands : 1
Top 10 lines
#4: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 1.4 KiB
commandparm = cmd.Parameters # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
95 other: 25.4 KiB
Total allocated size: 2435.2 KiB
=========================================================================
XXXXXXXXXXXXXXXX 3rd run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Number of commands : 1
Top 10 lines
#4: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 2.1 KiB
commandparm = cmd.Parameters # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
95 other: 26.1 KiB
Total allocated size: 2436.9 KiB
=========================================================================
XXXXXXXXXXXXXXXX 4th run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Number of commands : 1
Top 10 lines
#4: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 2.8 KiB
commandparm = cmd.Parameters # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
95 other: 26.3 KiB
Total allocated size: 2437.8 KiB
=========================================================================
XXXXXXXXXXXXXXXX 5th run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Number of commands : 1
Top 10 lines
#4: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 3.5 KiB
commandparm = cmd.Parameters # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
99 other: 27.5 KiB
Total allocated size: 2439.6 KiB
=========================================================================
XXXXXXXXXXXXXXXX 6th run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Number of commands : 1
Top 10 lines
#4: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 4.2 KiB
commandparm = cmd.Parameters # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
99 other: 28.0 KiB
Total allocated size: 2440.5 KiB
=========================================================================
XXXXXXXXXXXXXXXX 7th run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Number of commands : 1
Top 10 lines
#3: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 4.9 KiB
commandparm = cmd.Parameters # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
99 other: 28.0 KiB
Total allocated size: 2441.2 KiB
=========================================================================
XXXXXXXXXXXXXXXX 226402 run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Number of commands : 226402
Top 10 lines
#1: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 68831.9 KiB
commandparm = cmd.Parameters # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
286 other: 109.9 KiB
Total allocated size: 71371.8 KiB
=========================================================================
- Attachments
-
- grblmem_post.py
- (5.35 KiB) Downloaded 42 times
-
- Posts: 52
- Joined: Sat Jul 25, 2020 2:36 pm
Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa
Hi @gwicke
I think I tried your suggestion by a mofified version of : PathPost.py and PathPostProcessor.py
together with my grbl version : grblmem_post.py
All my changes have #JCL
in PathPost.py i've added :
In PathPostProcessor.py i've added a print to see if the dynamic load was still called or not :
Unfortunately, No change is Memory consumption
Not sure if I've done this correct as a Non Python expert.
I think I tried your suggestion by a mofified version of : PathPost.py and PathPostProcessor.py
together with my grbl version : grblmem_post.py
All my changes have #JCL
in PathPost.py i've added :
Code: Select all
import sys # JCL
sys.path.insert(0, '/home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post') # JCL
import grblmem_post # JCL
Code: Select all
def updateTooltip(self, item):
if item.text() in self.tooltips.keys():
tooltip = self.tooltips[item.text()]
else:
#processor = PostProcessor.load(item.text()) # JCL
processor = PostProcessor(grblmem_post) # JCL
Code: Select all
if postname and filename:
print("post: %s(%s, %s)" % (postname, filename, postArgs))
processor = PostProcessor(grblmem_post) # JCL
#processor = PostProcessor.load(postname) # JCL
gcode = processor.export(objs, filename, postArgs)
return (False, gcode)
Code: Select all
@classmethod
def load(cls, processor):
print("load called -----------------------------------") # JCL
PathLog.track(processor)
Not sure if I've done this correct as a Non Python expert.
- Attachments
-
- PathPost.py
- (16.6 KiB) Downloaded 38 times
-
- PathPostProcessor.py
- (4.46 KiB) Downloaded 46 times
-
- grblmem_post.py
- (5.35 KiB) Downloaded 42 times
-
- Posts: 52
- Joined: Sat Jul 25, 2020 2:36 pm
Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa
When removing the : commandparm = cmd.Parameters from the stripped grblmem_post.py,
the memory is not increasing anymore
When adding a second var with points to cmd.Parameters the increase of the memory doubles.
So any reference to cmd.Parameters adds to the problem.
Could it be the floats in the X, Y, Z, F parameters. ??
(cmd.Parameters = 'F': 1.666667, 'X': 22.684211, 'Y': 43.318421)
cmd.Name only has strings in it.
even a makes it worse
the memory is not increasing anymore
Code: Select all
for cmd in obj.Path.Commands: # for my model : 226402 Count using Adaptive Processing greatly increases the Path Obj.
i += 1
command = cmd.Name # example : c.Name = 'G1'
Code: Select all
for cmd in obj.Path.Commands: # for my model : 226402 Count using Adaptive Processing greatly increases the Path Obj.
i += 1
command = cmd.Name
commandparm = cmd.Parameters
commandparm1 = cmd.Parameters
Could it be the floats in the X, Y, Z, F parameters. ??
(cmd.Parameters = 'F': 1.666667, 'X': 22.684211, 'Y': 43.318421)
cmd.Name only has strings in it.
even a
Code: Select all
if cmd.Parameters
Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa
I know almost nothing about the internals of Python, but I believe when referencing an item like cmd.Parameters that every reference contained in cmd.Parameters is imported, not just the raw numbers that might result from float or other operations.
In other words, if the cmd.Parameters are not pure strings and numbers there is a lot of other stuff brought in as reference material. Totally useless here, but still exists. This would happen in lots of places, but usually it is harmless.
Gene
In other words, if the cmd.Parameters are not pure strings and numbers there is a lot of other stuff brought in as reference material. Totally useless here, but still exists. This would happen in lots of places, but usually it is harmless.
Gene
Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa
because all variables in python are objects, these assignments call the constructor and make a copy. But I don't see why they are not getting dereferenced to zero and being cleaned by GC once the function is finished. I clearly do not fully understand the full extent of the insanity in the way python was written.
x+=1 :
makes a new immutable object with the value x+1
creates a new object
copies type of x to new object
copies new immutable to value of x
increases ref count of immutable
increases ref count of new x obj
decreases ref count of old x obj
does a double back flip
does a knee slide towards the oppositions goalkeeper with both hands thrust into the air making a victory sign
this is followed by a lap of honour and a teeshirt swap
in most sensible languages x+=1 or x++ increments a memory location in RAM, most processors have a single CPU instruction to do that. It will be executed in one or two clock cycles. Depending on the context it will probably compile down to incrementing an internal CPU register.
x+=1 :
makes a new immutable object with the value x+1
creates a new object
copies type of x to new object
copies new immutable to value of x
increases ref count of immutable
increases ref count of new x obj
decreases ref count of old x obj
does a double back flip
does a knee slide towards the oppositions goalkeeper with both hands thrust into the air making a victory sign
this is followed by a lap of honour and a teeshirt swap
in most sensible languages x+=1 or x++ increments a memory location in RAM, most processors have a single CPU instruction to do that. It will be executed in one or two clock cycles. Depending on the context it will probably compile down to incrementing an internal CPU register.
-
- Posts: 52
- Joined: Sat Jul 25, 2020 2:36 pm
Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa
Any idea why the reference to cmd.Name is not increasing the memory and the reference to cmd.Parameters is ?
What is the difference , Only the content ?
I'm lost...
What is the difference , Only the content ?
I'm lost...
Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa
Without looking at the code in detail I would guess that command.Name is string object of very small size which already exists as an immutable. Thus it is just the ref count which gets incremented.
-
- Posts: 52
- Joined: Sat Jul 25, 2020 2:36 pm
Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa
I've decided not to spent anymore time trying to find the root cause.
Took another approach and made a workaround.
Can you please test this draft version of : grbltst_post.py
As far as I can tell, it's not eating up the memory and is faster too.
Next steps are regression testing and making sure it's doing the same as grbl_post.py
Took another approach and made a workaround.
Can you please test this draft version of : grbltst_post.py
As far as I can tell, it's not eating up the memory and is faster too.
Next steps are regression testing and making sure it's doing the same as grbl_post.py
- Attachments
-
- grbltst_post.py
- (23.63 KiB) Downloaded 43 times
Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa
https://bugs.python.org/issue1635741
Py_Finalize() doesn't clear all Python objects at exit
Created on 2007-01-15 10:26 by kylotan, last changed 2020-07-24 13:13 by shihai1991.
It should be remembered that the python is a constrictor, it kills its prey by suffocation.
Py_Finalize() doesn't clear all Python objects at exit
Created on 2007-01-15 10:26 by kylotan, last changed 2020-07-24 13:13 by shihai1991.
It should be remembered that the python is a constrictor, it kills its prey by suffocation.