Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increases

Here's the place for discussion related to CAM/CNC and the development of the Path module.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
gwicke
Posts: 10
Joined: Sun Oct 13, 2019 3:16 am
Location: California

Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa

Post by gwicke »

This is where postprocessors are dynamically imported: https://github.com/FreeCAD/FreeCAD/blob ... sor.py#L49

I think replacing this with a more conventional setup that imports each postprocessor module only once, without the exec shenanigans, will likely avoid the memory bloat issue.
janc.linders
Posts: 52
Joined: Sat Jul 25, 2020 2:36 pm

Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa

Post by janc.linders »

Did some further investigation.
Brought down the PostProcessing routine grbl_post.py back to it's bare minumum.
See attached grblmem_post.py
Not much left :-)

Added the Garbage Collect and tracemalloc tools to understand what is going on.

snippet of the basis code :

Code: Select all

    for cmd in obj.Path.Commands:      # for my model : 226402  Count using Adaptive Processing greatly increases the Path Obj.
      i += 1
      command = cmd.Name                # example : c.Name = 'G1'
      commandparm = cmd.Parameters      # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
  
In the first 7 runs i process only 1 Path Command from the Total Path Commands of 226402
In the last run I process all 226402 Path commands.

Each Path command now takes about 0.7 KiB in the code : commandparm = cmd.Parameter
This results in a Memory use which does not seem to be released of 226402 * 0.7 KiB = 158481,4 KiB per Full Run
which is different from the resulting Memory use at the end of a full run which is Total allocated size: 71371.8 KiB
Still a lot of memory which is not released.

The code from the other lines not shown in the tracemalloc Top 10 list does not increase in Memory
Why is the memory used by code : commandparm = cmd.Parameters not released ???

commands = the number of Path commands in my model generated by Adaptive Processing

Code: Select all

The output of the tracemalloc over several runs [code]FreeCAD 0.19, Libs: 0.19R21975 (Git)
© Juergen Riegel, Werner Mayer, Yorik van Havre and others 2001-2020
FreeCAD is free and open-source software licensed under the terms of LGPL2+ license.
FreeCAD wouldn't be possible without FreeCAD community.
  #####                 ####  ###   ####  
  #                    #      # #   #   # 
  #     ##  #### ####  #     #   #  #   # 
  ####  # # #  # #  #  #     #####  #   # 
  #     #   #### ####  #    #     # #   # 
  #     #   #    #     #    #     # #   #  ##  ##  ##
  #     #   #### ####   ### #     # ####   ##  ##  ##

connect failed: No such file or directory
Path workbench activated
PathPost.DEBUG: about to postprocess job: Job
PathPost.DEBUG: Ordering by Fixture
PathPost.DEBUG: Appending TC: T1__Default_tool
post: grblmem(/home/ubuntu/FLsun-Prusa-PartsUpgrade/files/MemTest.gcode, )
grblmem_post: GCode postprocessor loaded.
grblmem_post: GCode postprocessor loaded.

XXXXXXXXXXXXXXXX 1st run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Number of commands : 1
Top 10 lines
#10: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 0.7 KiB
    commandparm = cmd.Parameters      # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
58 other: 17.7 KiB
Total allocated size: 2373.1 KiB

XXXXXXXXXXXXXXXX 2nd run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                                                                                                                                             
Number of commands : 1
Top 10 lines
#4: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 1.4 KiB
    commandparm = cmd.Parameters      # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
95 other: 25.4 KiB
Total allocated size: 2435.2 KiB
=========================================================================

XXXXXXXXXXXXXXXX 3rd run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Number of commands : 1
Top 10 lines
#4: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 2.1 KiB
    commandparm = cmd.Parameters      # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
95 other: 26.1 KiB
Total allocated size: 2436.9 KiB
=========================================================================

XXXXXXXXXXXXXXXX 4th run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Number of commands : 1
Top 10 lines
#4: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 2.8 KiB
    commandparm = cmd.Parameters      # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
95 other: 26.3 KiB
Total allocated size: 2437.8 KiB
=========================================================================

XXXXXXXXXXXXXXXX 5th run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Number of commands : 1
Top 10 lines
#4: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 3.5 KiB
    commandparm = cmd.Parameters      # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
99 other: 27.5 KiB
Total allocated size: 2439.6 KiB
=========================================================================

XXXXXXXXXXXXXXXX 6th run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Number of commands : 1
Top 10 lines
#4: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 4.2 KiB
    commandparm = cmd.Parameters      # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
99 other: 28.0 KiB
Total allocated size: 2440.5 KiB
=========================================================================
                            
XXXXXXXXXXXXXXXX 7th run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                           
Number of commands : 1
Top 10 lines
#3: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 4.9 KiB
    commandparm = cmd.Parameters      # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}

99 other: 28.0 KiB
Total allocated size: 2441.2 KiB
=========================================================================

XXXXXXXXXXXXXXXX 226402 run XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Number of commands : 226402
Top 10 lines
#1: /home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post/grblmem_post.py:79: 68831.9 KiB
    commandparm = cmd.Parameters      # example : c.Parameters = {'F': 1.666667, 'X': 22.684211, 'Y': 43.318421}
                           
286 other: 109.9 KiB
Total allocated size: 71371.8 KiB
=========================================================================

Attachments
grblmem_post.py
(5.35 KiB) Downloaded 42 times
janc.linders
Posts: 52
Joined: Sat Jul 25, 2020 2:36 pm

Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa

Post by janc.linders »

Hi @gwicke

I think I tried your suggestion by a mofified version of : PathPost.py and PathPostProcessor.py
together with my grbl version : grblmem_post.py
All my changes have #JCL

in PathPost.py i've added :

Code: Select all

import sys  # JCL
sys.path.insert(0, '/home/ubuntu/FreeCAD/freecad-build/Mod/Path/PathScripts/post')   # JCL
import grblmem_post   # JCL

Code: Select all

    def updateTooltip(self, item):
        if item.text() in self.tooltips.keys():
            tooltip = self.tooltips[item.text()]
        else:
            
            #processor = PostProcessor.load(item.text()) # JCL
            processor = PostProcessor(grblmem_post)   # JCL

Code: Select all

        if postname and filename:
            print("post: %s(%s, %s)" % (postname, filename, postArgs))
            processor = PostProcessor(grblmem_post)   # JCL
            #processor = PostProcessor.load(postname) # JCL
            gcode = processor.export(objs, filename, postArgs)
            return (False, gcode)
In PathPostProcessor.py i've added a print to see if the dynamic load was still called or not :

Code: Select all

@classmethod
    def load(cls, processor):
        print("load called -----------------------------------")  # JCL
        PathLog.track(processor)
Unfortunately, No change is Memory consumption :-(

Not sure if I've done this correct as a Non Python expert.
Attachments
PathPost.py
(16.6 KiB) Downloaded 38 times
PathPostProcessor.py
(4.46 KiB) Downloaded 46 times
grblmem_post.py
(5.35 KiB) Downloaded 42 times
janc.linders
Posts: 52
Joined: Sat Jul 25, 2020 2:36 pm

Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa

Post by janc.linders »

When removing the : commandparm = cmd.Parameters from the stripped grblmem_post.py,
the memory is not increasing anymore

Code: Select all

for cmd in obj.Path.Commands:      # for my model : 226402  Count using Adaptive Processing greatly increases the Path Obj.
      i += 1
      command = cmd.Name                # example : c.Name = 'G1'
  
When adding a second var with points to cmd.Parameters the increase of the memory doubles.

Code: Select all

for cmd in obj.Path.Commands:      # for my model : 226402  Count using Adaptive Processing greatly increases the Path Obj.
      i += 1
      command = cmd.Name                
      commandparm = cmd.Parameters     
      commandparm1 = cmd.Parameters 
      
So any reference to cmd.Parameters adds to the problem.
Could it be the floats in the X, Y, Z, F parameters. ??
(cmd.Parameters = 'F': 1.666667, 'X': 22.684211, 'Y': 43.318421)
cmd.Name only has strings in it.

even a

Code: Select all

if cmd.Parameters
makes it worse
GeneFC
Veteran
Posts: 5373
Joined: Sat Mar 19, 2016 3:36 pm
Location: Punta Gorda, FL

Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa

Post by GeneFC »

I know almost nothing about the internals of Python, but I believe when referencing an item like cmd.Parameters that every reference contained in cmd.Parameters is imported, not just the raw numbers that might result from float or other operations.

In other words, if the cmd.Parameters are not pure strings and numbers there is a lot of other stuff brought in as reference material. Totally useless here, but still exists. This would happen in lots of places, but usually it is harmless.

Gene
User avatar
freman
Veteran
Posts: 2198
Joined: Tue Nov 27, 2018 10:30 pm

Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa

Post by freman »

because all variables in python are objects, these assignments call the constructor and make a copy. But I don't see why they are not getting dereferenced to zero and being cleaned by GC once the function is finished. I clearly do not fully understand the full extent of the insanity in the way python was written.

x+=1 :

makes a new immutable object with the value x+1
creates a new object
copies type of x to new object
copies new immutable to value of x
increases ref count of immutable
increases ref count of new x obj
decreases ref count of old x obj
does a double back flip
does a knee slide towards the oppositions goalkeeper with both hands thrust into the air making a victory sign
this is followed by a lap of honour and a teeshirt swap ;)

in most sensible languages x+=1 or x++ increments a memory location in RAM, most processors have a single CPU instruction to do that. It will be executed in one or two clock cycles. Depending on the context it will probably compile down to incrementing an internal CPU register.
janc.linders
Posts: 52
Joined: Sat Jul 25, 2020 2:36 pm

Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa

Post by janc.linders »

Any idea why the reference to cmd.Name is not increasing the memory and the reference to cmd.Parameters is ?
What is the difference , Only the content ?

I'm lost... :cry:
User avatar
freman
Veteran
Posts: 2198
Joined: Tue Nov 27, 2018 10:30 pm

Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa

Post by freman »

Without looking at the code in detail I would guess that command.Name is string object of very small size which already exists as an immutable. Thus it is just the ref count which gets incremented.
janc.linders
Posts: 52
Joined: Sat Jul 25, 2020 2:36 pm

Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa

Post by janc.linders »

I've decided not to spent anymore time trying to find the root cause.
Took another approach and made a workaround.

Can you please test this draft version of : grbltst_post.py

As far as I can tell, it's not eating up the memory and is faster too.
Next steps are regression testing and making sure it's doing the same as grbl_post.py
Attachments
grbltst_post.py
(23.63 KiB) Downloaded 43 times
User avatar
freman
Veteran
Posts: 2198
Joined: Tue Nov 27, 2018 10:30 pm

Re: Using the PATH WB and Adaptive processing not releasing memory. Repeating the PostProcessing into a GRBL file increa

Post by freman »

https://bugs.python.org/issue1635741
Py_Finalize() doesn't clear all Python objects at exit
Created on 2007-01-15 10:26 by kylotan, last changed 2020-07-24 13:13 by shihai1991.

It should be remembered that the python is a constrictor, it kills its prey by suffocation.
Post Reply