IDFv3 support - upgrade code

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
User avatar
easyw-fc
Posts: 2773
Joined: Thu Jul 09, 2015 9:34 am

Re: IDFv3 support - upgrade code

Postby easyw-fc » Thu Dec 24, 2015 8:02 pm

hi,
@abdullah , @sgrogan

i revised my thought.. I think the part of STEP association is nice and would be maintained!

When I tried it with its demo board the result is cool!
(it is what I do for kicad, but for idf format and it was done already in 2012!)
@sgrogan I would suggest to add to the stock Idf WB also the demo emn, emp files...
I found them in the official repo here
https://github.com/yorikvanhavre/FreeCA ... f/ISOL.emn
https://github.com/yorikvanhavre/FreeCA ... f/ISOL.emp
but for some reason the demo board is not packaged with the Idf workbench... so it is not possible to have an easy way to see the script in action
I attached the two scenarios:
1) Idf stock WB on ISOL.emn demo board
Idf-to-step.png
Idf-to-step.png (127.21 KiB) Viewed 1567 times
2) Idf Importer on ISOL.emn demo board
Idf-Import.png
Idf-Import.png (108.03 KiB) Viewed 1567 times
as you can see in fact Idf stock WB is an IDF board importer and a STEP exporter
the STEP models in Idflib are old, but the chain just work perfectly...

the best option IMO would be to have a merging of the two WB, with a selection dialog to decide if the user wants to load the Idf board or if wants to convert the Idf board to step...
In that way user will be aware of this nice option, which I cannot find in any FC doc...
only digging inside the code, but not easy to test if not learning IDF standard file protocol

please let me know your opinion
Maurice
abdullah
Posts: 3174
Joined: Sun May 04, 2014 3:16 pm

Re: IDFv3 support - upgrade code

Postby abdullah » Tue Jan 05, 2016 4:23 pm

Hi Maurice,

I have been away for a while and I am just taking a look to this thread.
easyw-fc wrote:i revised my thought.. I think the part of STEP association is nice and would be maintained!
Ok. We have to see how to "mix" all the different functionalities. It will take me some more time to reach there.
easyw-fc wrote:@sgrogan I would suggest to add to the stock Idf WB also the demo emn, emp files...
I found them in the official repo here
https://github.com/yorikvanhavre/FreeCA ... f/ISOL.emn
https://github.com/yorikvanhavre/FreeCA ... f/ISOL.emp
but for some reason the demo board is not packaged with the Idf workbench... so it is not possible to have an easy way to see the script in action
One thing that is not the official repo (I think), it is the personal repo (fork) of yorik. Nevertheless, those files are indeed in the official repo:
https://github.com/FreeCAD/FreeCAD/tree ... rc/Mod/Idf

I do not know for sure why they were left out. I guess it was because they are examples and they are not needed for the library functionality. A solution (for ubuntu, other package systems, automatic installers like ME's plugin manager) could be to provide the users with a separate "package" they can install "freecad-extras-idfexamples". Then those two files would be installed (for example) into Mod/Idf.
User avatar
easyw-fc
Posts: 2773
Joined: Thu Jul 09, 2015 9:34 am

Re: IDFv3 support - upgrade code

Postby easyw-fc » Tue Jan 05, 2016 6:13 pm

abdullah wrote:I do not know for sure why they were left out. I guess it was because they are examples and they are not needed for the library functionality.
Hi Abdullah,
Indeed the STEP models of the libs are useless if not linked to the ISOL.emn and ISOL.emp files
the STEP models of the Idflibs are associated to the IDF emp lib models of the ISOL.emn, ISOL.emp files
I think it was just a lapse
You can test the Idf.py routine in action just opening with FC the ISOL.emn (beside ISOL.emp) file and you will see the routine building up the conversion in a similar way in which kicad StepUp does with kicad boards...

I would suggest to add in the stock repo at least these two files, so the standard IDF importing routine can be fully used to better understand its functionality

in fact Idf stock WB is more than an IDF board importer, it is a MCAD converter from IDF to e.g. STEP, to produce detailed board models (substituting STEP models to IDF emp models, as configured/matched in the file footprints_models.csv)
Unfortunately I cannot found any doc of it in FC
as you suggested the right place for them would be (for example) into Mod/Idf

Maurice
abdullah
Posts: 3174
Joined: Sun May 04, 2014 3:16 pm

Re: IDFv3 support - upgrade code

Postby abdullah » Wed Jan 06, 2016 4:32 pm

Ok. I am revisiting this topic.

Because I do not see the whole idea about idf import clear, the first thing I have done is finished packaging your IDF importer and I have installed it in my computer:
https://launchpad.net/~freecad-communit ... /+packages

Only by doing this, when I try to open an emn file, I get this pop-up.
Idf_open_selection.png
Idf_open_selection.png (22.31 KiB) Viewed 1446 times
This allows me to select your importer, the stock importer, or opening it with PCB WB. With the default example board, I get this with the first:
IDFImporter.png
IDFImporter.png (21.67 KiB) Viewed 1446 times
With the second option, I get (note that component that looks like a SMD regulator on the corner of the board out of position, which should have been rendered next to the 603 components near the slot made on the PCB):
IDF-Stock.png
IDF-Stock.png (27.05 KiB) Viewed 1446 times
With the third option, without further configuration (just clicking ok), I get an empty board.

Observation 1: The stock IDF fails for the regulator, I guess it is the conversion from IDF to STP.

Observation 2: If two different behaviours for importing are needed (IDF/STEP), a way can be just providing two importers and letting the user select the one he wishes to. I am not claiming this to be the only or the best option. It is just a built-in possibility.

I have made several tests with my own boards exporting them as IDF using "Kicad 0.201509280901+6225" from Launchpad from:
deb http://ppa.launchpad.net/js-reynaud/ppa-kicad/ubuntu trusty main

None of them has real EMP information, so I only get a board imported (no components). One of them is this simplistic break-out board, which shows fine in KiCAD 3D view:
OwnBoard.png
OwnBoard.png (129.56 KiB) Viewed 1446 times
but when imported, not even the holes on the board are imported, only the perimeter. I have no special interest in this board, but I enclose the EMN, EMP files in case it helps you "improve" then importer, in case you are interested.
24V_non_inverting_pull_up_emn.zip
(928 Bytes) Downloaded 16 times
Observation 3: Sometimes the importer, does not seem to work right for the board and holes in it.

Regarding whether this should be merged or not. I have done some work trying to see what you did in your Github IDFImporter. I created a new branch of the FreeCAD project in my repo and just manually forced your changes on it. This is the result:

https://github.com/abdullahtahiriyo/Fre ... b4c7557cb1

I have just had a quick look to the modifications.

Observation 4: The current contents of the IdfImporter repository substitute parts of the Idf stock importer. For example disabling the IDF to step conversion (model substitution). So it is not directly mergeable.

Observation 5: I understand from the code and your comments that this work has some extra support from IDF v3.

Observation 6: If we decide to "evolve" the stock version, for example to get the IDF v3 support (regardless of the decision to apply the IDF to STEP substitution or not), you should work on a fork of FreeCAD as shown above. Otherwise it is not practical to see what has been added.

Observation 7: Your version of the Importer seems to require QT support (from the import statements). This would make it (I think, not a python guy) unsuitable for a version of FreeCAD running on the console and using the IDF importer. FreeCAD can be run without an UI. If a final version would need QT support, then probably should not be an extension of the stock IDF importer, but a separate product.

IDF/stock STEP Models/EMP examples

Observation 8: IDF is, AFAIK, a CAD format only used for EDA. This means that IDF has meaning within FreeCAD only as far as it allows importation of circuit boards (made with KiCAD, Eagle, or whatever other SW). It does not have any other implications.

Observation 9: If IDF format advances, I think that the IDF support should be updated. So in principle your work has a place in an IDF stock module. Only that the IDF stock module is not just an IDF importer but an IDF importer and STEP model "swapper".

Observation 10: Stock STEP Models are not only old, but it really seems like a very reduced amount of components available (around 30). I am not sure how many practical boards may be opened with them (other than the one in the example).

Observation 11: I still think particular examples should not be packaged together with FreeCAD. But I partly note your point in that "unless usable with other boards", the STEP Models and the CSV file are also rather useless.

Noting that most of the STEP models are rather example-specific with only negligibly usable for other boards. And noting the difficulty in finding an agreement on what to merge and how to modify the stock IDF importer (should we drop the STEP conversion or not, only when models are unknown, selector dialog killing the FreeCAD console mode...). I think it makes sense to have this IDF Importer separately as a plugin. Mostly because it is ONLY an IDF importer.

But please people of FreeCAD interested in EDA. May I have your opinion?
User avatar
easyw-fc
Posts: 2773
Joined: Thu Jul 09, 2015 9:34 am

Re: IDFv3 support - upgrade code

Postby easyw-fc » Wed Jan 06, 2016 9:40 pm

abdullah wrote:Observation 1: The stock IDF fails for the regulator, I guess it is the conversion from IDF to STP.
The stock IDF fails for the regulator because the model SOT23.stp is not fused in a single part...
abdullah wrote:Observation 3: Sometimes the importer, does not seem to work right for the board and holes in it.
the emn file has all the holes with a negative position... then the drill will not positioned at the right place
if you correct/delete the 'minus' in the .DRILLED_HOLES section, the board will be build with holes
is there a different option you used to export to idf on this board from kicad?
could you please post the kicad file, so I could check it... there is something wrong on kicad exporting to IDF
abdullah wrote:Observation 10: Stock STEP Models are not only old, but it really seems like a very reduced amount of components available (around 30). I am not sure how many practical boards may be opened with them (other than the one in the example).
the Idflibs models, footprints_models.csv and ISOL.emn, ISOL.emp are all a part of a working example of the Idf-to-step converter; the ISOl.emn helped me in see the script working building up the board and parts...
I think adding the two files would help people in better try/understanding the Idf stock model swapper
abdullah wrote:But I partly note your point in that "unless usable with other boards", the STEP Models and the CSV file are also rather useless.
in absence of a minimal documentation of Idf stock routines, you can use the ISOL.emn, csv file and step models to better understand how to create your own lib and csv file to create your board with step models
abdullah wrote:I think it makes sense to have this IDF Importer separately as a plugin. Mostly because it is ONLY an IDF importer.
if you prefer to maintain the use with freecad console, I agree with you that two Idf wb would be the right solution (anyway I use qt only for user messages)
thank you for your post :)
Maurice
PS I'm upgrading my kicad StepUp tools macro, to include the ability to load kicad board (.kicad_pcb) and build the MCAD board and parts directly from inside FreeCAD, so IDF option would be just useful for non kicad EDA users :)
abdullah
Posts: 3174
Joined: Sun May 04, 2014 3:16 pm

Re: IDFv3 support - upgrade code

Postby abdullah » Thu Jan 07, 2016 1:05 pm

easyw-fc wrote:

Observation 3: Sometimes the importer, does not seem to work right for the board and holes in it.
the emn file has all the holes with a negative position... then the drill will not positioned at the right place
if you correct/delete the 'minus' in the .DRILLED_HOLES section, the board will be build with holes
is there a different option you used to export to idf on this board from kicad?
could you please post the kicad file, so I could check it... there is something wrong on kicad exporting to IDF
Not really, just hit the IDFv3 export. I just tried opening another board with the same KiCAD version and it exported the holes fine...

Here you have the file:
24V_non_inverting_pull_up.zip
(22.49 KiB) Downloaded 15 times
Do you happen know why I do not have IDF information of components (EMP file) in any of my boards? Is there anything special to do in KiCAD so that this file gets populated with component information?
easyw-fc wrote:the Idflibs models, footprints_models.csv and ISOL.emn, ISOL.emp are all a part of a working example of the Idf-to-step converter; the ISOl.emn helped me in see the script working building up the board and parts...
I think adding the two files would help people in better try/understanding the Idf stock model swapper
I will try to make something about this.
easyw-fc wrote:
if you prefer to maintain the use with freecad console, I agree with you that two Idf wb would be the right solution (anyway I use qt only for user messages)
QT is a factor but only part of the history. Changing something that was designed to operate in a certain way (IDF+STEP swap), even if it is just an example in order to introduce another functionality (IDF only) may require quite some additional energy.

Additionally, as I see it, while the stock Idf importer is quite focused on the swap and how to do it (external csv files). Your IdfImporter module is focused just in being an IDF importer, requiring nothing else than the EMN and EMP files.

Moreover, providing it in your separate GitHub repository, you can make improvements today and tomorrow they will be available to Ubuntu users using the Community repository for tests. No pull requests, no code reviews. Very low development-testing latency.

You may PM MicroElly if you would like to have it included in the plugin manager. This will make it available also for Windows users using the plugin.
easyw-fc wrote:PS I'm upgrading my kicad StepUp tools macro, to include the ability to load kicad board (.kicad_pcb) and build the MCAD board and parts directly from inside FreeCAD, so IDF option would be just useful for non kicad EDA users :)
I will welcome this. I am manually positioning the most space requiring components in my boards manually and it is cumbersome. Let me know when it is ready in order to test it.

Unfortunately I have not managed to have it merged the extra functionality needed for system macros support into FreeCAD master yet. Meaning that even though your macro is indeed packaged in the community extras repository for Ubuntu and daily build (upon change of the source), it won't be available for testing for Ubuntu users until they use a FreeCAD version having that functionality.

Of course it can be installed "as always" directly from GitHub following your instructions as a user macro.

However, it follows "the same" logic, you may PM MicroElly when it is ready so that it is included in the plugin manager. Yorik also had some tool for installing macros recently. It may also be an option for distribution...

Keep the nice work :)
abdullah
User avatar
easyw-fc
Posts: 2773
Joined: Thu Jul 09, 2015 9:34 am

Re: IDFv3 support - upgrade code

Postby easyw-fc » Thu Jan 07, 2016 11:01 pm

abdullah wrote:Here you have the file:
24V_non_inverting_pull_up.zip
I opened the file with my new (not completed) kicad StepUp tools Macro and I found that the edge of the pcb is not closed... There is a small non coincident in the edge points that makes the kicad IDF exporter make a wrong IDF data.
edges-not-closed.png
edges-not-closed.png (93.77 KiB) Viewed 1381 times
if you review your edges than the IDF will import correctly pcb and holes... (PS the kicad idf exporter if found some non coincident edge, will try to correct the path, sometime moving the edge that you can see in FC, regard to the one in kicad)
I suggested already at the kicad forum to add a warning message, but it seems it is not implemented by now...
https://forum.kicad.info/t/idfv3-export-bug/1447/10
In my kicad StepUp macro if the edge is not closed, I will display a warning message and the user will be able to inspect the board edge 'wire' to check where it is wrong :D
abdullah wrote:Do you happen know why I do not have IDF information of components (EMP file) in any of my boards? Is there anything special to do in KiCAD so that this file gets populated with component information?
to add IDF components to kicad, so the IDF exporter could export them, you need to:
1. have a library of idf models or create your own idf models
2. add each idf model to each 3d kicad part (adding idf model beside wrl model)
than you will get an exported board with idf models
After that you need to create a csv file to associate idf model to step model and put the corresponding step models beside your csv file in idflibs
I think this is a very complicate process based on an outdated 3d standard (moreover I cannot find any idf lib around...), that can be avoided just using kicad StepUp script (now) or kicad StepUp Macro (soon) or kicad StepUp WB (later on) :)
abdullah wrote:I will welcome this. I am manually positioning the most space requiring components in my boards manually and it is cumbersome. Let me know when it is ready in order to test it.
you can have a try at kicad StepUp script ...
you just need to copy your step models in the same place of the wrl models, naming step file as the corresponding wrl file and launch the script... it works under windows, linux, osx ;)
Just PM me if you need more assistance

going back to my kicad StepUp macro dev work :)
keep in touch
Maurice
abdullah
Posts: 3174
Joined: Sun May 04, 2014 3:16 pm

Re: IDFv3 support - upgrade code

Postby abdullah » Fri Jan 08, 2016 1:08 pm

easyw-fc wrote:I opened the file with my new (not completed) kicad StepUp tools Macro and I found that the edge of the pcb is not closed... There is a small non coincident in the edge points that makes the kicad IDF exporter make a wrong IDF data.
Ok. Thanks for letting me know. :)
easyw-fc wrote:to add IDF components to kicad, so the IDF exporter could export them, you need to:
1. have a library of idf models or create your own idf models
2. add each idf model to each 3d kicad part (adding idf model beside wrl model)
than you will get an exported board with idf models
After that you need to create a csv file to associate idf model to step model and put the corresponding step models beside your csv file in idflibs
I think this is a very complicate process based on an outdated 3d standard (moreover I cannot find any idf lib around...), that can be avoided just using kicad StepUp script (now) or kicad StepUp Macro (soon) or kicad StepUp WB (later on) :)
Thanks for the info. I thought that there were already idf components in KiCAD. For me it makes no sense to pursue IDF now (it may still be handy in FC to visualize a PCB done with another EDA providing the information), and will surely take the path of the kicad Stepup macro/WB, as STEP models do make much more sense.
User avatar
easyw-fc
Posts: 2773
Joined: Thu Jul 09, 2015 9:34 am

Re: IDFv3 support - upgrade code

Postby easyw-fc » Wed Aug 24, 2016 3:53 pm

sgrogan wrote:
easyw-fc wrote:If this could be of some interest I could post the latest code...
anyway I'm adding this upgrade / FreeCAD WB to my kicad StepUp project
http://sourceforge.net/projects/kicadstepup/
to let kicad users to fully import exported IDF files in FreeCAD
Looks cool easyw-fc. You want this updated on https://launchpad.net/~freecad-communit ... ubuntu/ppa or testing first?
If you push the code to SF we could enable auto-build.
Hi @sgrogan
I updated the Idf importer (legacy version) to allow users to add Z-axis-rotation in the "footprints_models.csv" file
as for the following user post
viewtopic.php?f=8&t=17124#p135195
https://github.com/easyw/Idf
please let me know if this feature may be of some interest for all FC users
Maurice