Metadata: One Standard to Rule Them All

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
User avatar
onekk
Posts: 648
Joined: Sat Jan 17, 2015 7:48 am
Contact:

Re: Metadata: One Standard to Rule Them All

Postby onekk » Thu Apr 08, 2021 9:44 am

Maybe using something already supplid by python3.

One thing that it is not cited but that I've discussed for managing a large quantities of data is to have a way to include in a FCstd file a sqlite3 container.

squlite3 interface is supplied as a standard python module and include in FreeCAD (also in AppImages).

For this there will be only to need, A way to add a file in the FCstd "container" and let the user do it's magic with the content, (in other word, using the data inserted in the sqlite3 file).

But I was told that this could present some problems, (not very clear how, but I think something related to security).

The argument was brought up when discussion about annotation "styles" in Draft, and the question is where to store annotation syles and annotation content in a FCstd file.

I think that every security problem could be "managed" in two ways:

1) some checkbox in preferences, to allow "unsafe code execution" set to "false" by default and that the user could "activate at your risk"
2) a warning when opening a file that contains "dangerous code" , but I think that it will be very annoying in the long time.


For the format, please use something already present, don't add another special ibrary, I think that a "widespread" standard like JSON or XML will be enough, INI files is also "Human readable" and "hand modifiable" if it is needed or wanted.

I think that as FreeCAD seems to be very appealing as a substituion to many commercial CAD in particular for "little workshop" and "small (or very small) industries", every other complication will be a bad idea.

my two cents.

Regards to all.

Carlo D.
Introduzione a FreeCAD in Italiano: https://github.com/onekk/freecad-doc
User avatar
mnesarco
Posts: 289
Joined: Thu Mar 26, 2020 8:52 pm

Re: Metadata: One Standard to Rule Them All

Postby mnesarco » Thu Apr 08, 2021 2:00 pm

onekk wrote:
Thu Apr 08, 2021 9:44 am
One thing that it is not cited but that I've discussed for managing a large quantities of data is to have a way to include in a FCstd file a sqlite3 container.
sqlite3 can be good for massive data, but it is an overkill for metadata. Metadata must be small and human readable. I am using sqlite3 as my Document Format in another project and it works amazingly well. It could be a good idea to open another thread about that topic.
User avatar
chennes
Posts: 819
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: Metadata: One Standard to Rule Them All

Postby chennes » Thu Apr 08, 2021 7:39 pm

If anyone wants to see the draft implementation of a reader for the REP 149 "package.xml" metadata file format, I've just committed an initial version:
https://github.com/chennes/FreeCAD/blob ... taReader.h
https://github.com/chennes/FreeCAD/blob ... Reader.cpp

The main design goal was to implement exactly the format described in REP149. It also supports the inclusion of arbitrary additional XML metadata: unrecognized tags are not treated as errors. Simple tags are parsed as "generic" metadata, accessible via C++ operator[] (and ultimately eventually a similar Python interface, of course). Direct access to the DOM tree is provided for code that needs more complex XML elements for input.

ETA: OK, not quite exactly the format. Right now XHTML is not allowed in the description element.
Chris Hennes
Pioneer Library System
User avatar
onekk
Posts: 648
Joined: Sat Jan 17, 2015 7:48 am
Contact:

Re: Metadata: One Standard to Rule Them All

Postby onekk » Fri Apr 09, 2021 7:50 am

mnesarco wrote:
Thu Apr 08, 2021 2:00 pm
onekk wrote:
Thu Apr 08, 2021 9:44 am
One thing that it is not cited but that I've discussed for managing a large quantities of data is to have a way to include in a FCstd file a sqlite3 container.
sqlite3 can be good for massive data, but it is an overkill for metadata. Metadata must be small and human readable. I am using sqlite3 as my Document Format in another project and it works amazingly well. It could be a good idea to open another thread about that topic.
OK for critics, but, in general, better not to "reinvent the wheel",

OK for the implementation of XML as told by chennes, as it not involve another library to "take care of".

With an hobbyst "point of view" in mind, I think that the most important work that has to be done in FreeCAD is:

- Better documentation
- Some improvement regard Python Scripting, in some area like Path that is not "well scriptable yet", This is in my humble and biased opinion the most distinguishing function of FreeCAD and has to be leveraged as a "point of strength".
- Some work has to be done in Assembly, take a decision on which Assembly would be the "official" Assembly WorkBench
- TechDraw improvement and documentation as it seem to be a "most demanded" field from those who use FreeCAD for real "for money" work.
- more integration with OCCT, in term of Python function exposed, when you dig into sources, you will find many "hidden gems" but it seems that OCCT is "not fully implemented" in term of C++ to Python "bridge methods" and some hard coded assumptions are done, (@wmayer is making a great job on improving things, but it seems that not much developers are working on this thing, i know that is rather "low level" and not much visible work)

Metadata standardization is a good way to think things, as "simplicity is better than complexity" if simplicity is not slowing things down, to return on topics. When a metadata is big, or better what you intend for "metadata".

A standardized way to have metadata will permit to add to each object in the treeview a field of "note" in which a user could write (even accessible inside scripting) "This object will be attached to pieces A" or "this is a work copy of PN876543251", I think will be a good improvement, but maybe I have a wrong concept of metadata.

Sorry for the inconvenience of some OT parts in this post.

Regards

Carlo D.
Introduzione a FreeCAD in Italiano: https://github.com/onekk/freecad-doc
User avatar
mnesarco
Posts: 289
Joined: Thu Mar 26, 2020 8:52 pm

Re: Metadata: One Standard to Rule Them All

Postby mnesarco » Fri Apr 09, 2021 2:09 pm

onekk wrote:
Fri Apr 09, 2021 7:50 am
Carlo D.
Hi Carlo, You are talking about something completely different. The metadata discused in this thread is about describing extension (Addons) as software packages. Nothing related to internal FreeCAD objects.

Each extension (Addon) is an independent piece of software integrated in freecad using some API. There are some minimal info required for the integration, and that information must be provided by the extensions somehow. For example the Icon of the Workbench or the Location of the resources provided by theextension, the name of the authors, the license. That kind of info.

The basic idea is that each extension provides a standard manifest file containing such metadata.
User avatar
onekk
Posts: 648
Joined: Sat Jan 17, 2015 7:48 am
Contact:

Re: Metadata: One Standard to Rule Them All

Postby onekk » Fri Apr 09, 2021 3:14 pm

Sorry, I've totally misunderstood things.

Anyway setting a standard is always a good thing, documenting clearly it is another good thing, sadly is rarely done.

Sorry for the bothering and OT posts.

Regards to all.

Carlo D.
Introduzione a FreeCAD in Italiano: https://github.com/onekk/freecad-doc
User avatar
chennes
Posts: 819
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: Metadata: One Standard to Rule Them All

Postby chennes » Fri Apr 09, 2021 3:26 pm

mnesarco wrote:
Fri Apr 09, 2021 2:09 pm
There are some minimal info required for the integration, and that information must be provided by the extensions somehow. For example the Icon of the Workbench or the Location of the resources provided by theextension, the name of the authors, the license.
I'd appreciate it if people more familiar than me review the standard information included in the package.xml metadata format and give me advice on other things to natively support (and potentially require). The data in that standard is:

Required:
  • name
  • version (semantic versioning required by REP 149, not enforced by current reader (yet?))
  • maintainer (email required)
  • license (file optional, multiple allowed)
  • description (their standard allows XHTML, I did not implement that (yet?))
Optional:
  • url (website, bug tracker, repository, multiple allowed)
  • author (email optional, multiple allowed)
  • depend (package name, version requirement, multiple allowed)
  • conflict (package name, version requirement, multiple allowed)
  • replace (package name, version requirement, multiple allowed)
I suggest that we directly support an icon file and a screenshot file, both of which are optional from the parser's perspective. And as mentioned above, the way I wrote the parser, each type of package can potentially also require other metadata that the parser will provide access to even if it doesn't actually recognize the tags.
Last edited by chennes on Fri Apr 09, 2021 4:30 pm, edited 1 time in total.
Chris Hennes
Pioneer Library System
User avatar
mnesarco
Posts: 289
Joined: Thu Mar 26, 2020 8:52 pm

Re: Metadata: One Standard to Rule Them All

Postby mnesarco » Fri Apr 09, 2021 4:23 pm

chennes wrote:
Fri Apr 09, 2021 3:26 pm
I suggest that we directly support an icon file and a screenshot file, both of which are optional from the parser's perspective. And as mentioned above, the way I wrote the parser, each type of package can potentially also require other metadata that the parser will provide access to even if it doesn't actually recognize the tags.
To make it easier for the AddonManager, ExtMan, or any future extension manger there are some important fields (in addition to your mandatory section):

1. Repository URL (git)
2. Readme URL
3. Readme Format (html, markdown, wiki, ...)
4. Icon URL (Preferably a local path relative to the installation dir)
5. tags (an array of string to classify the extension ie [architecture, mechanics, UI, 3DPrinting, ...])
6. Version? Currently versioning is based on git commits, but probably semantic versioning would be better.
7. Content: currently in Mod dir there can be more than just Workbenches it is important to know what is in the package, a single extension can contain Many Workbenches, macros, any kind of python stuff, and if i understand well you are adding themes. So the extension managers needs to know what is included in the package.

There must be a small common mandatory set of fields, then there must be a section for each kind of extension, and some mandatory metadata on each type.

Some kind of struture like:

Code: Select all

Content:
  Workbenches:
     MyWorkbenc1:
       name: ...
       icon: ...
       ...
    MyAuxWorkbench:
      name: ...
      icon: ...
      ...
   Macros:
     macroXXX:
       ....
     macroYYY:
       ....
   Themes:
     ThemeA:
       .....
     ThemeB:
       .... 
   Libraries:
     Lib1:
       .....
     Lib2:
       ....
       
       
User avatar
chennes
Posts: 819
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: Metadata: One Standard to Rule Them All

Postby chennes » Fri Apr 09, 2021 4:29 pm

mnesarco wrote:
Fri Apr 09, 2021 4:23 pm
6. Version? Currently versioning is based on git commits, but probably semantic versioning would be better.
My apologies, I left that off, I'll edit. Version is indeed in there as a required field, and the REP standard requires semantic versioning (though my reader does not enforce it).
Chris Hennes
Pioneer Library System
User avatar
chennes
Posts: 819
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: Metadata: One Standard to Rule Them All

Postby chennes » Fri Apr 09, 2021 4:49 pm

mnesarco wrote:
Fri Apr 09, 2021 4:23 pm
1. Repository URL (git)
Is your suggestion that this be required? It's an allowed url value right now.

2. Readme URL
3. Readme Format (html, markdown, wiki, ...)
Maybe add "readme", "documentation" and/or "help" as a possible URL categories? What is the purpose of the "format" entry? Is the intention to allow inline display of the URL contents?

4. Icon URL (Preferably a local path relative to the installation dir)
I'd probably make it a plain string path relative to the location of package.xml, that's how the rest of the local file references work in REP149.

5. tags (an array of string to classify the extension ie [architecture, mechanics, UI, 3DPrinting, ...])
Given that the format here is XML, would this be OK as a tag with multiples allowed, rather than a comma-separated list? e.g.

Code: Select all

<tag>architecture</tag>
<tag>mechanics</tag>
<tag>ui</tag>
rather than

Code: Select all

<tag>architecture, mechanics, UI</tag>
7. Content
Yes, I think this is a good thing to support directly as well, basically as you describe. I'll add it to the framework so we have something working to poke at.
Chris Hennes
Pioneer Library System