Metadata: One Standard to Rule Them All

[chennes]

Over in this discussion we've been working on some ideas towards a new kind of Add-On, tentatively called "Themes". These encompass both stylesheets as well as user preferences (most often color and font, I expect). I'd like the Add On package to include a metadata file, and this seems like a nice testbed for developing a format that could later be extended to other types of Add-Ons.

There are two main decisions that need to be made: what "wire format" to use (e.g. JSON, XML, INI, custom text, etc.), and what information should be in this file. I think in an ideal world we'd adopt someone else's metadata standard, but I don't know that there are any really good candidates out there, it seems everyone rolls their own. So I guess that leaves us doing the same...

I think there are pros and cons to all of the wire formats: I happen to like JSON for being flexible enough for easy extensibility down the road, but also being very easy to parse for most simple metadata needs. XML support is of course already baked into FreeCAD, but I feel like XML is overkill for metadata. Using something like QGIS's metadata.txt format has the appeal of simplicity, but I'm concerned about its lack of flexibility. What are your thoughts on wire format?

In terms of information content, I think QGIS's list proves a great start. We'll also want some chunks of metadata that are specific to each type of Add-On (for themes, for example, something like a "Light or Dark" entry). Comments or ideas on what we should include in the metadata file?

[mnesarco]

I also support the idea of JSON wire format for metadata. In ExtMan I used ini format:

https://github.com/mnesarco/FreeCAD_Ext ... nifest.ini

But JSON will be a lot more flexible.

[hyarion]

My personal goto "wire-format" is json (using nlohmann::json for c++), but since the rest of freecad is xml based (some exceptions do exist) would it be better to use xml here too?
Reducing used technologies often makes projects easier to maintain and with a project of this size it might be good to keep it limited.
I would like to hear what for instance wwmayer has to say here, is there any preferred file format going forward for the project?

I don't have much input on what to store in the files other than that we should keep it as minimalistic as possible. It's always easier to extend a specification, but near impossible to reduce it. So I think it would be good to keep it simple to start with :)

[Kunda1]

pinged by pinger macro

@yorik can you weigh in on this proposal? Thanks!

[chennes]

the rest of freecad is xml based (some exceptions do exist)

We use JSON in several places: PartDesign uses it for hole standards, Arch uses it for IFC, Path uses it for tools, etc. Actually, we probably could stand to centralize our C++ JSON parsing in the same way we have centralized our XML -- right now the nlohmann library is in Part Design, but it might be better as part of the Core. (I assume the Python JSON parsing is being done using the standard Python import).

[mnesarco]

It is very important to distinguish between metadata and data. What i understand is that we are talking about metadata of extensions here so it means data describing the extension, not the data administered by the extension itself.

Each extension can store its data in whatever format, it doesnot require any standard. What should be standard is the metadata required for integration.

[chennes]

What i understand is that we are talking about metadata of extensions here so it means data describing the extension, not the data administered by the extension itself.

Yes, exactly. For example, QGIS plugins have the following metadata, which is probably pretty close to what we need:

• name - a short string containing the name of the plugin
• qgisMinimumVersion - dotted notation of minimum QGIS version
• qgisMaximumVersion (optional) - dotted notation of maximum QGIS version
• description - short text which describes the plugin, no HTML allowed
• about - longer text which describes the plugin in details, no HTML allowed
• version - short string with the version dotted notation
• author - author name
• email - email of the author, only shown on the website to logged in users, but visible in the Plugin Manager after the plugin is installed
• changelog (optional) - string, can be multiline, no HTML allowed
• experimental (optional) - boolean flag, True or False - True if this version is experimental
• deprecated (optional) - boolean flag, True or False, applies to the whole plugin and not just to the uploaded version
• tags (optional) - comma separated list, spaces are allowed inside individual tags
• homepage (optional) - a valid URL pointing to the homepage of your plugin
• repository - a valid URL for the source code repository
• tracker (optional) - a valid URL for tickets and bug reports
• icon (optional) - a file name or a relative path (relative to the base folder of the plugin’s compressed package) of a web friendly image (PNG, JPEG)
• category (optional) - one of Raster, Vector, Database and Web
• plugin_dependencies (optional) - PIP-like comma separated list of other plugins to install

[mnesarco]

Good!

As a reference, this is what i use in extman:

Code: Select all

;***************************************************************************
;* Meta Section                                                            *
;* General identification metadata                                         *
;***************************************************************************
[general]
version=0.1
name=ExtMan
title=Extension Manager
description=Search, Install, Update Extensions
iconPath=resources/icons/ExtManWorkbench.svg
license=LGPL2+
licensePath=LICENSE
authors=Frank Martinez
categories=Core
homepage=https://github.com/mnesarco/FreeCAD_ExtMan

;***************************************************************************
;* Packages dependencies                                                   *
;***************************************************************************
[dependencies]
workbenches=

pylibs=python3-pyside2.qtwebchannel,\
  python3-pyside2.qtnetwork,\
  python3-pyside2.qtwebenginecore,\
  python3-pyside2.qtwebenginewidgets

optionalpylibs=

external=

;***************************************************************************
;* Install                                                                 *
;***************************************************************************
[install]
notesPath=INSTALL.md
mode=git
checkUpdates=true

;***************************************************************************
;* Git config                                                              *
;***************************************************************************
[git]
repository=https://github.com/mnesarco/FreeCAD_ExtensionManager.git

[galou_breizh]

I think that in the future all the FreeCAD ecosystem will have to be package-based (the current issue with FreeCAD extension capability is that C++ modules must be integrated in the core repository). I already proposed to use colcon and even did a first positive test with it. Colcon uses xml for package description: package.xml, which contains already a lot of general metadata that we would need and is extensible.

[chennes]

That ROS metadata file looks pretty good to me: it's helpful that they are up to version 3 of the file. That says to me that they've been using it and improving it for a while now, and have solved some of the problems that we might run across if we were to invent our own format. We don't need all of the build dependency stuff that they have, but otherwise we could probably just use that format wholesale. The spec file is public domain, which is nice as well.

I'm game for using that format as-is for now and revisiting it once we have something coded and usable that we can test out. Are there any objections to trying to use the format described in REP 149 as the basis for our metadata file?