Implement fbs schema & json to fbs converter tool by Thomas-Mikhael · Pull Request #738 · eclipse-score/communication

Thomas-Mikhael · 2026-07-21T12:21:08Z

Define the .fbs schema file that mirrors the existing JSON configuration structure.
Build a JSON-to-FlatBuffer converter tool for generating FlatBuffer config from existing JSON configurations
Flag to control compiling the Flatbuffers (--config=flatbuffers), default is off
PyTest for the tool

+ Define the .fbs schema file that mirrors the existing JSON configuration structure. + Build a JSON-to-FlatBuffer converter tool for generating FlatBuffer config from existing JSON configurations + Flag to control compiling the Flatbuffers (--config=flatbuffers) + PyTest for the tool

castler · 2026-07-23T05:31:05Z

As discussed via Slack - I love that we go for the Flatbuffers support and the contribution you perform here! But it would be very welcome to see the roadmap and map for its qualification before continuing here. Stating again: This does not require a full and ready qualification, we would just like to judge if the qualification at the end can succeed.

Thomas-Mikhael · 2026-07-23T07:22:20Z

As discussed via Slack - I love that we go for the Flatbuffers support and the contribution you perform here! But it would be very welcome to see the roadmap and map for its qualification before continuing here. Stating again: This does not require a full and ready qualification, we would just like to judge if the qualification at the end can succeed.

Hi Jan, I agree.
Today I have internal meeting with the responsible Arch/Manager to discuss that
Will keep you updated.

castler

Thanks this is going into the right direction.

I still believe that we need a flatbuffers certifcation for the 1.0 - and not only in 2027. And we should have a parallel workstream with weekly (or bi-weekly) reporting about the tasks there.

Anyhow - for this PR, my major concern is that we know have 4 places where we need to know the different schemas:

In the JSON Parser (soonish also flatbuffers parser +1)
In the JSON Schema
In the Flatbuffers Schema
In the JSON to Flatbuffers converter

I think this is to much duplication and will make it not maintainable. We should strive to have a generic flatbuffers schema generatioin and also a generci json to flatbuffers converter - to reduce the duplication as much as possible.

castler · 2026-07-21T12:40:18Z


+# Enables FlatBuffers-based mw::com configuration tooling (disabled by default,
+# see score/mw/com/impl/configuration/flatbuffers_flags.bzl).
+common:flatbuffers --//score/mw/com/impl/configuration:enable_flatbuffers=true


Can we please name this experimental_enable_flatbuffer_configuration

castler · 2026-08-03T11:55:20Z

+//score/mw/com/impl/configuration/converter:json_to_flatbuffer
+//score/mw/com/impl/configuration/converter:mw_com_config.bin


We should guard this via the experimental flag - and since the experimental flag is per default off, these targets should net yet be publicly visible.

Is it okay if I remove them from here and add them to EXCLUDED_PUBLIC_TARGETS in quality/visibility_guard/parser.py?

No, as Long as it is Experimental it should Not be public

castler · 2026-08-03T11:56:47Z

+the pinned ``@flatbuffers//:flatc`` compiler, so ``.bin`` artifacts become part
+of the build graph and can be consumed by other targets (e.g. as ``data``).
+"""
+


you are missing a visibility declaration in this *.bzl file

I'll add it as
visibility(["//score/mw/com/impl/configuration/..."])

castler · 2026-08-03T11:58:56Z

+//
+// FlatBuffers schema mirroring the mw::com runtime configuration.
+//
+// This schema is the binary-serialization counterpart of the JSON configuration


I would have expected that we generate this file out of the JSON schema - otherwise, we get a problem with maintainability - as we have to take care in various places the the configuration is maintained.

That would not be safe, JSON Schema is loosy and defines only the structure.
1- It doesn't specify the size (Integer widths: uint8, 16, ...)
For an ASIL / safety-relevant config, silently inferring a wrong width (e.g. int64 where the C++ side expects uint16) is exactly the kind of defect that wouldn't be caught by flatc.
2- Enum sentinels e.g. INVALID = 0 is a deliberate "unset detection" (design mirroring) for QualityType
3- = null optional-scalar vs baked default
4- Table names for nested objects (e.g. ServiceTypeEvent vs InstanceEvent)
5- json_to_flatbuffer.py hardcodes the same key->field mapping

So yes, maintaining 2 Schemas is not great, but it's the safest way here (with existing one common testing for both - to come in next stories)
Also, JSON Schema is not changing that often, mostly stable, right?

What I suggest, is to implement a "drift-detection test" that parses the JSON schema and the .fbs and fails the build if their structure diverges.
This catches if someone edits the JSON schema and forgets the .fbs.
What do you think?

Its not only the two schemas, we will also have another instance of this mapping in the json to flatbuffer tool.
This will be (hopefully) a long-term solution, if flatbuffers is safety certified, and thus it will be a maintaince nightmare. I agree that the config does not change often, but it will change - and then you do not want to edit so many places.

I see multiple options:

1st) Have integers always as uint64. We have the very same problem with JSON and there we check the boundary values within the C++ code. If we go this way, it is the same mechanichs in JSON ant Flatbuffers. Same is true for enum sentinels or "null optional scalars. We should just handle both the same and not have defaults in the one, where we have optional values in the other

2nd) Generate the JSON schema from flatbuffers. As per my understanding this is directly supported by flatc. <- But this will be though since this will only work with "experimental flag turned on" most certainly?

I'll investigate the second option as I see it more robust.

I think we can have the JSON schema committed to the repo, and when a speial flag is on then it will be regenerated and overwritten, but this would be needed only with Schema change/update.
Then we will not need to maintain both schemas manually, but just the Flatbuffers schema and regenerate the JSON one.
Would this be okay if feasible?

yes that would be better IMHO

Ok, here's some analysis limitations for using fbs schema as our main schema to maintain and generate the JSON schema from it using 'flatc --jsonschema':

Hyphen '-' is forbidden in fbs, so I need to convert it to '_' for example, and restore it when generating the JSON schema (doable)

flatc --jsonschema output can't reproduce the rich hand-authored JSON schema (it loses titles/descriptions/defaults/min-max/string-enums)
To not tradeoff here, I'd try adding custom attributes to fbs and try writing a script that appends these attributes to the generated JSON Schema (will be much of code but if once done, would save the maintenance nightmare you mentioned)

I'll ping you once I have more updates

castler · 2026-08-03T11:59:40Z

+#
+# SPDX-License-Identifier: Apache-2.0
+# *******************************************************************************
+"""Convert an mw::com configuration JSON file into a FlatBuffer binary.


My hope is, that if we auto generate the *.fbs, we also get rid of this tool (or make it way more easy - as we do not have yet another place where we translate between the two tools

This converter tool is only a facilitating option for whom already has a JSON config and wants to use flatbuffer config binaries for its benefits.
It's converting to flatbuffer convertable json format (snake case)

User has the freedom to use any other method to get flatbuffer binaries directly.
For example:

Programmatic builder (FlatBufferBuilder in C++, the flatbuffers package in Python, etc.) with any other text formats: YAML, TOML, XML, CSV, .ini.

A database or network service — query rows/records and build directly.

In-memory C++ objects

I understand that, but Here we have then again the key mapping. Thats why auch a tool ja fine, but it should ne generic without hardcoding again the mappings

Thomas-Mikhael

I added my replies/questions for each comment

Thomas-Mikhael · 2026-08-03T14:50:22Z

+//
+// FlatBuffers schema mirroring the mw::com runtime configuration.
+//
+// This schema is the binary-serialization counterpart of the JSON configuration


That would not be safe, JSON Schema is loosy and defines only the structure.
1- It doesn't specify the size (Integer widths: uint8, 16, ...)
For an ASIL / safety-relevant config, silently inferring a wrong width (e.g. int64 where the C++ side expects uint16) is exactly the kind of defect that wouldn't be caught by flatc.
2- Enum sentinels e.g. INVALID = 0 is a deliberate "unset detection" (design mirroring) for QualityType
3- = null optional-scalar vs baked default
4- Table names for nested objects (e.g. ServiceTypeEvent vs InstanceEvent)
5- json_to_flatbuffer.py hardcodes the same key->field mapping

So yes, maintaining 2 Schemas is not great, but it's the safest way here (with existing one common testing for both - to come in next stories)
Also, JSON Schema is not changing that often, mostly stable, right?

What I suggest, is to implement a "drift-detection test" that parses the JSON schema and the .fbs and fails the build if their structure diverges.
This catches if someone edits the JSON schema and forgets the .fbs.
What do you think?

Thomas-Mikhael · 2026-08-03T15:24:59Z

+#
+# SPDX-License-Identifier: Apache-2.0
+# *******************************************************************************
+"""Convert an mw::com configuration JSON file into a FlatBuffer binary.


This converter tool is only a facilitating option for whom already has a JSON config and wants to use flatbuffer config binaries for its benefits.
It's converting to flatbuffer convertable json format (snake case)

User has the freedom to use any other method to get flatbuffer binaries directly.
For example:

Programmatic builder (FlatBufferBuilder in C++, the flatbuffers package in Python, etc.) with any other text formats: YAML, TOML, XML, CSV, .ini.

A database or network service — query rows/records and build directly.

In-memory C++ objects

Thomas-Mikhael · 2026-08-04T07:40:45Z

+//score/mw/com/impl/configuration/converter:json_to_flatbuffer
+//score/mw/com/impl/configuration/converter:mw_com_config.bin


Is it okay if I remove them from here and add them to EXCLUDED_PUBLIC_TARGETS in quality/visibility_guard/parser.py?

Thomas-Mikhael · 2026-08-04T08:05:58Z


+# Enables FlatBuffers-based mw::com configuration tooling (disabled by default,
+# see score/mw/com/impl/configuration/flatbuffers_flags.bzl).
+common:flatbuffers --//score/mw/com/impl/configuration:enable_flatbuffers=true


Thomas-Mikhael · 2026-08-04T08:06:47Z

+the pinned ``@flatbuffers//:flatc`` compiler, so ``.bin`` artifacts become part
+of the build graph and can be consumed by other targets (e.g. as ``data``).
+"""
+


I'll add it as
visibility(["//score/mw/com/impl/configuration/..."])

Thomas-Mikhael requested review from LittleHuba, bemerybmw, castler, crimson11 and hoe-jo as code owners July 21, 2026 12:21

github-project-automation Bot added this to COM - Communication FT Jul 21, 2026

Thomas-Mikhael requested a review from limdor as a code owner July 21, 2026 12:21

github-project-automation Bot moved this to Backlog in COM - Communication FT Jul 21, 2026

Thomas-Mikhael force-pushed the feature/flatbuffer_schema_and_converter_from_json branch from a2ca224 to 2510ac0 Compare July 21, 2026 13:12

anmittag assigned Thomas-Mikhael Jul 22, 2026

castler reviewed Aug 3, 2026

View reviewed changes

Thomas-Mikhael requested a review from castler August 4, 2026 08:07

Thomas-Mikhael commented Aug 4, 2026

View reviewed changes

		//score/mw/com/impl/configuration/converter:json_to_flatbuffer
		//score/mw/com/impl/configuration/converter:mw_com_config.bin

Uh oh!

Conversation

Thomas-Mikhael commented Jul 21, 2026

Uh oh!

castler commented Jul 23, 2026

Uh oh!

Thomas-Mikhael commented Jul 23, 2026

Uh oh!

castler left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Thomas-Mikhael left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants