Add testing approaches (LBRON-1713) by brechtvdv · Pull Request #19 · lblod/gitbook-decide-write-up

brechtvdv · 2026-07-31T14:26:49Z

No description provided.

Updated the testing approach section to clarify manual testing procedures and risks associated with the Policy Impact Report.

mention survey

Added validation of decision applicability dates to the interface description.

mirdono

Left some suggestions. Since they are often opinion-based, they are obviously up for discussion.

mirdono · 2026-08-05T08:19:10Z

+
+This risk is mitigated by adding a tooltip with the Linked Data definition of the link. For the example above, following tooltip now appears: The last date any part of the legislation is in force, if the date is known (can be seen as the end date of a dc:valid range for this resource).
+
+Also, ABB created guidelines (slides and video) for each city that lists all possible links and demonstrates with an example decision what the enrichment should be.


Question: Are these slides and videos somewhere (publicly) available so we can link to them here?

Fix: 0872502

Videos still need to be uploaded to Youtube, but I have sent a mail with the question to add

mirdono · 2026-08-05T08:19:10Z

+
+### Pilot-specific testing
+
+Each pilot has only a single environment, configured through its own docker-compose override capturing its specific setup (data sources, pilot-specific services). Rather than automated tests, pilots validate their environment manually — for example by running a pipeline or sharing datasets.


Suggestion: Maybe rephrase second sentence to something like "pilots can decide their own test approach."

Arguably, nothing stops the pilots from setting up automated tests for their app instance. But it is not offered out of the box, and requires (a lot of) effort on their part.

Fix 06798ab

mirdono · 2026-08-05T08:19:11Z

 The service uses a cron mechanism to periodically run the validations ([https://github.com/lblod/loket-report-generation-service/tree/master#reports](https://github.com/lblod/loket-report-generation-service/tree/master#reports)). By default, the SHACL validations will run every day at 03:00 ([https://github.com/lblod/app-decide/blob/development/config/reports/shacl-report.js#L35](https://github.com/lblod/app-decide/blob/development/config/reports/shacl-report.js#L35)).

-During testing, we discovered that validating all decisions causes a high load on the triple store: a validation issue in one municipality typically recurs for every decision. Therefore, we added sampling to validate a limited set of 100 decisions in each validation run by default.
+By navigating to the provided REST API `/shacl-reports/latest/issues`, cities see the results of the latest generated report.


Question: Can we explicitly link to the report on our TEST instance?

Currently, I would have to search in the dispatcher configuration to determine on which url this API can be accessed. It would be easier if we could link to if (or show an example if that is not possible), to give readers a better idea of what to expect from such reports.

Note, on our TEST environment the report-generation is currently still disabled. Is this intentional, still waiting for further improvements or something?

This was disabled to prevent load on the AI pipelines. I have turned it on now.

Fix e6069d2
To be tested when run is complete

mirdono · 2026-08-05T08:19:11Z

-Testing can be done by navigating to the provided REST API `/shacl-reports/latest/issues` .
-
-During DECIDe, we will monitor the validation results and analyze what their root causes are.
+ABB uses the API to monitor the validation results and analyze what their root causes are.


Question: Can we add an example of such an analysis and actions take for it?

If we already used the API to discover issues and analyze them, I would shortly mention it as an example usage. This would make the statement here more convincing.

Added a real example: 717b5b2

mirdono · 2026-08-05T08:19:11Z

+#### High load on triple store
+
+During testing, we discovered that validating all decisions causes a high load on the triple store: a validation issue in one municipality typically recurs for every decision. 
+Therefore, we added sampling to validate a limited set of decisions in each validation run. By default, 100 decisions are validated.


Question: Should we elaborate on this that it may cause issues to be missed?

By default the service is started with ONLY_KEEP_LATEST_REPORT: true. This may cause a report listing issues to be removed when a next report is generated. Alternatively, we could remove the environment variable so it defaults back to false.

Elaborated: 00edbf8

mirdono · 2026-08-05T08:19:11Z

+
 ### Risks & mitigations

+There is a risk that it is unclear for the reviewer what a certain link means. For example, `eli:no_longer_in_force`.


Suggestion: I would rephrase this section a bit to emphasize the (imo) actual underlying risk.

I think the actual, underlying risk described here is that users may not know what exactly it means to give a thumbs up or down for a row. That the meaning of a link may be unclear is only one potential cause of this. Users might also not (fully) understand what the mentioned type means of what they have to do when a row is only partially correct.

Imo we tried to mitigate such things by (in random order):

add tooltips in the HVT itself to explain what things mean; and

providing supporting material (slides + video) explaining and demonstrating the usage of the HVT.

mirdono · 2026-08-05T08:19:11Z

 ### Risks & mitigations

+There was the risk that the Policy Impact Report would have its own validation visualization, separate from the human validation tool, which would increase the maintenance effort.
+This is mitigated by showing a link to the Human Validation Tool and making sure the same filters are applied. For example, when the Policy Impact Report shows 17 decisions for the city of Freiburg with a positive impact on SDG 1, a user can review these exact 17 decisions for Freiburg in the Human Validation Tool.


Suggestion: Imo the actual mitigation is integrating the PIR validation in the HVT.

I think the actual mitigation for the described risk is that we developed the PIR validation within the HVT, along with the other validations. This allows to build it on the same principles, UI, UX, ... (for better or for worse.) The fact that you link to it from the PIR is just a consequence (if it was a different page within the same frontend we would still have a link). But passing sufficient data with the link so you immediately land on the correct HVT page is arguably a (significant) UX choice on our part.

Fix 2c9be78

mirdono · 2026-08-05T08:19:11Z

+
 ### Risks & mitigations

+No risks have been identified.


Suggestion: Imo the same risk as the PIR, having a validation separate from the HVT, applies here.

We could have developed a completely separate tool to validate RMZs (or codelist mapping in general). But choose to put in the same HVT for consistency etc. Albeit, putting it in the HVT was the logical decision here as there is no separate frontend for RMZs on our side (in contrast to the PIR scenario.)

Added: 2d2fdea

brechtvdv added 10 commits July 30, 2026 11:54

Add testing to uc0.0 data space

9e8f479

Add testing approach to uc0.0 hvt

7a85407

Clarify testing approach and risks for uc0.1

2a386b3

Updated the testing approach section to clarify manual testing procedures and risks associated with the Policy Impact Report.

Update write-up-uc0.1-policy-impact-report.md

0126fc7

mention survey

Add testing approach uc1

783a4ea

Added validation of decision applicability dates to the interface description.

Enhance testing approach section

e2826b2

Fix links in testing approach section

1aa61e8

Update testing approach data quality manager

86b0409

Add testing section repeatable data plan

677e433

Add testing approach vc

d009ac9

mirdono self-requested a review August 4, 2026 08:12

mirdono requested changes Aug 5, 2026

View reviewed changes

brechtvdv added 10 commits August 6, 2026 11:54

Fix triplestore spelling

844fd04

Add references to guidelines

0872502

Rephrase test data space

06798ab

Add link to REST API server

e6069d2

Add validation example

717b5b2

Add missing issues

00edbf8

rephrase risk validation

3d13aee

Fix up to date

25c202d

Rewrite mitigation pir

2c9be78

Add risk RMZ

2d2fdea


		This risk is mitigated by adding a tooltip with the Linked Data definition of the link. For the example above, following tooltip now appears: The last date any part of the legislation is in force, if the date is known (can be seen as the end date of a dc:valid range for this resource).

		Also, ABB created guidelines (slides and video) for each city that lists all possible links and demonstrates with an example decision what the enrichment should be.


		### Pilot-specific testing

		Each pilot has only a single environment, configured through its own docker-compose override capturing its specific setup (data sources, pilot-specific services). Rather than automated tests, pilots validate their environment manually — for example by running a pipeline or sharing datasets.


		### Risks & mitigations

		There is a risk that it is unclear for the reviewer what a certain link means. For example, `eli:no_longer_in_force`.

Uh oh!

Conversation

brechtvdv commented Jul 31, 2026

Uh oh!

mirdono left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants