Proposal: hierarchically linked bundle Needs exports

[AlexanderLanin]

Hierarchically linked bundle Needs exports

Context

The current needs_json action builds one composed documentation tree. A
change to any mounted docs_bundle() invalidates that action.

Issue #679 proposes cacheable per-bundle Needs exports and a final global
composition. That keeps bundle actions independent, but cross-bundle Need
links are deliberately unresolved in the bundle action. They are validated
only in the final composition.

This proposal explores a stricter alternative: a bundle may link only to an
explicitly declared bundle above it in the documentation hierarchy. The
target export is an input to the source bundle action, so every permitted
Need link is validated when its source bundle is built.

This is an architecture proposal for discussion, not an implementation plan.

Goals

• Produce cacheable Needs exports at bundle granularity.
• Validate every cross-bundle Need link in the bundle that owns the link.
• Keep the Bazel graph acyclic through a directed, hierarchical dependency
  model.
• Preserve stable composed docname paths and the existing public
  needs_json output.
• Keep docs_check as the comprehensive document-level integration check.
• Avoid standalone bundle-export builds that succeed despite unresolved Need
  links.

Non-goals

• Reconstructing HTML or a complete documentation build from bundle exports.
• Allowing arbitrary, bidirectional Need links between sibling or descendant
  bundles.
• Replacing Sphinx-Needs graph processing with a hand-written JSON merger.

Proposed model

1. Declare an upward traceability dependency

A source-bearing docs_bundle() declares zero or more explicit upward Needs
dependencies. A dependency names an ancestor bundle in the
documentation/architecture hierarchy. The bundle provider propagates this
structural relation; docs() does not declare it.

Need links in the bundle's sources may target:

• Needs owned by the bundle itself; or
• Needs exported by a declared upward dependency or its upward closure.

Links to siblings, descendants, or unrelated bundles are rejected. The
declaration is a Bazel dependency, so the producer export is built before the
bundle that references it.

The declared Bazel dependency defines the allowed upward direction. It must not
be inferred from Need links, because that would make the build graph depend on
parsed documentation content.

2. Bind a metamodel to each bundle

Each docs_bundle() declares the metamodel used to build its own Needs JSON.
The declaration is optional only because omitting it selects the built-in
SCORE metamodel.

The same model must be supplied to the local source bundle implicitly created
by docs(), through its existing metamodel = ... parameter. Projects using
a custom model should wrap both macros, so the model label is configured once
in project-local Starlark rather than repeated at every call site.

A source-bearing bundle exposes its own directly buildable Needs-JSON target.
It is built with its declared metamodel and does not depend on a consuming
docs() target.

Each action key includes:

• the bundle's sources and direct source-link data;
• the bundle's declared metamodel and documentation toolchain;
• the bundle-relative entry document; and
• the complete transitive closure of declared upward exports.

docs() composes the existing bundle outputs. The build rules deliberately do
not compare metamodels; a user who composes bundles with different models owns
their compatibility.

3. Compose a root component and its subcomponents

A component's own documentation is the local source bundle implicitly created
by docs(). Its subcomponents are explicit bundles and declare that local
source bundle as their upward dependency:

# //powertrain/BUILD.bazel
docs_bundle(
    name = "engine_bundle",
    source_dir = "engine/docs",
    upward_bundles = [":powertrain_source_bundle"],
    metamodel = ":powertrain_metamodel.yaml",
)

docs_bundle(
    name = "transmission_bundle",
    source_dir = "transmission/docs",
    upward_bundles = [":powertrain_source_bundle"],
    metamodel = ":powertrain_metamodel.yaml",
)

docs(
    name = "powertrain",
    source_dir = "docs",
    metamodel = ":powertrain_metamodel.yaml",
    bundles = [
        {
            "bundle": ":engine_bundle",
            "mount_at": "subcomponents/engine",
        },
        {
            "bundle": ":transmission_bundle",
            "mount_at": "subcomponents/transmission",
        },
    ],
)

The invocation creates :powertrain_source_bundle from docs/ only, and
:powertrain_docs_bundle from that source bundle plus both mounted
subcomponents. engine_bundle and transmission_bundle each expose their
own directly buildable Needs-JSON target. The engine and transmission actions
import only the powertrain source export; they do not import each other.

Therefore, an engine maintainer can run:

bazel build //powertrain:engine_bundle_needs_json

That target builds only the engine sources, its declared metamodel, and the
powertrain source export required by upward_bundles; it does not build the
composed documentation tree or the transmission export.

docs() must distinguish its own-source bundle from its composed documentation
bundle. An upward dependency may target only the own-source bundle, which has
no dependency on mounted children. The composed bundle contains the own-source
bundle plus its mounted children and is used for document-level checks. This
keeps the graph acyclic: the powertrain's composed bundle depends downward on
the engine and transmission bundles, while those exports depend upward only on
the powertrain's own-source export.

There is no self-registration: both relationships are explicit. The
powertrain declares its mounted children through bundles; each child declares
its parent through upward_bundles. A bundle export is an internal Bazel
interface: it may be consumed by declared upward/downward build relationships,
but is not a published needs_json artifact for external documentation
projects.

4. Validate links against imported upward exports

The bundle action imports the complete declared upward closure through the
supported Sphinx-Needs external-needs mechanism. Including the closure is
necessary because an imported parent may itself link to an ancestor. Their
contents are treated as validation inputs, not as part of the bundle's
published ownership.

Normal Sphinx-Needs processing then validates outgoing links, link conditions,
constraints, and duplicate IDs over the local Needs plus the allowed upward
interface. Any unresolved Need link is fatal in the bundle action.

Bundle exports must use a distinct base_url per imported source so duplicate
IDs cannot be silently replaced by the external-needs loader.

5. Publish a local export, not imported Needs

After validation, the bundle exports only Needs owned by its bundle.
Imported upward Needs are not re-exported. This preserves a small, explicit
interface and prevents transitive copies from masking ownership or duplicate
IDs.

A bundle export is an internal build artifact. Its bundle-relative docname is
not a published contract. Only Need IDs are stable across bundle boundaries.

6. Compose the final graph at the top level

The top-level needs_json imports all reachable local exports and runs the
normal Sphinx-Needs finalization once:

• rebase each owned Need's docname to its final mount_at;
• calculate full-graph backlinks, dead-link state, and constraints;
• run SCORE graph checks and metrics; and
• export the historical public output.

The composition must not introduce a third-party mutation patch for
Sphinx-Needs internals. It requires a supported upstream operation to
re-host a bundle Need with its final source location and local/external
semantics. The operation must be specified field by field before
implementation.

docs_check remains responsible for parsing, navigation, roles, document
references, rendering, and other checks requiring the mounted source tree.

Dependency shape and cache behavior

feature export
       ^
       | validated upward link
component export
       ^
       | validated upward link
unit export

all local exports --> lightweight top-level composition --> needs.json

A unit edit rebuilds its export and the top-level composition. A feature
edit also rebuilds exports that explicitly depend on that feature. Unrelated
branches remain cached.

This deliberately trades some invalidation for early, deterministic link
validation. The cost is bounded by the declared upward dependency closure,
rather than by every bundle in the documentation tree.

Required restrictions

Link direction

The declared upward_bundles Bazel dependencies define the allowed direction.
A link that cannot be classified as local or as targeting one of those
dependencies is invalid. This avoids hidden dependencies; Bazel rejects any
dependency cycle.

needextend

needextend must be restricted to Needs owned by the same bundle (and,
if necessary, the same document). Applying it to imported upward Needs would
make an interface mutable and invalidate the ownership model.

Dynamic functions, variants, and schema selects

Bundle-local processing must not depend on Needs outside the local-plus-
upward closure. Phase 1 must inventory dynamic functions, variants,
constraints, link conditions, and docname/is_external/is_import
dependent schema selects. Unsupported graph-dependent behavior must fail
clearly rather than produce an export with different semantics.

Document references

Document references (:doc:, :ref:, includes, images) remain bundle-local
and host-to-bundle only. A bundle linter is required to make document-reference
self-containment enforceable.

Foreign mounted bundles

The final composition must distinguish a Need that is local for rendered URL
purposes from one that remains foreign for gates and metrics. One
is_external flag cannot safely represent both meanings; the required
Sphinx-Needs upstream API must expose this distinction or preserve equivalent
metadata.

Validation model

Bundle-export builds make fatal:

• duplicate IDs in the local-plus-upward closure;
• missing or disallowed Need links;
• invalid link conditions and local constraints; and
• unsupported graph-dependent features.

The top-level composition makes fatal:

• duplicate IDs across all local exports;
• invariants that require the complete graph;
• backlink and dead-link calculation failures;
• SCORE graph checks; and
• Needs-based metrics and gates.

Suggested implementation phases

1. Define the hierarchy contract

   • Specify the upward_bundles declaration and its transitive closure.
   • Inventory existing cross-bundle links to assess migration feasibility.
   • Decide whether multiple upward dependencies are allowed.

2. Establish prerequisites

   • Enforce bundle-local needextend as fatal.
   • Add a bundle linter for document-reference self-containment.
   • Characterize dynamic functions, variants, constraints, link conditions,
     Need parts, and schema selects.

3. Prototype upstream integration

   • Specify and propose the supported Sphinx-Needs re-hosting API required
     for final composition.
   • Verify external-needs import/export behavior, ownership filtering, and
     unique base_url handling.

4. Build and validate bundle exports

   • Add metamodel to docs_bundle() and default it to the built-in model.
   • Make every source-bearing bundle expose its own Needs-JSON target.
   • Import declared upward exports and make unresolved links fatal.
   • Export only locally owned Needs.
   • Verify the intended cache boundaries.

5. Compose and prove equivalence

   • Produce the final graph with normal Sphinx-Needs post-processing.
   • Compare diagnostics and output with the monolithic build.
   • Cover duplicate IDs, invalid directions, missing links, constraints,
     Need parts, source links, metrics, and foreign mounted bundles.

6. Migrate consumers

   • Make the composed graph the public needs_json.
   • Move metrics_json and traceability_gate after equivalence is proven.
   • Keep docs_check as the comprehensive integration check.

Alternatives considered

Fully independent exports with final-only link validation

This is the direction in issue #679. It maximizes bundle-export reuse but delays
cross-bundle link feedback until the global composition.

Arbitrary explicit bundle dependencies

This validates links early but permits a graph that can become cyclic and
does not reflect the architectural traceability direction. Restricting
dependencies to upward edges makes the graph understandable and analyzable.

Raw pre-post-processing exports

This may be necessary if finalized Sphinx-Needs exports cannot support the
required upstream re-hosting operation. It needs a supported serialization
boundary upstream and does not remove the need for the hierarchy contract.

Future improvement: resolve metamodels in docs()

This proposal intentionally starts with a metamodel declared on every
docs_bundle(). It gives each bundle a directly buildable Needs-JSON target
with clear ownership and avoids the additional target-generation and
configuration-propagation machinery required for a docs()-owned model.

After the bundle-export model is established, we may move metamodel resolution
to docs(). docs(metamodel = ...) would then create bundle-specific
Needs-JSON actions with its resolved model, and metamodel would be removed
from docs_bundle().

That later design would make the same bundle usable in different docs()
contexts with different metamodels. This is a useful side effect, not a goal
of the initial work. It also needs independently buildable generated targets
for subbundles and explicit model propagation. It is out of scope for this
proposal.

Decision requested

Do we agree to evaluate this constrained model?

1. A bundle may link only locally or to explicitly declared upward bundles.
2. Declared upward exports are Bazel inputs and make cross-bundle links
   fatal during bundle-export builds.
3. Each docs_bundle() declares the metamodel for its own Needs-JSON target;
   projects with a custom model wrap both docs_bundle() and docs().
4. Bundle exports include only their owned Needs; they are internal artifacts.
5. The top-level composition finalizes the complete graph and preserves
   public output compatibility.
6. A supported Sphinx-Needs re-hosting API, bundle-local needextend, and
   document-reference self-containment are prerequisites.