Add A Schema-Aware Standard Fortran Namelist Parser

[MuellerSeb]

Replace the schema-less f90nml parsing path used by nml-tools validate with a
record-aware parser and schema-aware evaluator designed around standard Fortran
namelist semantics.

The parser should preserve groups, assignments, designators, values, nulls,
repetition, record boundaries, and source locations without immediately
guessing a nested Python container shape. A separate evaluator should use the
resolved nml-tools schemas to determine ranks, shapes, scalar types, string
lengths, and derived component order.

For this feature, "all valid namelist" means all standard intrinsic namelist
syntax that can be interpreted using the nml-tools schema model. The grammar
should be complete from the start, while semantic support for complex values,
nested derived types, and component arrays can follow the corresponding schema
features.

Arbitrary user-defined formatted I/O is not generically reproducible in
Python and remains explicitly outside this conformance target.

Motivation

f90nml has to infer Python containers without knowing the declared Fortran
objects. Several standard namelist forms are therefore inherently ambiguous to
it:

settings(1) = .true., 1
settings(:)%flag = .true., .false.
settings%flag = .true., .false.
label(2:4) = "abc"

With a schema, nml-tools knows whether settings is a scalar or array, whether
flag is a scalar or array component, and whether label(2:4) is an array
section or character substring. It can evaluate these assignments directly
instead of reconstructing their meaning from an already collapsed mapping.

nml-tools also has a narrower storage model than a general Fortran parser:
generated and schema-described arrays have lower bound one, their shapes are
known from constants and runtime dimensions, and generated derived component
order follows resolved schema properties order. Imported derived layout order
is already an application contract when positional namelist input is used.

Verified Behaviors

• settings(1) = .true., 1 retains .true. as the indexed value and warns
  that 1 was not assigned to a variable and was removed.
• settings(:)%flag = .true., .false. raises an internal TypeError while
  processing the open section bounds.
• settings(:)%flag(2) = .true., .false. fails through the same path.
• settings%flag = .true., .false. becomes a mapping equivalent to
  {"settings": {"flag": [true, false]}}. Without declarations, that result
  cannot distinguish a component array from a scalar component selected across
  an array of derived values.
• label(2:4) = "abc" is materialized as indexed array data with
  _start_index bookkeeping, although the same syntax can designate a
  substring of a scalar character object.
• Indexed and sectioned arrays are eagerly materialized as nested lists plus
  _start_index metadata. nml-tools currently removes that metadata before
  validation, losing assignment-level information.
• Variables and overlapping assignments are collapsed into their inferred
  containers instead of being retained as an ordered operation stream.

These are not isolated conversion bugs. The parser lacks the declaration
information required to choose a unique container representation.

Draft f90nml PR #184
explores derived-array parsing but encounters the same ambiguity between
intrinsic arrays, arrays of derived values, and array-valued components. It is
useful research but is not a sufficient base for schema-aware validation.

f90nml remains useful as a temporary regression oracle for syntax already
accepted by nml-tools. It should not be vendored or patched as part of this
feature.

Standard Conformance Boundary

Use clauses 8.9, 12.6.3, 12.6.4, 13.10, and 13.11 of the
Fortran 2023 interpretation document
as the normative basis for the parser and evaluator.

The target is standard formatted namelist input under a declared external-unit
decimal mode. It includes intrinsic effective-item expansion for derived types
but excludes arbitrary derived-type defined formatted I/O.

In particular:

• A user-defined assignment(=) operator is not invoked by namelist input and
  does not alter positional component expansion.
• A derived type can instead define formatted input. For a namelist transfer,
  its procedure receives iotype="NAMELIST" and may implement arbitrary syntax
  and behavior. A generic Python parser cannot execute that application code.
• Processor extensions and processor-specific acceptance are not part of
  strict mode, even when a supported compiler happens to accept them.
• Nondefault lower bounds are valid Fortran but are outside the current
  nml-tools schema model. The evaluator uses lower bound one for every
  schema-described dimension.
• Multiple namelist groups in one physical file are an nml-tools container
  convention. The parser should preserve them in source order while the CLI
  retains its existing unknown-group and duplicate-group policies.

Architecture

Record-Aware Lexer

Add a private lexer that consumes text records and emits tokens with source
spans. It must retain end-of-record information where the standard gives record
boundaries meaning, especially for comments and character input.

The lexer should recognize syntax, not choose Python value types. Numeric,
logical, and undelimited word-like tokens should retain their original text
until a schema-resolved target determines how they are interpreted.

Lossless Parser IR

Represent the source as ordered groups and assignments. A suitable conceptual
model is:

ParsedFile
  groups: list[ParsedGroup]

ParsedGroup
  name
  assignments: list[Assignment]
  source_span

Assignment
  designator
  values: list[Value | NullValue | RepeatedValue]
  source_span

A designator should preserve its part names and parenthesized selector groups
without deciding whether a selector is an array subscript list or substring.
That distinction belongs in schema-aware evaluation.

The IR must preserve duplicate groups, repeated variables, overlapping
sections, nulls, and source spelling. It must not use nested lists or mappings
as its primary representation.

Schema-Aware Evaluator

Resolve each group and designator against a resolved schema, constants, and
runtime dimensions. Expand the selected object into ordered scalar effective
items, then consume the assignment values from left to right.

Maintain explicit state for each scalar leaf:

• its current effective value, if any;
• whether it is concretely initialized by a default;
• whether the namelist explicitly assigned it;
• whether a consumed null left it unchanged; and
• its source assignment for diagnostics.

This state avoids overloading Python None for absent input, a namelist null,
an unset generated sentinel, and a future nullable schema value. It also
provides the foundation needed by the derived default/required presence policy.

After all assignments have been applied serially, materialize or directly
validate the resulting values with the existing schema constraints. Explicit
input overrides defaults, null input leaves the prior state unchanged, and a
later overlapping assignment overrides an earlier one.

Complete Feature Checklist

Groups, Records, And Comments

• Parse a standard & immediately followed by a case-insensitive group
  name.
• Require the standard separation between the group name and its contents.
• Parse zero or more assignments followed by /.
• Preserve blank records and record boundaries where semantically relevant.
• Recognize standard ! comments only in positions permitted by namelist
  input.
• Ensure / inside a comment or delimited character value does not end the
  group.
• Treat line endings as separators outside character values without
  inventing a null value at end of record.
• Preserve multiple groups and duplicate group occurrences in source order.
• Match group and object names case-insensitively while retaining source
  spelling for diagnostics.
• Diagnose unterminated groups, unexpected text, malformed assignments, and
  invalid group identifiers with line and column information.

Object Designators

• Preserve a whole scalar or array object designator.
• Parse scalar integer subscripts.
• Parse multidimensional subscript lists.
• Parse section triplets lower:upper:stride with any permitted omitted
  bound or stride.
• Support positive and negative section strides and reject a zero stride.
• Resolve omitted section bounds from the schema-described one-based shape
  and stride direction.
• Parse arbitrary % component chains in the IR, even while the current
  schema evaluator limits derived values to one level.
• Parse a character substring range separately from its optional array
  selector, including forms such as labels(1)(2:4).
• Recognize complex part designators %re and %im when the resolved target
  is complex.
• Reject vector subscripts, image selectors, expressions, names, and kind
  suffixes in namelist subscripts and bounds.
• Reject a subscript count that does not match the resolved rank.
• Reject out-of-range scalar subscripts and section elements.
• Reject zero-sized arrays, zero-sized sections, and zero-length substring
  designators where the standard prohibits them.
• Enforce the data-reference rule that at most one part reference has
  nonzero rank.

The evaluator should consequently distinguish these cases:

settings(:)%flag       ! Valid when flag is scalar.
settings(:)%flag(2)    ! Valid when flag is an array and (2) selects one element.
settings(:)%flag(:)    ! Invalid: two part references have nonzero rank.
settings%flag          ! Meaning follows the declared ranks of both parts.

Values And Separators

• Preserve a scalar literal as raw source text until its target effective
  item is known.
• Parse leading, trailing, and repeated value separators as standard null
  values where applicable.
• Parse r* as r null values.
• Parse r*value as repetition of one source value.
• Require a positive unsigned repeat count without a kind suffix.
• Support blank-separated values.
• Support comma separators in POINT decimal mode.
• Support semicolon separators in COMMA decimal mode.
• Keep the decimal mode explicit because comma has different lexical roles
  in the two modes.
• Stop value parsing at the next syntactic designator followed by = or at
  the group terminator without relying on an inferred Python container length.
• Reject a value sequence longer than the selected effective-item sequence.
• Leave unmentioned trailing effective items unchanged when fewer values
  are supplied.

Intrinsic Values

• Parse signed and unsigned integer input without kind suffixes.
• Parse real input with leading or trailing decimal points, exponent-letter
  forms, and exponent forms without an explicit exponent letter where the
  standard permits them.
• Parse the decimal symbol according to POINT or COMMA mode.
• Cover standard IEEE infinity, NaN, and hexadecimal-significand input forms
  where the target processor and nml-tools value model support them.
• Reject nonstandard real spellings in strict mode even if Python or one
  compiler accepts them.
• Follow standard logical input rules, including permitted values beginning
  with T or F, instead of f90nml's narrower default strict_logical list.
• Parse complex input as a parenthesized pair using the separator associated
  with the decimal mode.
• Treat one null as applying to a complete complex effective item rather
  than to one part of the pair.
• Require apostrophe- or quote-delimited character values in strict mode.
• Support doubled matching delimiters inside character values.
• Support empty character values and embedded blanks, separators, comment
  markers, and slashes.
• Support standard character continuation across input records without
  inserting an extra character.
• Apply Fortran character assignment padding and truncation using the
  schema-described storage length.
• Reject kind suffixes on all namelist input constants.

Effective-Item Expansion

• Expand an intrinsic array in Fortran array-element order, with the first
  subscript varying fastest.
• Expand a selected section in the order defined by its subscript triplets.
• Expand an intrinsic derived value in declaration/component order when it
  does not use defined formatted I/O.
• Recursively expand an array of derived values by array element first and
  component order second.
• Use resolved schema properties order for generated derived types.
• Treat schema order versus declaration order as an explicit caller
  contract for imported derived types using positional input.
• Apply the same expansion to whole objects, indexed elements, sections,
  component selections, and nested component paths.
• Consume each repeated literal against each target item separately so a
  repetition crossing heterogeneous components still receives type checking.
• Permit a partial value sequence and leave its unconsumed tail unchanged.
• Consume nulls without changing the selected effective item.
• Reject excess values instead of warning and discarding them.
• Apply assignments serially and let later assignments win for overlapping
  effective items.

Current nml-tools Schema Evaluation

• Evaluate integer, number, logical, and fixed-length character scalars.
• Evaluate fixed-shape, runtime-sized, multidimensional, sparse optional,
  and supported flexible-tail intrinsic arrays.
• Evaluate whole-array buffers, scalar element assignments, and array
  sections using one-based schema bounds.
• Evaluate simple scalar derived values with intrinsic scalar components.
• Evaluate arrays of simple derived values at any configured rank.
• Support component notation and positional whole-derived assignment.
• Support indexed positional derived assignment such as
  settings(1) = .true., 1 without concatenating adjacent array elements.
• Support whole derived-array and section buffers in effective-item order.
• Resolve $ref types before choosing component order and constraints.
• Apply existing enums, numeric bounds, string lengths, requiredness, shape,
  and unknown-property checks to explicitly assigned values.
• Retain enough presence information for defaults and required checks to be
  updated by the separate derived presence/default policy.

Future Schema-Backed Evaluation

• Add complex scalar and array schema support before accepting complex
  values semantically.
• Add character substring evaluation, including merging a substring into
  the initialized parent value.
• Add nested derived values using the same recursive effective-item model.
• Add intrinsic array components and component sections.
• Add derived components that are arrays where the one-nonzero-rank-part
  designator rule is satisfied.
• Add arrays of nested derived values without changing the parser IR.
• Diagnose syntactically valid but currently unsupported schema categories
  as capability errors rather than syntax errors.

Diagnostics

• Report the input file, line, and column for lexical and syntactic errors.
• Include the group name and original designator for evaluation errors.
• Include the resolved schema path or derived component path when known.
• Distinguish malformed syntax, unknown schema objects, unsupported standard
  features, bounds errors, type conversion failures, and constraint failures.
• Diagnose overlong buffers at the assignment that supplied the excess
  value.
• Diagnose every prohibited f90nml extension clearly in strict mode and
  mention the compatibility option where applicable.

Compatibility Mode

Standard syntax should be the default. Add an explicit
--allow-f90nml-extensions option to nml-tools validate for users migrating
existing files.

Compatibility mode may accept:

• $group group openers;
• $end and &end terminators;
• # comments where f90nml currently permits them; and
• undelimited character values where a resolved character target makes the
  intent unambiguous.

Compatibility mode should change accepted syntax only. It must not recreate
f90nml's schema-less list nesting, _start_index representation, discarded
values, or derived-array guesses.

The private parser should accept an explicit POINT or COMMA decimal-mode
parameter and default to POINT. CLI validation should remain POINT by default
to match generated from_file(...) readers. Exposing COMMA mode for generated
files requires generation and validation to select the same external-unit mode.

Implementation Roadmap

Phase 1: Lexer And Lossless Parser

• Add a private record-aware parser module with typed, source-located IR nodes.
• Cover the complete standard group, designator, value, null, repeat, comment,
  and separator grammar.
• Keep the production CLI on f90nml during this phase.
• Add syntax tests independently of schema evaluation.

Phase 2: Current-Schema Evaluator

• Resolve parser designators against loaded schemas, constants, and dimensions.
• Implement effective-item selection, expansion, value conversion, assignment
  order, null behavior, and presence state for all currently supported schema
  types.
• Feed the evaluated state into existing validation constraints without first
  forcing it through f90nml-shaped containers.
• Add focused comparisons with existing valid CLI input and regressions for the
  audited f90nml failures.

Phase 3: CLI Migration

• Make the schema-aware parser the implementation of nml-tools validate.
• Add --allow-f90nml-extensions for the documented migration syntax.
• Preserve file-profile handling and existing missing, unknown, and duplicate
  group policies.
• Keep a temporary differential test corpus against f90nml for overlapping
  supported syntax.
• Remove the runtime f90nml dependency, import, _normalize_f90nml_values, and
  f90nml-specific documentation after parity is established.

Phase 4: Extended Schema Semantics

• Connect complex input when complex schemas and generators are added.
• Connect substrings, nested derived values, and component arrays when those
  schema features are introduced.
• Reuse the same parser and effective-item evaluator rather than adding
  feature-specific string rewrites.

Phase 5: Stable Public API

• Keep lexer and IR modules private while their representation evolves.
• After CLI and TUI use has stabilized them, expose a high-level public API for
  parsing and validating a namelist file.
• Do not initially expose low-level token or mutable IR compatibility promises.

Test Plan

Pure Python Tests

• Add table-driven lexer and parser corpora for every standard syntax category
  in the checklist.
• Test both accepted and rejected subscript, section, substring, component, and
  complex-part designators.
• Test comments and strings across records so /, !, commas, and semicolons
  are interpreted in the correct lexical context.
• Test POINT and COMMA modes separately.
• Test null and repeat expansion, including nulls at the start, middle, and end
  of value sequences.
• Test ordered overlapping assignments and duplicate designators.
• Test source locations and diagnostic categories.

Schema Evaluation Tests

• Cover scalar intrinsic types and all supported array shape variants.
• Cover multidimensional Fortran order, positive and negative sections, omitted
  bounds, and scalar subscripts.
• Cover scalar, indexed, sectioned, and whole-array derived buffers.
• Cover component notation and positional notation in the same group.
• Cover defaults, optional values, required values, sparse input, and nulls
  without conflating their presence states.
• Cover schema errors separately from syntactically valid but unsupported
  standard features.

Regression And Compiler Tests

• Add regressions for every verified f90nml failure and ambiguity listed above.
• Preserve all current CLI, file-profile, derived-buffer, and multidimensional
  array validation behavior.
• Parse the existing generated templates and handwritten example namelists.
• Add a standard conformance fixture corpus that is also read by intrinsic
  Fortran namelist I/O under the existing gfortran, ifx, and flang-new CI matrix.
• Compare effective values for accepted standard fixtures rather than treating
  compiler acceptance of extensions as proof of standard conformance.
• Test strict rejection and opt-in compatibility behavior independently.

Acceptance Criteria

• A lossless parser IR represents standard namelist assignments without schema-
  less Python container guesses.
• Current nml-tools schemas can evaluate all standard namelist forms applicable
  to their supported intrinsic and simple-derived types.
• settings(1) = .true., 1 assigns one complete derived array element.
• Component sections and character substrings are resolved from schema type and
  rank rather than f90nml heuristics.
• Nulls, repeats, partial assignments, and overlapping assignments follow
  standard effective-item behavior.
• Arrays use one-based nml-tools bounds and Fortran element order.
• Strict mode rejects known extensions; compatibility mode accepts only its
  documented syntax additions.
• Errors identify the source location, designator, and semantic reason.
• CLI migration preserves current schemas, file profiles, and generated input
  workflows.
• f90nml can be removed as a runtime dependency after migration parity.
• DTIO and unsupported schema categories are reported as explicit capability
  boundaries.

Out Of Scope

• Executing arbitrary user-defined formatted I/O procedures.
• Giving defined assignment operators a role in namelist transfer.
• Reproducing processor-specific extensions in strict mode.
• Supporting nondefault array lower bounds before the schema model can describe
  them.
• Making the lexer or parser IR public in the first implementation.
• Copying, vendoring, or modifying f90nml