Memoize HermitCrab's sequential analysis cascade

[johnml1135]

Summary

• Ports a memoization scheme (from an archived prototype) for HermitCrab's Unordered-mode analysis cascade: repeated cascade states reached via different rule-unapplication orders are computed once and replayed, instead of re-expanded — this is the source of a well-known ~98% expansion redundancy on template-heavy (Bantu-style) grammars.
• Off by default. The existing parallel cascade is completely unchanged; you opt in with new Morpher(traceManager, language, maxDegreeOfParallelism: 1), which selects the new sequential+memo path.
• Acceptance criterion is analysis-set equality (canonical morpheme-signature sets), not byte-identical objects — a memo-replayed Word isn't guaranteed field-for-field identical to a freshly-computed one, even when it represents the same analysis.

See memoization.md (added in this PR) for the full design, the key-completeness audit, and the verification methodology.

Follow-up: #457 stacks a data-structure rearchitecture (array-backed, copy-on-write Shape) on top of this branch and re-measures the same two heavy words — read that PR for whether it's worth the larger diff.

Why off by default — the actual tradeoff

This is not a pure win, and the corpus evidence says so directly:

• On a known-pathological heavy word (H1), sequential+memo measured 6.2x faster than today's parallel default (isolated to ~6.3x attributable to the memo mechanism itself, via a mutation test separating memo-on/off from sequential/parallel — parallelism alone contributed only ~2% on this word).
• On typical (non-template) grammars — Indonesian was tested directly, 3 repeated runs to average out noise — sequential+memo runs ~25-28% slower in aggregate than the parallel default: it loses parallelism, and the memo has little order-variant redundancy to collapse there. Forcing the memo off entirely (mutation-tested) widens the slowdown to ~25-47%, confirming most — not all — of this is the lost thread, not the memo itself.
• On a 70-word Sena slice including one heavy word: 53/69 (77%) of words are individually faster under the memo, 16/69 (23%) slower — but in aggregate wall-clock, the corpus finishes 2.42x faster, because the heavy word dominates the sum. Both of these are true at once and don't contradict each other.

So flipping the library's default cascade mode trades typical-word latency for pathological-word latency. Whether that's the right tradeoff depends on a given corpus's word-difficulty distribution — a decision for a future PR with its own evidence, not this one.

Verification

• Unit tests: AnalysisStateKey order-invariance/hash/equality, Word.ReplayOnto graft correctness (including a test that specifically distinguishes "grafted the right subtree" from "grafted the wrong one" via distinct lexical entries), the InProgress in-flight re-entry guard, positive-replay-vs-unmemoized-result-set equivalence (including trail order), and an mrule/template equivalence battery against real analysis-rule content via MorpherTests.
• Corpus runs (local, uncommitted grammars — Sena, Indonesian, Amharic; this repo never commits real grammar/corpus data): zero analysis-set divergences across all three, with both memo tables exercised non-vacuously (hundreds of thousands of hits, not a no-op) — but this excludes the heaviest words, which timed out and never entered the comparison (60 of 312 in the full Sena run). That's exactly the class of word the memo targets, so it's a real gap, not a formality: soundness on the hard cases rests on the two heavy words checked individually below, not on the corpus aggregate.
• The two heavy words were checked individually to close that gap: H1 (the word above) is 0 divergences on a positive, 2-analysis match. H2 did not complete within a 400s timeout even with the memo on — an inconclusive (not failing) data point, documented in memoization.md rather than glossed over. (Stack RUSTIFY's array/COW rearchitecture on top of memoization #457 revisits H2 with the array-backed rearchitecture and gets a very different result.)
• Full test suite: 78/78 HermitCrab, 803/806 SIL.Machine (3 pre-existing unrelated skips), 83/83 Thot — all green.

Test plan

• dotnet build Machine.sln
• dotnet csharpier check .
• dotnet test across HermitCrab, SIL.Machine, and Thot test projects
• Corpus verification harness (MemoCorpusVerification, [Explicit]) run manually against local Sena/Indonesian/Amharic grammars — 0 divergences on the words that completed within the timeout budget

---

This change is 

[codecov-commenter]

Codecov Report

❌ Patch coverage is 92.22798% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.47%. Comparing base (0b58830) to head (70bff72).

Files with missing lines	Patch %	Lines
....Machine.Morphology.HermitCrab/AnalysisStateKey.cs	73.07%	6 Missing and 8 partials ⚠️
src/SIL.Machine.Morphology.HermitCrab/Morpher.cs	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #456      +/-   ##
==========================================
+ Coverage   73.33%   73.47%   +0.14%     
==========================================
  Files         445      448       +3     
  Lines       37317    37503     +186     
  Branches     5118     5148      +30     
==========================================
+ Hits        27367    27556     +189     
+ Misses       8825     8815      -10     
- Partials     1125     1132       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ddaspit]

@ddaspit made 1 comment.
Reviewable status: 0 of 11 files reviewed, 1 unresolved discussion (waiting on johnml1135).

---

a discussion (no related file):
Before I do a full review, it would be great if you could rebase this on master. I suspect that some of the logic might need to be updated to work with the changes in PR #452.

[johnml1135]

Rebased this branch onto current master (0b588308, v3.9.2).

Decisions made during the rebase:

• Kept the original memoization patch semantically unchanged; git range-diff maps the original commit exactly to rebased commit 9cbbcfb9.
• Adapted the memoized sequential cascade to PR Support LT-22605: Add ability to limit HermitCrab parses #452's MaxAlternatives contract. The memoized cascade remains a RuleCascade, propagates the configured limit, and now checks the distinct output count on both normal expansion and memo-replay paths.
• Added a regression test that exercises both paths. This avoids a case where maxDegreeOfParallelism: 1 could bypass the parse limit while the default parallel cascade enforced it.
• Kept memoization opt-in and made no change to the existing parallel/default behavior.
• Used an explicit RuleCascade<Word, ShapeNode> cast so the conditional expression remains compatible with the repository's C# language version.

Verification on final commit 70bff728:

• dotnet build Machine.sln --nologo — succeeded
• dotnet test Machine.sln --no-build --nologo — 974 passed, 0 failed, 4 skipped/explicit
• dotnet csharpier check . — 702 files checked