Skip to content

LT-22711: Fix affix process rule corruption when splitting a sense into a new entry - #398

Open
johnml1135 wants to merge 9 commits into
masterfrom
fix/affix-process-postclone
Open

LT-22711: Fix affix process rule corruption when splitting a sense into a new entry#398
johnml1135 wants to merge 9 commits into
masterfrom
fix/affix-process-postclone

Conversation

@johnml1135

@johnml1135 johnml1135 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Splitting a sense into a new entry no longer corrupts the affix process rule on the copy. MoAffixProcess.PostClone was leaving stray default items on the clone, deleting real ones, or both, depending on how many allomorphs the entry had and what order they were in.

The reviewer's question on a nine-line production diff is why it needs seven tests. Because the old code was wrong in five distinct ways that masked each other, and because three of them only appear in specific entry shapes — a stem allomorph ordered before the affix process, two affix processes on one entry, an entry reached through the WfiMorphBundle failover path. A single happy-path test passes against the broken code. The review worth your time is the replacement rule, not the deletions.

Where to look

  • The new rule compares the clone's counts against the source's own (this is the source) and removes exactly the surplus. Every previous version guessed from the clone's shape, and every count heuristic has a case it cannot distinguish.
  • The output surplus is recomputed after the input removal, because removing the default input cascades via RemoveObjectSideEffectsInternal into removing the default output. Assuming two independent removals is what deleted real data.
  • PostClone is called once per top-level source object but receives the copy map for the whole pass, including every sibling allomorph's clones. Scoping to copyMap[Hvo] is the core of the fix.
  • MoveSenseToCopy reaches MoAffixProcess through three call sites, not the two that are obvious; the third (CreateMatchingAllomorphInTargetEntry) fires in the normal case, when a process affix's Form is blank.
  • MoAffixProcess still does not implement ICloneableCmObject; it remains on the generic reflection clone path.

Deliberately not here

  • MoAffixProcess.SetCloneProperties — would remove the seed-then-strip dance entirely, but needs hand-rolled ContentRA remapping that PhRegularRule does not, and landing it untested is worse than this fix.
  • Narrowing PostClone(Dictionary<int, ICmObject>) to the object's own clone. Breaking public API; deserves its own decision.
  • PhTerminalUnit.CodesOS and MoStemName.RegionsOC share the vulnerable shape but are unreachable from any clone call site today.

Verification. Full SIL.LCModel.Tests 1716 passed / 0 failed / 18 skipped on net8.0. Against the test-only commit, 4 of 5 reproduction tests fail. net462 was not built or run — a pre-existing IdlImp toolchain issue in this environment, unrelated to this change; CI will be first to exercise it. Requires a package bump in FieldWorks to ship. Fixes LT-22711.


Reading this a year from now — start here

The investigation notes and architecture review were deliberately deleted from the branch rather than merged; their content is synthesised below. They were working documents for a one-time analysis.

The short version: MoAffixProcess.SetDefaultValuesAfterInit seeds every new instance with a default PhVariable input and MoCopyFromInput output — needed so the affix process slice works (FWR-1619). CopyObject builds clones through the normal factory, so a clone is born holding those defaults, and the real cloned content is appended after them. PostClone exists solely to strip the seeds back off. It was doing that wrong.

The five defects, and how they masked each other
  1. return where continue was meant. The loop walked the entire shared copy map and bailed at the first value that was not an IMoAffixProcess. Unless the affix process happened to sort first, the cleanup never ran at all — leaving a leading default pair, which renders as an untouched, freshly-created rule.
  2. Repeated stripping. PostClone fires once per top-level source object but iterated all clones, so with two affix-process allomorphs the second invocation stripped the first one's already-repaired clone, deleting real content.
  3. Wrong scope. It should have looked up its own clone rather than iterating the map at all.
  4. The cascade. Removing the default InputOS[0] triggers RemoveObjectSideEffectsInternal, which also removes the matching default OutputOS[0]. The unconditional second RemoveAt(0) then deleted a real, surviving output. This one bites even the single-allomorph case, which the original analysis had assumed was safe.
  5. Zero real content. An affix process with legitimately empty InputOS/OutputOS still leaked a default pair: a Count > 1 guard cannot tell "genuinely empty" from "one leaked default", since both read as count 1 once the fresh default is seeded.

Defect 4 was found by writing the tests; defect 5 by adversarial review of the first fix.

Decisions, and why

Compare against the source, not the clone. PostClone runs on the source object, so the source's own InputOS.Count/OutputOS.Count are in hand. The clone should end with exactly that many items; anything more is seed. This is not a heuristic and has no undistinguishable case, which is why it subsumes all five defects in one rule instead of patching them individually.

Recompute the output surplus rather than deriving it. The cascade means the two removals are not independent. Measuring OutputOS's surplus after the input removal is the difference between correct and destructive.

Keep MoAffixProcess on the reflection clone path. SetCloneProperties short-circuits both CopyObject passes, so an implementation would have to redo the OutputOSInputOS ContentRA remapping by hand. PhRegularRule gets away with its implementation because it has no equivalent cross-list reference.

Paths not taken

Narrowing the PostClone signature. PostClone(Dictionary<int, ICmObject>) hands every implementer the entire copy map and thereby invited defects 1–3. Narrowing it to the object's own clone would remove that footgun — but it is a breaking public API change with three implementers, and it would do nothing for the two latent classes below, where the problem is that no hook was written at all. The genuinely categorical fix is to handle seed-then-append inside CopyObject itself.

Fixing the latent classes. PhTerminalUnit.CodesOS and MoStemName.RegionsOC both seed owned-sequence defaults and implement neither ICloneableCmObject nor PostClone — the same shape as this bug. Neither is reachable from any of the 21 CloneLcmObject(s) call sites today, so both are latent. A future "duplicate phoneme" feature would trip it. PhRegularRule is the model to copy.

Surprising findings

The reported symptom was wrong, and this is now settled. The bug was reported as "both entries look right at first, but coming back later one has the old version" — read as a save/live-state problem. A save/reload round-trip test through the real XML backend shows the corruption is present in memory before saving; reload merely persists data that was already wrong. The reporter has since confirmed the corruption was observed before saving as well, which corroborates the test. There is no timing effect at the LCM level, and no evidence of a second live-state or cache defect above LCM — none is needed to explain the symptom.

A third clone path. MoveSenseToCopy clones LexemeFormOA and AlternateFormsOS — and then, via UpdateReferencesForSenseMoveCreateMatchingAllomorphInTargetEntry, can clone the source affix process a second, independent time. It triggers when a WfiMorphBundle references the moved sense's morph and the morph's Form is blank, which is the normal state for a process affix. Confirmed corrupted pre-fix and correctly repaired post-fix; now permanently tested.

One documented worry was unfounded. MoveSenseToCopy clones lexeme form and alternate forms with separate CopyObject instances, hence separate copy maps, which looked like it could mis-remap cross-references. Schema inspection shows MoForm and its subclasses have no reference field crossing between the two. Non-issue.

Evidence

Mutation testing, run rather than argued:

Mutation Result
Restore whole-map iteration two-allomorph test red
Restore unconditional second RemoveAt(0) 4 of 5 red
Remove index 1 instead of index 0 (wrong object, right count) all in-memory tests red

That last one matters: before the tests were strengthened, two of them asserted counts and containment only, and passed while the wrong object was being removed. They now assert identity at each position and reference-equality of each mapping's ContentRA to its exact target slot.

An unrelated pre-existing KeyNotFoundException in LcmAtomicRefPropertyChanged.Undo() surfaces during teardown of the WfiMorphBundle path test. It reproduces identically at the pre-fix and post-fix commits, so it is not caused by this change; the test commits the action handler in a finally so the crash cannot mask a real result.


This change is Reviewable

…seToCopy

Analysis of MoAffixProcess.PostClone's three known defects (A: return
instead of continue, B: repeated RemoveAt(0) across allomorphs, C: no
scoping to the object's own clone), with a proposed fix and test plan.
Five tests from the bug report's test plan, all exercising
LexEntry.MoveSenseToCopy's clone of an affix-process allomorph via the
generic CopyObject reflection path and MoAffixProcess.PostClone.

Four of five currently fail against unfixed PostClone:
- single non-trivial allomorph: OutputOS ends up short by one real
  entry (a previously undocumented interaction: removing the default
  InputOS[0] cascades via RemoveObjectSideEffectsInternal to also
  remove the default OutputOS[0], and PostClone's own unconditional
  RemoveAt(0) then strips a second, real, OutputOS entry)
- stem allomorph before the affix process (defect A): the process's
  defaults are never stripped at all
- two affix-process allomorphs (defect B): the first allomorph's
  clone is over-stripped, the second's defaults are never touched
- save/reload round trip: the corruption is already present
  in-memory before saving, and persists unchanged through reload
  (no separate "right then wrong later" mechanism was found; the
  clone is simply wrong from the moment of cloning)

The ContentRA-remapping test (case 4) passes: CopyObject's reference
pass already targets each clone's own InputOS correctly, independent
of the PostClone defects.
PostClone was iterating the entire shared copyMap (which, when several
allomorphs are cloned together via CopyObject, contains every clone
from every allomorph) and returning on the first entry that wasn't an
IMoAffixProcess (defect A), and re-stripping every affix-process clone
it found on every invocation instead of just its own (defect B). Fix:
look up this object's own clone via copyMap[Hvo] and repair only that.

Also fixes a previously undocumented interaction: removing the
default InputOS[0] can cascade, via RemoveObjectSideEffectsInternal,
into also removing the matching default OutputOS[0]. The old code's
unconditional OutputOS.RemoveAt(0) would then strip a second, real,
entry. The fix captures references to the two specific default
objects before removing anything, and only removes the default output
if it wasn't already cascade-removed.

All 5 reproduction tests (4 in-memory + 1 save/reload round trip) now
pass, along with the pre-existing AffixProcessesRemainUnchangedWhenSenseMovedToNewEntry.
Covers: the copyMap-shaped PostClone API and why it invited this bug,
what SetDefaultValuesAfterInit's seed-then-strip seam should really be,
confirms no other PostClone implementation has the same defect,
evaluates (but does not implement) a full SetCloneProperties rewrite,
and lists what still needs verification in a running FLEx.
Mutation testing found that TwoProcessAllomorphs_NeitherLosesRealContent
and ContentRAPointsIntoOwnClonesInputOS both passed against a mutant
that captured index 1 instead of index 0 as the "default" to remove --
right count, wrong object removed, real content lost. Count and
Contains()-based assertions can't see that.

Added AssertNonTrivialRuleClonedCorrectly, checking ClassID at each
InputOS/OutputOS position and reference-equality of each mapping's
ContentRA to the exact clone InputOS slot it must target, and applied
it to all four in-memory clone tests. Verified the index-1 mutation now
fails all five (previously: two of five still passed).

Also documented, rather than removed, two other mutations the reviewer
found to be currently-behavioral-no-ops (IsValidObject guard on
defaultOutput; Remove(defaultInput) vs RemoveAt(0)): kept both as
defensive code with a one-line rationale each, since dropping them
would only be justified by today's invariants, not any test.
MoveSenseToCopy reaches MoAffixProcess a second, independent time
through CreateMatchingAllomorphInTargetEntry (OverridesLing_Lex.cs:1803),
called from UpdateReferencesForSenseMove, whenever a WfiMorphBundle
references the moved sense's morph. An affix process's Form is
normally left blank (Form's own doc comment says it's undefined for
process affixes), which makes IsMatchingAllomorph fail to match the
already-cloned LexemeFormOA and forces this second, independent
CopyObject<IMoForm> clone -- a path neither the original bug report
nor the four existing reproduction tests touch.

Verified red at the pre-fix PostClone body (OutputOS Expected 2, But
was 1 -- same shape as the single-allomorph case) and green with the
current fix, confirming the Hvo-scoped PostClone generalizes correctly
to this path without further changes.

Wrapped the assertions in try/finally with an explicit
ActionHandlerAccessor.Commit() to sidestep a separate, pre-existing
bug (reproduces identically before and after this fix): undoing this
test's WfiMorphBundle reference changes throws KeyNotFoundException
out of LcmAtomicRefPropertyChanged.Undo() during TestTearDown's
UndoAll(). Not fixed here -- out of scope and pre-existing.
The Count > 1 guard from the previous fix could not distinguish a
genuinely empty affix process (0 real inputs/outputs, legal at the LCM
level -- MoAffixProcess has no IsFieldRequired guard) from one leaked
default: both look like clone count 1 after a naive single strip.
Reviewer's probe demonstrated this: a source with InputOS/OutputOS
Cleared to empty still ended up with a leaked default PhVariable/
MoCopyFromInput pair after MoveSenseToCopy (source 0, clone 1).

PostClone now compares the clone's counts against THIS (the source
object it belongs to, always in hand) and removes exactly the surplus
leading items, computed from that difference rather than guessed from
the clone's shape. The output surplus is recomputed after the input
removal, not assumed independent, since removing the default input can
itself cascade (via RemoveObjectSideEffectsInternal) into removing the
default output.

Verified red against the Count > 1 fix (Expected 0, But was 1) and
green against this fix; added the zero-content case as a permanent
regression test. Full 4-defect + round-trip + failover-path test set
(8 tests) still green.
Records the mutation-testing round: identity-at-position gap in two
tests (fixed), the confirmed-but-previously-untested WfiMorphBundle
failover clone path, the residual zero-content bug and its categorical
fix, the two mutations kept as documented defensive code, and the
reviewer's Class-A sweep / disproved copy-map worry noted as follow-ups
requiring no action here.
The bug analysis and architecture self-review were working documents for
this fix. Their conclusions are now carried by the code and its tests; the
reasoning, decisions and paths not taken live in the pull request body so
they inform review without merging into the tree.
@github-actions

Copy link
Copy Markdown

LCM Tests

    16 files  ± 0      16 suites  ±0   2m 13s ⏱️ +7s
 2 882 tests + 7   2 862 ✅ + 7   20 💤 ±0  0 ❌ ±0 
11 476 runs  +28  11 308 ✅ +28  168 💤 ±0  0 ❌ ±0 

Results for commit c6f36b3. ± Comparison against base commit 9fdb060.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant