LT-22711: Fix affix process rule corruption when splitting a sense into a new entry - #398
Open
johnml1135 wants to merge 9 commits into
Open
LT-22711: Fix affix process rule corruption when splitting a sense into a new entry#398johnml1135 wants to merge 9 commits into
johnml1135 wants to merge 9 commits into
Conversation
…seToCopy Analysis of MoAffixProcess.PostClone's three known defects (A: return instead of continue, B: repeated RemoveAt(0) across allomorphs, C: no scoping to the object's own clone), with a proposed fix and test plan.
Five tests from the bug report's test plan, all exercising LexEntry.MoveSenseToCopy's clone of an affix-process allomorph via the generic CopyObject reflection path and MoAffixProcess.PostClone. Four of five currently fail against unfixed PostClone: - single non-trivial allomorph: OutputOS ends up short by one real entry (a previously undocumented interaction: removing the default InputOS[0] cascades via RemoveObjectSideEffectsInternal to also remove the default OutputOS[0], and PostClone's own unconditional RemoveAt(0) then strips a second, real, OutputOS entry) - stem allomorph before the affix process (defect A): the process's defaults are never stripped at all - two affix-process allomorphs (defect B): the first allomorph's clone is over-stripped, the second's defaults are never touched - save/reload round trip: the corruption is already present in-memory before saving, and persists unchanged through reload (no separate "right then wrong later" mechanism was found; the clone is simply wrong from the moment of cloning) The ContentRA-remapping test (case 4) passes: CopyObject's reference pass already targets each clone's own InputOS correctly, independent of the PostClone defects.
PostClone was iterating the entire shared copyMap (which, when several allomorphs are cloned together via CopyObject, contains every clone from every allomorph) and returning on the first entry that wasn't an IMoAffixProcess (defect A), and re-stripping every affix-process clone it found on every invocation instead of just its own (defect B). Fix: look up this object's own clone via copyMap[Hvo] and repair only that. Also fixes a previously undocumented interaction: removing the default InputOS[0] can cascade, via RemoveObjectSideEffectsInternal, into also removing the matching default OutputOS[0]. The old code's unconditional OutputOS.RemoveAt(0) would then strip a second, real, entry. The fix captures references to the two specific default objects before removing anything, and only removes the default output if it wasn't already cascade-removed. All 5 reproduction tests (4 in-memory + 1 save/reload round trip) now pass, along with the pre-existing AffixProcessesRemainUnchangedWhenSenseMovedToNewEntry.
Covers: the copyMap-shaped PostClone API and why it invited this bug, what SetDefaultValuesAfterInit's seed-then-strip seam should really be, confirms no other PostClone implementation has the same defect, evaluates (but does not implement) a full SetCloneProperties rewrite, and lists what still needs verification in a running FLEx.
Mutation testing found that TwoProcessAllomorphs_NeitherLosesRealContent and ContentRAPointsIntoOwnClonesInputOS both passed against a mutant that captured index 1 instead of index 0 as the "default" to remove -- right count, wrong object removed, real content lost. Count and Contains()-based assertions can't see that. Added AssertNonTrivialRuleClonedCorrectly, checking ClassID at each InputOS/OutputOS position and reference-equality of each mapping's ContentRA to the exact clone InputOS slot it must target, and applied it to all four in-memory clone tests. Verified the index-1 mutation now fails all five (previously: two of five still passed). Also documented, rather than removed, two other mutations the reviewer found to be currently-behavioral-no-ops (IsValidObject guard on defaultOutput; Remove(defaultInput) vs RemoveAt(0)): kept both as defensive code with a one-line rationale each, since dropping them would only be justified by today's invariants, not any test.
MoveSenseToCopy reaches MoAffixProcess a second, independent time through CreateMatchingAllomorphInTargetEntry (OverridesLing_Lex.cs:1803), called from UpdateReferencesForSenseMove, whenever a WfiMorphBundle references the moved sense's morph. An affix process's Form is normally left blank (Form's own doc comment says it's undefined for process affixes), which makes IsMatchingAllomorph fail to match the already-cloned LexemeFormOA and forces this second, independent CopyObject<IMoForm> clone -- a path neither the original bug report nor the four existing reproduction tests touch. Verified red at the pre-fix PostClone body (OutputOS Expected 2, But was 1 -- same shape as the single-allomorph case) and green with the current fix, confirming the Hvo-scoped PostClone generalizes correctly to this path without further changes. Wrapped the assertions in try/finally with an explicit ActionHandlerAccessor.Commit() to sidestep a separate, pre-existing bug (reproduces identically before and after this fix): undoing this test's WfiMorphBundle reference changes throws KeyNotFoundException out of LcmAtomicRefPropertyChanged.Undo() during TestTearDown's UndoAll(). Not fixed here -- out of scope and pre-existing.
The Count > 1 guard from the previous fix could not distinguish a genuinely empty affix process (0 real inputs/outputs, legal at the LCM level -- MoAffixProcess has no IsFieldRequired guard) from one leaked default: both look like clone count 1 after a naive single strip. Reviewer's probe demonstrated this: a source with InputOS/OutputOS Cleared to empty still ended up with a leaked default PhVariable/ MoCopyFromInput pair after MoveSenseToCopy (source 0, clone 1). PostClone now compares the clone's counts against THIS (the source object it belongs to, always in hand) and removes exactly the surplus leading items, computed from that difference rather than guessed from the clone's shape. The output surplus is recomputed after the input removal, not assumed independent, since removing the default input can itself cascade (via RemoveObjectSideEffectsInternal) into removing the default output. Verified red against the Count > 1 fix (Expected 0, But was 1) and green against this fix; added the zero-content case as a permanent regression test. Full 4-defect + round-trip + failover-path test set (8 tests) still green.
Records the mutation-testing round: identity-at-position gap in two tests (fixed), the confirmed-but-previously-untested WfiMorphBundle failover clone path, the residual zero-content bug and its categorical fix, the two mutations kept as documented defensive code, and the reviewer's Class-A sweep / disproved copy-map worry noted as follow-ups requiring no action here.
The bug analysis and architecture self-review were working documents for this fix. Their conclusions are now carried by the code and its tests; the reasoning, decisions and paths not taken live in the pull request body so they inform review without merging into the tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Splitting a sense into a new entry no longer corrupts the affix process rule on the copy.
MoAffixProcess.PostClonewas leaving stray default items on the clone, deleting real ones, or both, depending on how many allomorphs the entry had and what order they were in.The reviewer's question on a nine-line production diff is why it needs seven tests. Because the old code was wrong in five distinct ways that masked each other, and because three of them only appear in specific entry shapes — a stem allomorph ordered before the affix process, two affix processes on one entry, an entry reached through the
WfiMorphBundlefailover path. A single happy-path test passes against the broken code. The review worth your time is the replacement rule, not the deletions.Where to look
thisis the source) and removes exactly the surplus. Every previous version guessed from the clone's shape, and every count heuristic has a case it cannot distinguish.RemoveObjectSideEffectsInternalinto removing the default output. Assuming two independent removals is what deleted real data.PostCloneis called once per top-level source object but receives the copy map for the whole pass, including every sibling allomorph's clones. Scoping tocopyMap[Hvo]is the core of the fix.MoveSenseToCopyreachesMoAffixProcessthrough three call sites, not the two that are obvious; the third (CreateMatchingAllomorphInTargetEntry) fires in the normal case, when a process affix'sFormis blank.MoAffixProcessstill does not implementICloneableCmObject; it remains on the generic reflection clone path.Deliberately not here
MoAffixProcess.SetCloneProperties— would remove the seed-then-strip dance entirely, but needs hand-rolledContentRAremapping thatPhRegularRuledoes not, and landing it untested is worse than this fix.PostClone(Dictionary<int, ICmObject>)to the object's own clone. Breaking public API; deserves its own decision.PhTerminalUnit.CodesOSandMoStemName.RegionsOCshare the vulnerable shape but are unreachable from any clone call site today.Verification. Full
SIL.LCModel.Tests1716 passed / 0 failed / 18 skipped on net8.0. Against the test-only commit, 4 of 5 reproduction tests fail. net462 was not built or run — a pre-existingIdlImptoolchain issue in this environment, unrelated to this change; CI will be first to exercise it. Requires a package bump in FieldWorks to ship. Fixes LT-22711.Reading this a year from now — start here
The investigation notes and architecture review were deliberately deleted from the branch rather than merged; their content is synthesised below. They were working documents for a one-time analysis.
The short version:
MoAffixProcess.SetDefaultValuesAfterInitseeds every new instance with a defaultPhVariableinput andMoCopyFromInputoutput — needed so the affix process slice works (FWR-1619).CopyObjectbuilds clones through the normal factory, so a clone is born holding those defaults, and the real cloned content is appended after them.PostCloneexists solely to strip the seeds back off. It was doing that wrong.The five defects, and how they masked each other
returnwherecontinuewas meant. The loop walked the entire shared copy map and bailed at the first value that was not anIMoAffixProcess. Unless the affix process happened to sort first, the cleanup never ran at all — leaving a leading default pair, which renders as an untouched, freshly-created rule.PostClonefires once per top-level source object but iterated all clones, so with two affix-process allomorphs the second invocation stripped the first one's already-repaired clone, deleting real content.InputOS[0]triggersRemoveObjectSideEffectsInternal, which also removes the matching defaultOutputOS[0]. The unconditional secondRemoveAt(0)then deleted a real, surviving output. This one bites even the single-allomorph case, which the original analysis had assumed was safe.InputOS/OutputOSstill leaked a default pair: aCount > 1guard cannot tell "genuinely empty" from "one leaked default", since both read as count 1 once the fresh default is seeded.Defect 4 was found by writing the tests; defect 5 by adversarial review of the first fix.
Decisions, and why
Compare against the source, not the clone.
PostCloneruns on the source object, so the source's ownInputOS.Count/OutputOS.Countare in hand. The clone should end with exactly that many items; anything more is seed. This is not a heuristic and has no undistinguishable case, which is why it subsumes all five defects in one rule instead of patching them individually.Recompute the output surplus rather than deriving it. The cascade means the two removals are not independent. Measuring
OutputOS's surplus after the input removal is the difference between correct and destructive.Keep
MoAffixProcesson the reflection clone path.SetClonePropertiesshort-circuits bothCopyObjectpasses, so an implementation would have to redo theOutputOS→InputOSContentRAremapping by hand.PhRegularRulegets away with its implementation because it has no equivalent cross-list reference.Paths not taken
Narrowing the
PostClonesignature.PostClone(Dictionary<int, ICmObject>)hands every implementer the entire copy map and thereby invited defects 1–3. Narrowing it to the object's own clone would remove that footgun — but it is a breaking public API change with three implementers, and it would do nothing for the two latent classes below, where the problem is that no hook was written at all. The genuinely categorical fix is to handle seed-then-append insideCopyObjectitself.Fixing the latent classes.
PhTerminalUnit.CodesOSandMoStemName.RegionsOCboth seed owned-sequence defaults and implement neitherICloneableCmObjectnorPostClone— the same shape as this bug. Neither is reachable from any of the 21CloneLcmObject(s)call sites today, so both are latent. A future "duplicate phoneme" feature would trip it.PhRegularRuleis the model to copy.Surprising findings
The reported symptom was wrong, and this is now settled. The bug was reported as "both entries look right at first, but coming back later one has the old version" — read as a save/live-state problem. A save/reload round-trip test through the real XML backend shows the corruption is present in memory before saving; reload merely persists data that was already wrong. The reporter has since confirmed the corruption was observed before saving as well, which corroborates the test. There is no timing effect at the LCM level, and no evidence of a second live-state or cache defect above LCM — none is needed to explain the symptom.
A third clone path.
MoveSenseToCopyclonesLexemeFormOAandAlternateFormsOS— and then, viaUpdateReferencesForSenseMove→CreateMatchingAllomorphInTargetEntry, can clone the source affix process a second, independent time. It triggers when aWfiMorphBundlereferences the moved sense's morph and the morph'sFormis blank, which is the normal state for a process affix. Confirmed corrupted pre-fix and correctly repaired post-fix; now permanently tested.One documented worry was unfounded.
MoveSenseToCopyclones lexeme form and alternate forms with separateCopyObjectinstances, hence separate copy maps, which looked like it could mis-remap cross-references. Schema inspection showsMoFormand its subclasses have no reference field crossing between the two. Non-issue.Evidence
Mutation testing, run rather than argued:
RemoveAt(0)That last one matters: before the tests were strengthened, two of them asserted counts and containment only, and passed while the wrong object was being removed. They now assert identity at each position and reference-equality of each mapping's
ContentRAto its exact target slot.An unrelated pre-existing
KeyNotFoundExceptioninLcmAtomicRefPropertyChanged.Undo()surfaces during teardown of theWfiMorphBundlepath test. It reproduces identically at the pre-fix and post-fix commits, so it is not caused by this change; the test commits the action handler in afinallyso the crash cannot mask a real result.This change is