[network] Register the Node handler on Node, not Endpoint - #1172
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @thc1006. Thanks for your PR. I'm waiting for a nephio-project member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
fbaf679 to
6e88874
Compare
The controller sets up two watches on the same type:
Watches(&invv1alpha1.Endpoint{}, &endpointEventHandler{...}).
Watches(&invv1alpha1.Endpoint{}, &nodeEventHandler{...}).
nodeEventHandler.add type-asserts its argument to *invv1alpha1.Node and
returns silently when that fails, so the second watch delivers only
Endpoints and drops every one of them. No Node change has ever enqueued
a Network. The reconciler reads Node state on each pass through
getProviderNodes, matching on the same provider and topology labels the
handler checks, so a node joining or leaving a topology is picked up only
when something else happens to trigger a reconcile.
Register the handler against Node, and add the RBAC marker for reading
them. The marker closes a gap between what the code declares and what it
does rather than changing a deployed permission: the ClusterRole in
nephio-project/catalog already grants nodes, and this repository holds no
generated role for these markers to update.
The tests cover the handler's matching rules: provider and topology both
have to match, and every matching Network is enqueued with its namespace
and name intact.
Registering the handler by hand is what allowed the mistake, so the pair
is now bound by a type parameter. nodeEventHandler is typed on
*invv1alpha1.Node and attached through source.Kind, whose object and
handler share one parameter, so pointing it at Endpoint again does not
compile:
in call to source.Kind, type handler.TypedEventHandler[*Node, ...]
does not match inferred type handler.TypedEventHandler[*Endpoint, ...]
The runtime type assertion goes with it, and a List failure is logged
rather than dropping the event in silence.
An update carries two Nodes, and listing per Node scanned the whole
Network cache twice and added each matching Network twice. Those extra
Adds only fold into one reconcile while no worker picks the first item
up in between:
q.Add(req); q.Get() -> item is processing, dirty bit cleared
q.Add(req) -> dirty bit set again
q.Done(item) -> queued a second time
A handler cannot rely on that, so the eligible topologies of both sides
are collected first and the cache is scanned once. The tests count the
scans, because a second one is invisible in the requests alone while
nothing is consuming the queue.
Collecting them also gives somewhere to say that an unset topology label
is not the empty topology, which would otherwise match any Network that
left its own topology unset, and lets an update between two Nodes of
another provider return without listing at all.
The update cases exercise the reason both sides are enqueued: a Node
moving between topologies has to wake the Network it left as well as the
one it joined, and the same holds when the provider label is gained or
lost.
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Review found the topology check conflating two different labels. A map
lookup gives "" both for a key that is absent and for one set to the
empty string, and Kubernetes allows the latter: Network.spec.topology
has no minLength, so topology: "" is a value the schema accepts. The
reconciler already treats it as one, because getProviderNodes selects
with MatchingLabels, which matches the empty value and not the missing
key:
selector topology="" matched 1: [empty-value] # not [label-absent]
So a Network with an empty topology consumed Nodes that this handler
would never wake it for. The two-value lookup distinguishes them, and
the cases cover an empty topology on its own, moving in and out of it,
and a Node deleted from it.
Delete was reached only through the shared body and never exercised.
That is the case the watch exists for as much as any: a Node leaving the
inventory has to wake its Network so the config rendered from it stops
naming a node that is gone. It is sent as a real TypedDeleteEvent now.
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
6e88874 to
14a338d
Compare
The network controller registers two watches on the same type:
nodeEventHandler.addopens with a type assertion and returns silently when it fails:Every event the second watch delivers is an
Endpoint, so the assertion has never succeeded. NoNodechange has ever enqueued aNetwork.What that costs
The reconciler reads Node state on each pass, through
getProviderNodes, filtered on exactly the labels the handler matches:So the data is consumed but never watched. A node joining or leaving a topology, or having its provider or topology label changed, is picked up only when something else — a
Networkwrite, anEndpointevent, the ownedNetworkInstance/VLANIndex/Networkresources — happens to trigger a reconcile. Between those, the rendered device config silently reflects a topology that no longer exists.The fix
Register the handler against
Node— and bind the pair by a type parameter, so the mistake cannot be made again:source.Kindtakes the watched object and the handler through one type parameter, so pointing this handler atEndpointagain stops compiling:The handler is typed on
*invv1alpha1.Nodethroughout, so the runtime type assertion that silently swallowed every event is gone, and a failedListis logged instead of dropping the event without trace. Plus the RBAC marker, which was missing for a resource the controller both lists and now watches://+kubebuilder:rbac:groups=inv.nephio.org,resources=nodes,verbs=get;list;watchOn
make manifestsI did not run it, because it cannot see this marker and there is nothing here for it to update. Three things say so:
The
manifeststarget lives inoperators/nephio-controller-managerand runscontroller-gen ... paths="./...". That module contains exactly one package:The reconcilers are a separate module (
controllers/pkg), pulled in as a blank import, so./...never reaches them.That operator has no
config/directory, so there is no generated role to refresh — running the target would create one from scratch.No checked-in YAML in this repository contains the resources the existing markers declare (
grep -rl networkinstances --include='*.yaml' .is empty), which confirms these markers have never been generated into anything here.The effective ClusterRole is hand-maintained in
nephio-project/catalogatnephio/core/nephio-operator/app/controller/clusterrole-network.yaml, and it already grantsnodes:So the marker closes a gap between what the code declares and what it needs, and no deployed permission changes. Say the word if you would rather I run the target anyway and commit whatever it produces.
Proof manifests
A passing table proves little by itself. Removing the handler's provider check fails
other provider; removing its topology check failsother topologyandevery matching network; and droppingObjectOldfrom the update path fails both migration cases, which is the guarantee those cases exist for.The package had no test file before this, so its coverage was 0%.
What the tests do and do not pin
The registration itself is guarded by the compiler rather than by a test, which is the stronger of the two:
Builder.Watchestakesclient.Objectand an untyped handler, so it accepts any pairing, and no unit test can inspect what a built controller ended up watching without standing up a manager and a cache.The tests pin the handler's behaviour, and the update cases exercise why both sides of an update are enqueued:
Dropping
ObjectOldfrom the handler fails the first two, so the migration guarantee is pinned rather than assumed. They also count cache scans, for the reason below.Absent and empty are two different labels
The first version of this change read the topology with a plain map lookup and skipped anything that came back
"". That conflates a label which is not there with one set to the empty string, and Kubernetes allows the second:Network.spec.topologyis required but carries nominLength, sotopology: ""is a value the schema accepts.The reconciler already treats it as one.
getProviderNodeslists withclient.MatchingLabels, and that selector matches the empty value rather than the missing key:selector topology="" matched 1: [empty-value] # and not [label-absent]So a
Networkwith an empty topology was consuming Nodes that this handler would never have woken it for — the same silent staleness this PR exists to fix, one level down. The two-value lookup distinguishes them, and dropping either half fails a different case.Why the update path lists once
The first version of this change called the handler body twice, once per side of the update. That scans the whole Network cache twice and calls
queue.Addtwice for each matching Network, and I described the second Add as being folded away by the workqueue. That is only true while nothing is consuming the queue — which is exactly the situation a unit test that drains after the call creates, and not one a handler can depend on:Run against
client-go's queue, the no-consumer case ends at length 1 and the interleaved case also ends at length 1 — the second one having already handed an item out, so it is two reconciles rather than one.So
Updatenow collects the eligible topologies of both sides, scans the cache once, and enqueues the union. One Add per Network by construction, no reliance on queue timing, and an update between two Nodes of some other provider returns without listing at all. A second scan is invisible in the resulting requests while nothing is consuming, so the tests assert the scan count directly; restoring the per-Node call fails them.Notes
r.devices) that fix: resolve concurrent state corruption in NetworkController #1082 deals with. The two overlap mechanically — same reconciler, likely conflicting diffs — but neither is a prerequisite for the other today: nothing passesMaxConcurrentReconcilesto this builder, so controller-runtime's default of 1 applies (controller.go:245) and there is one worker. If concurrency is ever enabled here, fix: resolve concurrent state corruption in NetworkController #1082's fix becomes necessary for correctness.watch_endpoint.gostill uses the untyped shape and a runtime type assertion. Converting it the same way would make both registrations compiler-checked and let the two bodies be shared, but it is a change to code this fix does not otherwise touch. Happy to follow up.controllers/pkghas nomanifeststarget of its own, and neithermake unitnormake lintregenerates anything, so the working tree after all four commands is clean apart from the two files in this diff.