Part of #570.
Summary
A lifecycle node comes up and never reaches active. The bringup order changed, a
configure callback returned FAILURE, or nobody activated the node after a manual
transition. The process runs, the node is in the graph, and nothing reports a problem.
Other detectors cannot see this. The node publishes nothing, but a node that is not active
is not supposed to publish, so silence is correct here and proves nothing. To find it by
hand the operator has to suspect the right node and run ros2 lifecycle get on it.
Proposed solution
Let the operator name the nodes that must be active, and raise GRAPH_NODE_INACTIVE when
one of them is present but stays in another state for longer than a grace period.
The detector watches nothing until it is configured. A node in unconfigured is only a
fault if somebody expected it to be active, and the plugin cannot know that by itself.
Three things it has to be careful about.
A node that is absent is not this detector's job. That is the presence class
(GRAPH_NODE_DISAPPEARED), and reporting it here too would report one failure twice under
two codes.
Clearing has to be as strict as raising. A clear says every required node is healthy. If
the detector has not read a state yet, it knows nothing and must say nothing. This matters
most right after a gateway restart, when a fault is already stored and no state has been
read yet: a clear at that moment would heal a fault that is still real, and the fault
manager would count it toward healing.
Node identity can move. An app id can stay the same while it points at a different node.
Anything cached per node has to notice that, or one node's lifecycle state gets enforced
against another node.
Also in scope
The shared lifecycle state cache keys entries by app id and never re-checks whether the id
still points at the same node. A node that is replaced under the same id keeps the old
node's state and the old event subscription. This detector is the first one that acts on
that state directly, so the cache needs to treat a moved binding as a new node: drop the
old label, subscribe again, and read the new one.
Part of #570.
Summary
A lifecycle node comes up and never reaches
active. The bringup order changed, aconfigure callback returned FAILURE, or nobody activated the node after a manual
transition. The process runs, the node is in the graph, and nothing reports a problem.
Other detectors cannot see this. The node publishes nothing, but a node that is not active
is not supposed to publish, so silence is correct here and proves nothing. To find it by
hand the operator has to suspect the right node and run
ros2 lifecycle geton it.Proposed solution
Let the operator name the nodes that must be active, and raise
GRAPH_NODE_INACTIVEwhenone of them is present but stays in another state for longer than a grace period.
The detector watches nothing until it is configured. A node in
unconfiguredis only afault if somebody expected it to be active, and the plugin cannot know that by itself.
Three things it has to be careful about.
A node that is absent is not this detector's job. That is the presence class
(
GRAPH_NODE_DISAPPEARED), and reporting it here too would report one failure twice undertwo codes.
Clearing has to be as strict as raising. A clear says every required node is healthy. If
the detector has not read a state yet, it knows nothing and must say nothing. This matters
most right after a gateway restart, when a fault is already stored and no state has been
read yet: a clear at that moment would heal a fault that is still real, and the fault
manager would count it toward healing.
Node identity can move. An app id can stay the same while it points at a different node.
Anything cached per node has to notice that, or one node's lifecycle state gets enforced
against another node.
Also in scope
The shared lifecycle state cache keys entries by app id and never re-checks whether the id
still points at the same node. A node that is replaced under the same id keeps the old
node's state and the old event subscription. This detector is the first one that acts on
that state directly, so the cache needs to treat a moved binding as a new node: drop the
old label, subscribe again, and read the new one.