ros2_medkit_graph_watchdog

Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.

Detectors read the graph and raise faults through a ReportFault service client on the gateway node; the faults surface via FaultManager on the gateway /faults API. That client, like the lifecycle subscriptions below, lives in the plugin’s own callback group and is driven by the plugin’s own executor from the tick thread, never by the gateway’s ROS executor.

This package carries the plugin skeleton, the central reliability gate that holds raises until the graph has quiesced, and five detectors, qos_mismatch, orphan, param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes, each against its own issue, and their fault codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).

Build

With the gateway built and sourced:

colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose

Load into the gateway

ros2 run ros2_medkit_gateway gateway_node --ros-args \
  -p plugins:="[graph_watchdog]" \
  -p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"

Configuration (plugins.graph_watchdog.<key>)

Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so mode: off arrives as false, not the string "off"; both forms disable the detector.

Unknown keys are reported. Anything under detectors.<id> that the detector does not read produces one startup warning naming the key and listing the ones that exist, so a typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector id itself is warned about the same way, with the registered ids listed.

lifecycle_expectation keys

node_death keys

Zero-config, unlike every detector above: there is no require_active-style list of nodes to watch, because every armed App in the graph is a candidate.

tick_interval_ms is not an own key of this detector, but this is the one detector that reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.

Liveness, and why membership in the snapshot is not enough. App::is_online is what this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest keeps a bound App present with only is_online cleared once its ROS binding disappears, and hybrid discovery inherits that same shape; counting snapshot membership alone would make a manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps is_online: true - its process is still running, only its ROS 2 lifecycle state changed - so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s concern, not this one’s (see “The boundary with lifecycle_expectation” below).

When the ROS graph keeps a leftover of a node after its participant left - an entry with an empty enclave and no endpoints - the App leaves the runtime-only snapshot only while the gateway remembers the node: a read of the graph saw it running, the late discovery sample that left the leftover arrived before the gateway forgot the node, and the node was not among the names forgotten once the gateway remembers more than 1024 departed nodes. The gateway forgets a node on the first read that finds no entry of it more than 10 s after the first read that found none; a late sample that arrives before that read stays hidden, however late. Reads come from refreshes, the gateway’s start and, in runtime_only mode, a request for an App or a Function that a refresh removed from the entity cache while the request ran. Otherwise the leftover stays in the snapshot as an online App, and this detector does not see the node die (see “How long a departed node keeps being listed” in the gateway’s docs/config/server.rst).

File truncated at 100 lines see the full file

Changelog for package ros2_medkit_graph_watchdog

0.7.0 (2026-08-27)

• Initial release of the package: a gateway plugin that raises faults for silent failures in the ROS 2 graph - problems that leave every node alive and every topic present, so nothing in the stack reports them today. The package carries the plugin skeleton, a central reliability gate that holds raises until the graph has quiesced, and a detector registry. Detectors are configured under plugins.graph_watchdog.<key>, each with a raise / advisory / off mode, and an unknown key under detectors.<id> is reported rather than silently ignored, so a typo cannot quietly disable a check (#571)
• qos_mismatch detector: raises GRAPH_QOS_MISMATCH for a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571)
• orphan detector: raises GRAPH_ORPHAN for a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578)
• param_drift detector: raises GRAPH_PARAM_DRIFT when a node's live parameter value diverges from the declared expectation (#580)
• lifecycle_expectation detector: raises GRAPH_NODE_INACTIVE for a node named in require_active that is not in the active lifecycle state once its grace window has passed. It raises two further codes for the cases where the state could not be established at all: GRAPH_NODE_UNREADABLE when the lifecycle service does not answer, and GRAPH_NODE_NOT_MANAGED when the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering on GRAPH_NODE_INACTIVE alone sees neither unmeasured case. A node is matched by its App::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587)
• node_death detector: raises GRAPH_NODE_DISAPPEARED for a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624)
• Two silent-fault classes are not delivered in this release. GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET have their fault codes reserved in the frozen GRAPH_* namespace and land in later changes, each against its own issue
• The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
• Contributors: \@bburda