Watchdogs and Fault Recovery: Designing a Machine That Can Recover Predictably

A machine hangs, the watchdog resets it, and the screen comes back. That can be a useful recovery step. It can also repeat a dispense, lose a part's state, or enter the same fault again. A watchdog is one mechanism in a recovery design. The important question is what the machine is allowed to do before, during, and after that reset.

This guide focuses on functional supervision and recovery architecture. It does not treat an ordinary watchdog as a substitute for application-specific safety functions and their validation. Control-system design covers the wider machine boundary. Here, the aim is to turn a restart from a hopeful side effect into a defined, testable sequence.

1. Define what healthy progress means

List the tasks and conditions that must progress for the application to remain useful. A sequence controller may need to process events, respect timeouts, and maintain an interpretable state. A communication task may legitimately wait for an unavailable network. A sensor can remain unchanged during a normal dwell. Health criteria must distinguish those conditions rather than requiring every value to change continuously.

Define evidence at the appropriate level: completion of a bounded work unit, a meaningful state update, or another reviewed condition. A heartbeat incremented by an unrelated timer can continue while the main application is stuck. Conversely, a timeout chosen without normal worst-case behavior can create unnecessary resets. The supervisor needs a model of expected work, not simply proof that some code is still running.

2. Understand the selected watchdog implementation

Review the exact processor, watchdog resources, clock sources, timeout behavior, startup settings, reset scope, and software framework configuration. Independent, windowed, task, interrupt, or boot supervision features vary by device and implementation. Do not combine capabilities from several family members into one assumed mechanism.

ST's watchdog overview distinguishes watchdog implementations and points to the relevant device reference manuals. Espressif's watchdog documentation describes interrupt, task, and startup-related supervision in the documented framework. Use the actual target and software version, including debug and configuration behavior, when defining the evidence.

Record what a timeout does and what it does not reset. A peripheral, external driver, communication module, or retained state may behave differently from the processor. The recovery architecture must include those boundaries instead of assuming a processor restart restores the complete product.

3. Make the feed policy part of the application design

Choose which conditions permit the watchdog to be refreshed and how their progress is evaluated. If a supervisor aggregates task health, define expected periods, stale reports, blocked tasks, startup, and deliberately inactive modes. Keep the supervisor itself bounded and review the paths that can falsely report success.

Avoid feeding solely from a path that remains alive while the important application work is not progressing. At the same time, do not treat every slow external service as grounds to reset local control. Connected-machine architecture explains why local behavior during network loss should be explicit. A reset cannot restore a mobile network that is unavailable at the site.

4. Review physical behavior during reset and startup

Identify output states before firmware initialization, during reset, after reset, and while external components remain powered. Review hardware defaults, driver enables, retained external states, and any required supervisory arrangement through the responsible design process. The intended response depends on the application; de-energizing every output is not a universal complete solution for every mechanism.

Define safety-related behavior separately with the appropriate specialists. A watchdog reset can be one input to the application's fault response without supplying all the necessary architecture, diagnostics, or validation. Industrial PCB requirements helps make startup and output assumptions visible before hardware design.

Use controlled bring-up to establish early observations, then verify the relevant assembled-machine behavior. A reassuring boot message does not show what an actuator did during the interval when the processor was not executing normal code.

5. Preserve the evidence needed after a fault

Identify reset-cause information, machine state, operation identity, configuration, firmware version, and relevant event history. Capture information according to the target's documented behavior and storage limitations. Some data may be volatile or changed during initialization. Do not make the fault-recording path depend on the same stalled resource whose failure is being investigated.

Persistent state needs an interruption-aware design and a verification plan. Decide how incomplete or invalid records are identified and what the startup policy does with them. Logging everything is not the goal; retain enough trustworthy context to decide whether work can resume, needs reconciliation, or requires service. Remote diagnostics expands the investigation use case.

Reset or fault indication:
Unit, board, firmware, and configuration:
Last interpretable machine state:
Operation identity and irreversible-action status:
Validity of retained records:
Relevant sensor and external-module conditions:
Startup decision and permitted next action:
Repeated-fault history and escalation:
Verification result and remaining limitations:

6. Reconcile before resuming the operation

On startup, check the information needed to establish a defined operating state. A retained state name does not necessarily prove the physical position or action outcome. Sensors, external modules, product presence, transaction records, and configuration may need reconciliation. Define which evidence permits continuation and which condition requires an authorized intervention.

A constructed dispensing example illustrates the problem. A reset occurs after an action started but before delivery was recorded. Returning to idle loses the unresolved operation; repeating the action can deliver twice. Payment-enabled state machines explains why the system needs completed, failed, and uncertain outcomes rather than a single restart instruction.

Apply the same reasoning to a process without payment. A part may remain clamped, an axis may have lost a reference, or another device may still hold a command. Recovery should be defined for the actual machine state and responsibilities, not copied from an example whose only output was a blinking LED.

7. Bound repeated failures and handle update-related resets

Define how repeated startup or runtime failures are recognized and reported. Decide whether the application enters a restricted or service-required mode and what actions are permitted. The policy depends on the fault, process, available evidence, and application requirements. Unlimited automatic restarts can hide a persistent fault or repeatedly disturb the mechanism.

Review firmware update, configuration migration, diagnostic, and service modes. Watchdog and boot policies must support their legitimate timing while still detecting the failure conditions they are intended to catch. OTA update planning explains image validation and recovery context. Do not solve a failing update by permanently disabling supervision without reviewing the complete behavior.

• Identify which reset indications and records distinguish repeated failures.
• Define permitted behavior when the application cannot establish a valid startup state.
• Include update and configuration compatibility in the recovery policy.
• Give service staff actionable records and a restoration procedure.
• Record any limits and escalation responsibilities explicitly.

8. Test supervision and the whole recovery sequence

Use an appropriately reviewed test plan to exercise lost progress, delayed work, communication outage, storage errors where relevant, restart, and invalid retained state. Verify detection and reset behavior, but also outputs, records, messages, reconciliation, and the final permitted state. Identify the exact hardware, software, configuration, and setup.

Check normal long operations too so the policy is not tested only by provoking faults. Record debug or development settings that differ from release. Factory acceptance planning helps connect these checks to the machine requirements rather than treating a successful forced reset as the entire test.

9. Release the policy, configuration, and evidence together

Keep the health criteria, watchdog configuration, reset behavior, persistence assumptions, startup decisions, tests, known limitations, and service procedure in the release package. Review changes when timing, framework, hardware, or machine behavior changes. A timeout value copied into a configuration file is not an adequate description of the recovery design.

SolveMpire can discuss embedded firmware, electronics, and machine fault handling. Bring the sequence, important deadlines, reset observations, outputs, retained-state needs, and support policy. Predictable recovery means the system can justify its next action after a fault, not merely demonstrate that it can start the same program again.