The release-note caution "Do not update if your system is working"
may be less common than it was a decade ago, but the implied risk has
not gone away. Easier recovery keeps a failed update from bricking the
device, but it still wastes the customer's time and turns the next install
from a button press into a risk to schedule. So customers learn to avoid
installing, while developers learn to avoid change, all from fear of
flashing.

At Oxide, the Service Processor and Root of Trust run our Hubris-based
firmware in place of a typical BMC or EC, on our servers, power shelf
controllers, and network switches.

There is no call-home; customers update when they choose, and once a rack
ships, whether we can reach it is up to them. We require confidence before
release, and hope to earn customer trust afterwards. Improving our testing regime
is one way to strengthen both.

sp-test is a new harness for direct hardware testing. It uses
tools developers know, can emulate parts of the control plane,
and lets a test declare what it needs so it runs where it applies and
skips without failure when it doesn't. Tests once described only in PR
comments can now be checked in and reused.

Tests can run unchanged from a developer's bench to hardware-based
CI, so someone with no board of their own can get results from
scarce, shared hardware. The hardest tests to run are often the ones most
worth sharing: the update path exercised as it is in production, code
that runs only when something has gone wrong, and fault insertion tests.
Telemetry, task dumps, and logs make test results actionable for someone
who was not there when it failed.

This pattern is not specific to Hubris or to Oxide; any team could build the
same kind of testing.
A live demo on an off-the-shelf ST Nucleo board makes it concrete.