Workshop preparation: Hands-on Hubris testing with sp-test

[!WARNING]
Pre-release material, for education. The workshop runs against
unreleased revisions of sp-test, sp-emu and Hubris, cut for a hands-on
session rather than published as a supported release. The interfaces the
exercises teach are still moving. Learn the shapes here, and check the tool
in front of you before you rely on a flag.

What to do before the session, and the ways to take part. On the day we will make
sure everyone has the workshop materials and hand out Nucleo boards.

What sp-test does

sp-test is a new tool to test Hubris. It:

• drives the tools developers already use: the Hubris debugger humility, the
  Management Gateway Service tool faux-mgs, and others
• flashes firmware by emulating the control plane's firmware-update flows, adding
  instrumentation and optional fault injection
• uses a target's hardware and firmware properties to filter test suites, so tests
  that do not apply are skipped rather than failed
• runs the same test against the sp-emu emulator and real service-processor
  hardware
• records each run's environment and results so they can be shared and meaningfully
  compared
• discovers and correlates SP and RoT debug probes, IPCC serial links, and Ethernet
  access

Before the workshop

• Pull the workshop container and run it once to confirm it starts:

```sh
podman pull ghcr.io/oxidecomputer/sp-test-osfc26:osfc2026

mkdir -p ~/workshop-work
podman run -d --name workshop \
-v ~/workshop-work:/work:Z \
ghcr.io/oxidecomputer/sp-test-osfc26:osfc2026 sleep infinity

podman exec workshop bash -lc \
'until faux-mgs -l warn --sp-sim-addr "[::1]:11111" state >/dev/null 2>&1; do sleep 5; done; echo ready'

podman exec -it workshop bash -l
```

The container shares ~/workshop-work at /work. Edit tests and exercises
in your own editor and run them inside. Keep :Z on SELinux hosts
(Fedora, RHEL and their derivatives) and drop it elsewhere. On macOS keep
the directory under /Users. The podman VM shares that part of the
filesystem and no other. Docker runs the image the same way.

The link is confirmed by email before the conference.

• Re-watch Matt Keeter's 2023 OSFC talk, "Unplugging the Debugger: Live and
  post-mortem debugging in a remote system"
  (https://talks.osfc.io/open-source-firmware-conference-2023/talk/9TTFZF/), on
  debugging a running Hubris system with humility.
• For the design of Hubris itself, re-watch Cliff Biffle's 2021 OSFC talk, "On
  Hubris and Humility: developing an OS for robustness in Rust"
  (https://www.osfc.io/2021/talks/on-hubris-and-humility-developing-an-os-for-robustness-in-rust/).

Ways to take part

Pick the path that matches how far you want to go. You can start with the emulator
and move up during the session.

1. Emulator only (quickest start)

If you are happy running the full toolset against the emulated Hubris devices,
the workshop container is all you need. One tag carries both architectures and
podman takes the one your machine needs: about 268 MB down and 492 MB on disk
for amd64, 266 MB and 508 MB for arm64. sp-emu can also run outside a
container if you prefer.

2. Nucleo STM32H753ZI development board

To test against a real Nucleo during the session, have ready:

• an available Ethernet port on your laptop, configured for point-to-point IPv6 with
  the Nucleo (or an L2 switch with free ports)
• an available USB port and a micro-B USB cable for the Nucleo's built-in ST-LINK
  debugger

Configuring a laptop to talk to the board (IPv6, permissions, and other system
administration) is well within an OSFC attendee's reach. Setup documentation helps;
hands-on help for individual system issues during the session will be limited, since
the container path avoids all of it.

3. Build from source (developer mode)

To modify and build Hubris, humility, or the other tools, set up a full toolchain:

• install Rust. Hubris pins its own toolchain
  (rust-toolchain.toml); do not override it.
• install jq and unzip from your package manager, and toml2json from
  crates.io: cargo install toml2json.
• clone Hubris and build a Nucleo image
  that includes control_plane_agent. Master has it as of 46a7fb814, so build
  from master: cargo xtask dist app/demo-stm32h7-nucleo/app-h753.toml.
• Humility: fetch the prebuilt release binary if you can (recommended; building it
  from source interacts badly with the pinned toolchain). To build it anyway, clone
  Humility and run
  cargo install --path humility-bin --locked.
• build faux-mgs: clone
  management-gateway-service
  and run cargo install --path faux-mgs --locked.
• sp-test has no public source yet. oxidecomputer/sp-test is an internal
  repository, and the snapshot cut for this workshop is not published either.
  Use the container, which carries the binary and the full source under
  /opt/workshop/reference/sp-test. To build from that copy:
  cargo install --path sp-test --locked.
• if the Nucleo's built-in ST-LINK firmware is old, reflash it with
  STSW-LINK007 (not
  available on illumos).

Learning and experimenting progression

Independent of which target you use, the progression from running some quick tests
to the full Hubris developer experience varies by interest and experience.

• Stage 1, run and read: run the example tests in the container and read
  their output.
• Stage 2, write and filter:
• write and modify tests and test suites
• gate tests on capabilities, so a test skips where it does not apply
• run test suites against different Hubris targets
• Stage 3, developer mode (ambitious or take-home): modify and build
  Hubris, test the change with the developer tools, and write a test or fold
  it into a suite.

Hands-on Hubris testing with sp-test

OSFC 2026, Thursday 17 September, 14:30 to 16:30, Workspace // Workshop Room 2,
Pakhuis de Zwijger. Two hours, classroom setup, 20 places.
Schedule: https://talks.osfc.io/osfc-2026/schedule/

[!WARNING]
Pre-release material, for education. The workshop runs against
unreleased revisions of sp-test, sp-emu and Hubris, cut for a hands-on
session rather than published as a supported release. The interfaces the
exercises teach are still moving. Learn the shapes here, and check the tool
in front of you before you rely on a flag.

You explore the interfaces of Oxide's service processor, then write automated
tests of it with sp-test, on an emulator and on real hardware. The session
assumes no Rust or Hubris experience, and a short primer opens it.

Before the workshop

Three things, in the order of time needed to complete. Each has a section below.

• Download and run the container. Everyone needs it. It holds the service
  processor emulation and the tools.
• Watch two earlier OSFC talks. They cover the tools and the OS.
• Set your laptop up for the Nucleo, only if you want to drive a board
  during the session. Bring a micro-USB cable, an Ethernet cable, and a free
  Ethernet port you can leave up without DHCP, for a point-to-point IPv6 link
  to the board. I bring some spare adapters and cables.

Download the container

The image is published on ghcr.io. The pull command goes out by email before
the conference. Run it once at home, and again the day before the workshop to
pick up any late change.

One tag carries both architectures, so podman takes the one your machine
needs and Apple Silicon runs the emulator natively. The pull moves about
268 MB and unpacks to about 492 MB, or 266 MB and 508 MB on arm64. sp-emu
saturates one core, and anything that runs a container comfortably runs
this. Podman and Docker both run the image, as does remote access to a
machine that does.

Pull it, then give it a directory of your own. The container shares that
directory at /work and never owns it, so you edit tests and exercises in
your own editor and run them inside. What you write there outlives the
container:

podman pull ghcr.io/oxidecomputer/sp-test-osfc26:osfc2026

mkdir -p ~/workshop-work
podman run -d --name workshop \
    -v ~/workshop-work:/work:Z \
    ghcr.io/oxidecomputer/sp-test-osfc26:osfc2026 sleep infinity

Keep :Z on SELinux hosts (Fedora, RHEL and their derivatives). Drop it
elsewhere.

Starting the container boots an emulated gimlet-c: the SP, the RoT beside it,
and bootleby. Expect tens of seconds. Wait for the SP rather than the shell
prompt, then open as many shells as you like:

podman exec workshop bash -lc \
    'until faux-mgs -l warn --sp-sim-addr "[::1]:11111" state >/dev/null 2>&1; do sleep 5; done; echo ready'

podman exec -it workshop bash -l

Your shell lands with the exercises in /work and the environment already
describing the running system, so sp-test needs no paths from you. Leaving
the shell keeps the container running. podman stop workshop and podman start workshop resume the same emulator later; podman rm -f workshop throws
the instance away and leaves ~/workshop-work alone.

On macOS the podman machine mounts /Users, so pick a directory under it. A
home directory elsewhere, on an external volume say, cannot be shared into the
container. The error names statfs rather than the mount:

Error: statfs /Volumes/.../workshop-work: no such file or directory

The emulated SP keeps its flash between runs, the way a board on a bench does.
An image you flash in one session is still there in the next. sp-emu-testbed.sh --reprovision gets a clean one back.

The image holds sp-emu, sp-test, humility, faux-mgs, some other
support utilities, the exercises, and prebuilt Hubris archives. It is the
tested path on every operating system. Once pulled, it needs no network.

Download from a high bandwidth connection before the workshop if you
can. Twenty people fetching a few hundred megabytes at once may not
perform well on the day. There will be a server in the room handing out
the image, and USB sticks as well, so arriving without it is recoverable.

Watch two talks from earlier OSFCs

Neither is required. Both leave the material fresh.

• Matt Keeter, 2023, live and post-mortem debugging with humility:
  https://talks.osfc.io/open-source-firmware-conference-2023/talk/9TTFZF/
• Cliff Biffle, 2021, on the design of the Hubris OS:
  https://www.osfc.io/2021/talks/on-hubris-and-humility-developing-an-os-for-robustness-in-rust/

If you want to drive your own Nucleo, set your laptop up first

The boards will be pre-flashed and ready to run.

Build the tools natively: at least humility and faux-mgs.
Prebuilt binaries are available, and you can compile your own.
attendee-prep.md has the commands.
Budget an evening, not ten minutes, and do it before you travel.

What the session is

Hubris is Oxide's Rust embedded operating system for service processors.

faux-mgs and faux-ipcc exercise the same external APIs as our control plane.
humility queries, debugs, and analyzes an SP or Root of Trust (RoT) on a board
configured for development. sp-test drives tests portably, at a developer's
bench or in hardware CI. Results land in a structured directory, ready to read,
or for a CI run to post.

sp-test matches every test against the target in front of it. A test
lists what it needs as required_capabilities in its test-info.toml:
strings like sp:task:control_plane_agent, sp:task:net, or
sp:socket:hiffy. The target's capabilities are collected at run time
from the Hubris archive, the running firmware, and the interfaces the
testbed exposes. An unmet requirement skips the test rather than passing
or failing it. One properly qualified test suite can therefore run against
any testbed.

Most of the two hour session is devoted to hands-on work.

After a short review and demonstration that you can follow along with on your
laptop, you explore the system, find something to break, and write a test that
catches it. Most of what follows is open work time. Near the end, feedback,
questions, and any demonstration you would like to give are welcome.

One practical request: sit with someone if you can. There is one instructor for
the room, so a neighbour is often the faster answer. Talking it through also
helps you find a problem worth working on.

Optional: your own board

There are 20 STM32H753ZI Nucleo-144 boards to give away, handed out
at the door. Start in the container and move to a board when you are
ready. No exercise depends on having the physical board working. I also
bring a small server with Nucleo boards attached that you can run tests
against. Access to it will be resolved by the time of the workshop.

Driving your own board needs two free ports: USB for the on-board ST-LINK,
over a micro-USB cable, and Ethernet for a point-to-point IPv6 link to
the board. If you have those and a spare micro-USB cable, bring them. Set
the link up beforehand, so session time goes to the exercises rather
than to network configuration.

How you reach the board depends on your operating system, and this is the only
place it matters.

Linux. The container can connect to the Nucleo board. Pass the probe
in and share the host network:

podman run --rm --network=host \
    --device /dev/bus/usb/001/006 \
    --group-add keep-groups \
    <image> bash

--group-add keep-groups carries your plugdev membership inside, so the probe
opens without root. The device numbers come from lsusb, and they change when
you replug the board. On Debian and derivatives, apt install passt first;
rootless Podman needs it.

macOS. Check that your home directory is somewhere the podman VM shares. A
home on an external volume is not mounted by default, and -v ~/workshop:/work
then fails with statfs: no such file or directory. podman machine init -v <dir>:<dir> fixes it.

The container runs in a VM that doesn't allow raw USB access, so a
physical board needs the tools built natively: humility, faux-mgs, sp-test,
and sp-emu. Xcode command line tools and rustup are enough; no Homebrew
packages are needed, and the probe opens without sudo. attendee-prep.md has the
commands. Everything else works in the container, including the exercises that
use a debug probe, because the emulator serves one over TCP. Note that the
st-util method is not practical for our purposes.

Windows. Same VM boundary. WSL2 with usbipd is the usual answer. People do
build Hubris and humility natively on Windows. I have not, so I cannot help
with it during the session.

I am bringing roughly 10 sets of (USB cable, Ethernet cable, and USB Ethernet
adapter). A set needs one USB-C or USB-A port for the adapter and a second port
for the board.

What you leave with

• the tests you wrote,
• sharable results that others can reproduce,
• your own runtime environments to continue development and testing of Hubris

And, I hope some new thoughts about testing embedded software.