Jeff Mixon / Shape of a managed OpenWrt fleet: a sketch

What this post is, and what it isn’t

A lot of my professional work over the years has been some variant of “take a vendor router, replace its firmware with something based on OpenWrt, and run a managed fleet of them.” The exact customer, the exact product, the exact device family — those are not the interesting parts and they are not the parts I am free to write about. The shape the platform keeps reaching, the trade-offs that keep recurring, the things you only learn after the third iteration — those are.

This is a sketch of that shape, abstracted hard.

The recurring shape

Every variation I’ve shipped has converged on roughly the same set of boxes:

admin UI (Next.js)
   ↕
auth API (NestJS)  ←→  core/services APIs (NestJS)
                              ↕
                    device-side OpenWrt firmware
                              ↕
                          MQTT broker  (telemetry)
                              ↕
                       HashiCorp Vault  (secrets / signing keys)
                              ↕
                          GitLab CI   (firmware build + release)

The leverage is in the seams between them, not in any single box. The admin UI is a Next.js application talking to one API for auth and another for fleet operations. The fleet API issues cryptographically signed configuration blueprints to devices over HTTPS; those blueprints carry everything device-side OpenWrt needs to bring itself up correctly. MQTT flows the other way for telemetry. Vault holds the signing keys, the per-tenant secrets, and the AppRole credentials services use to fetch them. GitLab CI builds the firmware images and orchestrates the release flow.

Why signed blueprints, specifically

A device that pulls its configuration from a server has to answer one question loudly: did this configuration come from a server I trust, or did it come from someone running an attacker access point I happened to associate with? The answer needs to be in the bits of the configuration itself, not in the channel that delivered them.

So configurations are signed objects. The fleet API signs with an RSA or EC P-256 key (per environment); devices ship with the matching public key embedded in firmware and refuse anything that does not verify. The signing key never leaves Vault — the API fetches a wrapped operation, not the material. Rotating the signing key is a planned operation rather than a spelunking expedition.

The win is that the channel becomes uninteresting. You can mirror the configuration store across regions, you can cache configurations in CDNs, you can deliver them over channels you don’t fully trust, and a device sees the same answer in every case: signature valid or signature invalid.

The flasher is where you earn the operator’s trust

Every device family ships with the same problem: the first time a device is brought onto the fleet, it is running stock vendor firmware. Replacing that firmware is destructive, it is partially irreversible (depending on how the bootloader is laid out), and it is being performed by an operator who in the worst case is not the person who designed the device. The flasher is the tool that has to handle this without ruining anybody’s day.

The flow I keep coming back to:

1. The flasher runs in a Docker container. Operators get a single docker run line. No “install these forty packages” instructions, no per-distro toolchain hunt. The container has been built and tested as a unit.
2. Device survey first, write second. Before anything destructive, the flasher enumerates board info — chip family, bootloader version, current firmware version, MAC. The operator sees what the tool sees. If those match an expected target, the path forward is unambiguous. If they don’t, the tool refuses and explains why.
3. Confirm before write. Even with all the checks passing, the final irreversible write requires an explicit confirmation. There is no --yes flag for this. The minute you add one is the minute someone wrappers it and brick-flashes a hundred devices in a row at 3am.
4. Recovery is in the documentation, not in folklore. The most common recovery paths (serial console flow, TFTP recovery, U-Boot variable reset) are documented for each device family alongside the install command. Operators do not have to dig.

The OpenWrt ImageBuilder and signed-image pipeline live one tier up: the firmware itself is built by GitLab CI from pinned sources, signed, and shipped to the artifact store the flasher reads from. Operators are never flashing a binary that came out of someone’s laptop.

What stays the same vs. what changes

What stays the same across implementations: the box diagram, the signed-blueprint pattern, the operator-trust shape of the flasher, the “one API per concern” carving, the MQTT direction (devices to broker for telemetry; never the other way for control).

What changes: the device family, the bootloader specifics (U-Boot for some chips, custom for others — there is a separate effort for QCA95xx and a separate one again for the TF-A 2.9 / U-Boot 2023.07.02 / UBI flash layout on MT7622-class devices), the QoS layer on top (CAKE SQM autorate is a frequent flyer), the per-device anti-forensic / privacy hardening, the upgrade orchestration cadence.

What also changes: where I draw the line between “this is OpenWrt’s job” and “this is my custom package’s job.” Earlier iterations leaned harder on custom packages; later iterations have been more disciplined about upstreaming and feeding LuCI through.

Closer

There is no novel pattern in any individual box above. The work is in the seams: how the configuration store handles tenant scoping, how the auth API issues per-device credentials, how Vault is structured so that a leaked admin token can’t decrypt customer secrets, how the flasher tells an operator to stop. The platform shape works because each of those seams has been the place a previous version broke, and each one has the scars to prove it.

If you’re building one of these and you don’t know where to start: start with the flasher. It is the part operators touch first, it is the part that builds trust if it works and destroys it if it doesn’t, and getting its safety flow right will teach you more about your platform’s failure modes than any architecture diagram.