# Shape of a managed OpenWrt fleet: a sketch

> An architectural sketch of the platform shape I keep building for managed OpenWrt device fleets — admin portal, signed-config provisioning API, MQTT telemetry, Vault-backed secrets, and a Docker-isolated flasher with a safety flow operators actually trust.

By Jeff Mixon, published 2024-08-20. Canonical: https://www.jeffmixon.com/post/openwrt-fleet-provisioning/

## What this post is, and what it isn't

A lot of my professional work over the years has been some variant of
**"take a vendor router, replace its firmware with something based on
OpenWrt, and run a managed fleet of them."** The exact customer, the exact
product, the exact device family — those are not the interesting parts and
they are not the parts I am free to write about. The *shape* the platform
keeps reaching, the trade-offs that keep recurring, the things you only
learn after the third iteration — those are.

This is a sketch of that shape, abstracted hard.

## The recurring shape

Every variation I've shipped has converged on roughly the same set of boxes:

```
admin UI (Next.js)
   ↕
auth API (NestJS)  ←→  core/services APIs (NestJS)
                              ↕
                    device-side OpenWrt firmware
                              ↕
                          MQTT broker  (telemetry)
                              ↕
                       HashiCorp Vault  (secrets / signing keys)
                              ↕
                          GitLab CI   (firmware build + release)
```

The leverage is in the seams between them, not in any single box. The admin
UI is a Next.js application talking to one API for auth and another for
fleet operations. The fleet API issues **cryptographically signed
configuration blueprints** to devices over HTTPS; those blueprints carry
everything device-side OpenWrt needs to bring itself up correctly. MQTT
flows the other way for telemetry. Vault holds the signing keys, the
per-tenant secrets, and the AppRole credentials services use to fetch them.
GitLab CI builds the firmware images and orchestrates the release flow.

## Why signed blueprints, specifically

A device that pulls its configuration from a server has to answer one
question loudly: *did this configuration come from a server I trust, or did
it come from someone running an attacker access point I happened to
associate with?* The answer needs to be in the bits of the configuration
itself, not in the channel that delivered them.

So configurations are signed objects. The fleet API signs with an RSA or
EC P-256 key (per environment); devices ship with the matching public key
embedded in firmware and refuse anything that does not verify. The signing
key never leaves Vault — the API fetches a wrapped operation, not the
material. Rotating the signing key is a planned operation rather than a
spelunking expedition.

The win is that the channel becomes uninteresting. You can mirror the
configuration store across regions, you can cache configurations in CDNs,
you can deliver them over channels you don't fully trust, and a device
sees the same answer in every case: signature valid or signature invalid.

## The flasher is where you earn the operator's trust

Every device family ships with the same problem: the *first time* a device
is brought onto the fleet, it is running stock vendor firmware. Replacing
that firmware is destructive, it is partially irreversible (depending on
how the bootloader is laid out), and it is being performed by an operator
who in the worst case is not the person who designed the device. The
flasher is the tool that has to handle this without ruining anybody's day.

The flow I keep coming back to:

1. **The flasher runs in a Docker container.** Operators get a single
   `docker run` line. No "install these forty packages" instructions, no
   per-distro toolchain hunt. The container has been built and tested as a
   unit.
2. **Device survey first, write second.** Before anything destructive, the
   flasher enumerates board info — chip family, bootloader version, current
   firmware version, MAC. The operator sees what the tool sees. If those
   match an expected target, the path forward is unambiguous. If they
   don't, the tool refuses and explains why.
3. **Confirm before write.** Even with all the checks passing, the
   final irreversible write requires an explicit confirmation. There is no
   `--yes` flag for this. The minute you add one is the minute someone
   wrappers it and brick-flashes a hundred devices in a row at 3am.
4. **Recovery is in the documentation, not in folklore.** The most common
   recovery paths (serial console flow, TFTP recovery, U-Boot variable
   reset) are documented for each device family alongside the install
   command. Operators do not have to dig.

The OpenWrt ImageBuilder and signed-image pipeline live one tier up: the
firmware itself is built by GitLab CI from pinned sources, signed, and
shipped to the artifact store the flasher reads from. Operators are never
flashing a binary that came out of someone's laptop.

## What stays the same vs. what changes

What stays the same across implementations: the box diagram, the
signed-blueprint pattern, the operator-trust shape of the flasher, the
"one API per concern" carving, the MQTT direction (devices to broker for
telemetry; never the other way for control).

What changes: the device family, the bootloader specifics (`U-Boot` for
some chips, custom for others — there is a separate effort for QCA95xx and
a separate one again for the TF-A 2.9 / U-Boot 2023.07.02 / UBI flash
layout on MT7622-class devices), the QoS layer on top (CAKE SQM autorate
is a frequent flyer), the per-device anti-forensic / privacy hardening,
the upgrade orchestration cadence.

What also changes: where I draw the line between "this is OpenWrt's job"
and "this is my custom package's job." Earlier iterations leaned harder on
custom packages; later iterations have been more disciplined about
upstreaming and feeding LuCI through.

## Closer

There is no novel pattern in any individual box above. The work is in the
seams: how the configuration store handles tenant scoping, how the auth API
issues per-device credentials, how Vault is structured so that a leaked
admin token can't decrypt customer secrets, how the flasher tells an
operator to stop. The platform shape works because each of those seams has
been the place a previous version broke, and each one has the scars to
prove it.

If you're building one of these and you don't know where to start: start
with the flasher. It is the part operators touch first, it is the part
that builds trust if it works and destroys it if it doesn't, and getting
its safety flow right will teach you more about your platform's failure
modes than any architecture diagram.