This server

Instead the public surface is a checked-in inventory, and the collectors enforce it in the direction that fails safe. A container appears only if it carries a monitoring.public label. The renderer raises on a name, a status or a count it does not recognise, rather than passing it through. Adding a number to the public API is a commit against that inventory.

A collector that raises writes no file at all, the exporter keeps serving the last good one, and the API marks the group stale. Time sources show the pattern most clearly: peer addresses become city labels before the metric is written, so there is no version of the file with an address in it to leak.

Two services bind a host port at all, and both bind loopback only. Everything else talks over a private Docker network. Containers drop all capabilities and run with no-new-privileges; the API also runs as an unprivileged uid on a read-only root filesystem, with a 16 MB noexec,nosuid tmpfs as the only writable thing it can see.

Administration goes over a private mesh rather than the public internet. Security patches apply unattended and SMART watches both disks, which is most of what decides whether a box like this is still healthy in a year.

Two scripts read docker inspect, docker stats and chronyc, then write five numbers per container and twenty about the clock. Each writes a temp file and renames it, so the exporter can never read half a file, and the directory is mounted read-only into the exporter.

The clock sampler has to run on the host, because chronyd binds loopback in the host namespace and its socket directory is root-only. I first ran it as a user unit without lingering enabled, so it died with the login session and the metric stopped without anything going red.

Each group in the response carries its own state: live, stale, partial or unavailable. The cache only overwrites itself with a response it trusts, so if Prometheus disappears mid-scrape the API keeps serving the last good snapshot with the affected groups marked, and the dashboard draws dashes in those tiles instead of a spinner that never resolves.

The ceilings sit on the read rather than the write. A metric store answering slowly, or answering with a million points, is the realistic way a read-only API becomes a denial of service against its own host.

The gap is the cpuset rather than luck. Capture and compaction are confined to two cores by the kernel, so the half of the box that serves this page cannot be starved by the half that writes the archive.

A rollback trap is armed before anything changes. Its target defaults to the parent commit and can be named explicitly, which matters the one time you need to skip back over a bad one. The crontab that drives the collectors is captured before it is rewritten and restored with everything else, because a half-rolled-back deploy that leaves new cron lines behind is the failure that would actually have bitten.