feat(health): add health checks and maintenance page #12

Merged

ryan.mcgee merged 34 commits from feat/health-maintenance into main

2026-10-15 18:07:56 +00:00

ryan.mcgee commented

2026-10-14 14:37:40 +00:00

Owner

Summary

Health checks and a maintenance page for HTTP resources, and the status of targets reported back from the nodes. Stacked on #11 (feat/api-coverage).

Changes

Health check (path, interval 5 to 3600 s, timeout shorter than the interval): rendered as a Traefik healthCheck on every service. Validation (Resources.HealthCheck) is shared by the changeset and the renderer, which drops invalid stored settings.
Maintenance page, per resource: on answers every request for the host with the page (503, Retry-After) and renders no backend, access control or headers; auto renders the resource as usual with an errors middleware (502 to 504, including "no healthy server") in front. Without a usable control address on serves nothing (never exposes the backend) and auto leaves the page out.
GET /maintenance/:id: public, any HTTP method and Accept, no session, no layout and no assets (Traefik answers every path of the host with it, so /assets/app.css would be the page again), CSP default-src 'none', escaped message. It shows only the host and the message, never the resource's name. Anything else is a plain 404.
Target health over OTLP (your choice of design): the node's Traefik pushes metrics to POST /api/traefik/v1/metrics with its node token. traefik_service_server_up is decoded straight from the OTLP wire format, so no protobuf dependency; the decoder is tested against a real gzip capture from Traefik 3.7 (host details scrubbed), hand-built edge cases and a fuzz run. Limits: 2 MB body, 16 MB inflated (bounded, with an incomplete-stream check), 5000 data points; a report that cannot be read is refused whole.
Who may report what: a node can only report on HTTP resources that are assigned to it and in its own organization; the rest of a report is ignored. Stored per node (service_health); a target is down if any node that reported in the last 5 minutes says so, up if all do, unknown otherwise.
Surfaces: resource form and page (health column, live via PubSub), node page (metrics.otlp snippet), API (health_check_*, maintenance_*, health on targets), blueprints (health_check, maintenance sections; exported only when non-default), audit fields. Dev Traefik now reports to the dev app.
Two migrations (columns on resources; service_health), portable.

Verification

mix format --check-formatted, mix compile --warnings-as-errors, mix credo --strict (no issues), mix dialyzer (0 errors).
mix test: 963 passed on SQLite and on PostgreSQL.
Migration round trip with data (migrate, populate, roll back 2, migrate): rows survive, defaults restored, on both backends.
Live smoke on the compose stack: health check set through the API, Traefik reports the server UP and the API shows up; stopping the whoami container makes the API show down within seconds and the visitor gets the automatic page (503, any path, any method); restarting recovers it. In the browser: "On" mode through the form, the visitor sees the real page, the resource page shows the badge, health check and up. Dev database restored afterwards.

Notes

Alerting (next) will use the change events Health.ingest/3 already returns (down, back up, first seen down).
Raw TCP/UDP resources have no health check or maintenance (Traefik's TCP health check is out of scope here).

## Summary - Health checks and a maintenance page for HTTP resources, and the status of targets reported back from the nodes. Stacked on #11 (`feat/api-coverage`). ## Changes - **Health check** (path, interval 5 to 3600 s, timeout shorter than the interval): rendered as a Traefik `healthCheck` on every service. Validation (`Resources.HealthCheck`) is shared by the changeset and the renderer, which drops invalid stored settings. - **Maintenance page**, per resource: `on` answers every request for the host with the page (503, `Retry-After`) and renders no backend, access control or headers; `auto` renders the resource as usual with an `errors` middleware (502 to 504, including "no healthy server") in front. Without a usable control address `on` serves nothing (never exposes the backend) and `auto` leaves the page out. - **`GET /maintenance/:id`**: public, any HTTP method and Accept, no session, no layout and no assets (Traefik answers every path of the host with it, so `/assets/app.css` would be the page again), CSP `default-src 'none'`, escaped message. It shows only the host and the message, never the resource's name. Anything else is a plain 404. - **Target health over OTLP** (your choice of design): the node's Traefik pushes metrics to `POST /api/traefik/v1/metrics` with its node token. `traefik_service_server_up` is decoded straight from the OTLP wire format, so no protobuf dependency; the decoder is tested against a real gzip capture from Traefik 3.7 (host details scrubbed), hand-built edge cases and a fuzz run. Limits: 2 MB body, 16 MB inflated (bounded, with an incomplete-stream check), 5000 data points; a report that cannot be read is refused whole. - **Who may report what:** a node can only report on HTTP resources that are assigned to it and in its own organization; the rest of a report is ignored. Stored per node (`service_health`); a target is `down` if any node that reported in the last 5 minutes says so, `up` if all do, `unknown` otherwise. - **Surfaces:** resource form and page (health column, live via PubSub), node page (`metrics.otlp` snippet), API (`health_check_*`, `maintenance_*`, `health` on targets), blueprints (`health_check`, `maintenance` sections; exported only when non-default), audit fields. Dev Traefik now reports to the dev app. - Two migrations (columns on `resources`; `service_health`), portable. ## Verification - `mix format --check-formatted`, `mix compile --warnings-as-errors`, `mix credo --strict` (no issues), `mix dialyzer` (0 errors). - `mix test`: 963 passed on SQLite and on PostgreSQL. - Migration round trip with data (migrate, populate, roll back 2, migrate): rows survive, defaults restored, on both backends. - Live smoke on the compose stack: health check set through the API, Traefik reports the server UP and the API shows `up`; stopping the `whoami` container makes the API show `down` within seconds and the visitor gets the automatic page (503, any path, any method); restarting recovers it. In the browser: "On" mode through the form, the visitor sees the real page, the resource page shows the badge, health check and `up`. Dev database restored afterwards. ## Notes - Alerting (next) will use the change events `Health.ingest/3` already returns (down, back up, first seen down). - Raw TCP/UDP resources have no health check or maintenance (Traefik's TCP health check is out of scope here).

ryan.mcgee added 2 commits

2026-10-14 14:37:40 +00:00

feat(resources): add health checks and maintenance e605ff72c3

- resources gain a health check (path, interval, timeout) rendered as
  a Traefik healthCheck on every service; validation is shared by
  the changeset and the renderer
- maintenance mode: on answers every request with a public,
  self-contained page and renders no backend; auto shows it when
  Traefik answers 502 to 504; fail closed without a control address
- public /maintenance/:id (503, any method, escaped message)
- form, show page, API fields and blueprint sections
  (health_check, maintenance), audited like other resource fields

feat(health): show target health from OTLP reports b19b5aa9cf

- nodes push Traefik metrics to POST /api/traefik/v1/metrics (node
  token); traefik_service_server_up is read straight from the OTLP
  wire format, checked against a real capture, with strict limits
  (size, inflated size, data points) and no protobuf dependency
- reports only count for resources assigned to the reporting node
  and its organization; stored per node in service_health
- targets carry health (up, down, unknown): resource page (live),
  API, node page snippet to enable it
- dev Traefik reports to the dev app

ryan.mcgee referenced this pull request

2026-10-14 17:32:48 +00:00

feat(alerts): alert by webhook and email #13

ryan.mcgee changed target branch from feat/api-coverage to main

2026-10-15 18:07:50 +00:00

ryan.mcgee added 1 commit

2026-10-15 18:07:53 +00:00

merge main into feat/health-maintenance 64148430a4

ryan.mcgee merged commit bfb7a79583 into main

2026-10-15 18:07:56 +00:00

ryan.mcgee referenced this pull request from a commit

2026-10-15 18:07:58 +00:00