feat(alerts): alert by webhook and email #13

Merged

ryan.mcgee merged 35 commits from feat/alerting into main

2026-10-15 18:08:03 +00:00

ryan.mcgee commented

2026-10-14 17:32:34 +00:00

Owner

Summary

Alerts: organizations add signed webhooks and email recipients and choose what they hear about: a node stops polling or comes back, a target fails its health check or recovers. Stacked on #12 (feat/health-maintenance).

Changes

Channels (alert_channels, at most 20 per organization): kind, name, destination, events, on/off, and the outcome of the last delivery. Managed by admins signed in as users on /orgs/:org/alerts. Not in the API, deliberately: a leaked API key must not be able to redirect an organization's alerts. Members cannot read them (a webhook URL is often a secret). Admin changes are audited naming a webhook's host, never its URL, and no email address.
Webhook secret: generated, shown once, stored encrypted with a key derived from secret_key_base (signing needs the secret itself, so it cannot be a hash; changing secret_key_base makes stored secrets unreadable, New secret replaces one). Requests carry x-traefik-control-timestamp and x-traefik-control-signature: v1=<hex>, the HMAC-SHA256 of "<timestamp>.<body>".
SSRF guard (Alerts.Network): a destination is refused when any address it resolves to is private, loopback, link-local, CGNAT, documentation, multicast and so on (IPv4 and IPv6, including IPv4-mapped, NAT64 and 6to4 forms), at save time and again at every delivery. The request then goes to the address that was checked, with the host name kept for Host and TLS, so a name cannot change its answer in between. No redirects, 5 s limit, answers never read or kept. TRAEFIK_CONTROL_WEBHOOK_ALLOW_PRIVATE=true lifts it for operators who trust the network. Names are only looked up when saving, never while the form validates (see below), and every lookup is time-bounded.
Delivery: background tasks under a supervisor (max 100), retried after 5 and 30 s, outcome recorded on the channel; Delivery.deliver/2 never raises. Best effort, no persistent queue: an attempt in flight is lost on restart (documented). A burst alerts the first 20 events; a flapping target alerts on every change.
Node monitor (Nodes.Monitor): every 30 s compares last_seen_at with TRAEFIK_CONTROL_NODE_OFFLINE_AFTER (default 180, min 90, 0 = off). Presence changes are claimed with conditional updates, so with several instances exactly one alerts (tested with four concurrent checks). Nodes whose state was unknown settle silently.
Target alerts come from Health.ingest/3 (down, back up, or first seen down).
Email: new dependency gen_smtp 1.3.0 (approved; BSD-2, current, pulls in ranch) for Swoosh's SMTP adapter. TRAEFIK_CONTROL_SMTP_* / TRAEFIK_CONTROL_MAIL_FROM; the server certificate is always verified unless TLS is none. Without an SMTP host, email channels cannot be added. Plain text only.
Migration: alert_channels and nodes.presence, portable (the array column round-trips on both backends).

Bugs found by running it for real

Validating the form on every keystroke resolved the host name each time; in a container with a slow resolver this froze the page. DNS now happens only on save.
username: nil in the SMTP config makes Swoosh raise even with auth: :never; credentials are now passed only when given.
Swoosh decides its MIME encoding at compile time: a _build made before gen_smtp was added fails every send with BadMapError until mix deps.compile swoosh --force (documented in the README; a clean build is fine). An exception in the mail adapter used to crash the LiveView and the background task before the failure was recorded; deliveries now never raise.
A tampered, non-UUID channel id crashed the page on PostgreSQL only (SQLite tolerates it); malformed ids are now "not found".

Verification

mix format --check-formatted, mix compile --warnings-as-errors, mix credo --strict (no issues), mix dialyzer (0 errors).
mix test: 1060 passed on SQLite and on PostgreSQL. The webhook tests run against a real local HTTP server (signatures verified, retries, redirects refused, timeouts, private destinations refused at send time, the Host header kept); the node monitor tests include the concurrent claim.
Migration round trip with data on both backends.
Live smoke on the compose stack with a real SMTP server (Mailpit) and a webhook receiver: a private destination is refused with the default settings; with it allowed, a webhook and an email channel were added in the browser, Test delivered to both, and a webhook signature was verified independently with openssl. Then real incidents: stopping whoami alerted target.down on both channels, restarting it target.up (a healthy first sighting alerted nothing); stopping the node's Traefik alerted node.offline about 2.5 minutes after its last poll, restarting it node.online. Each event arrived once per channel. The audit log named the host and no URL, token or address. The dev database was restored afterwards.

Not covered

TLS verification of an HTTPS webhook through the pinned address (host name kept via Mint's hostname option) is not exercised against a real TLS server, and SMTP with STARTTLS or ssl against a real certificate was not run (Mailpit ran with TLS none).

## Summary - Alerts: organizations add signed **webhooks** and **email** recipients and choose what they hear about: a node stops polling or comes back, a target fails its health check or recovers. Stacked on #12 (`feat/health-maintenance`). ## Changes - **Channels** (`alert_channels`, at most 20 per organization): kind, name, destination, events, on/off, and the outcome of the last delivery. Managed by admins signed in as users on `/orgs/:org/alerts`. **Not in the API**, deliberately: a leaked API key must not be able to redirect an organization's alerts. Members cannot read them (a webhook URL is often a secret). Admin changes are audited naming a webhook's host, never its URL, and no email address. - **Webhook secret:** generated, shown once, stored encrypted with a key derived from `secret_key_base` (signing needs the secret itself, so it cannot be a hash; changing `secret_key_base` makes stored secrets unreadable, **New secret** replaces one). Requests carry `x-traefik-control-timestamp` and `x-traefik-control-signature: v1=<hex>`, the HMAC-SHA256 of `"<timestamp>.<body>"`. - **SSRF guard** (`Alerts.Network`): a destination is refused when any address it resolves to is private, loopback, link-local, CGNAT, documentation, multicast and so on (IPv4 and IPv6, including IPv4-mapped, NAT64 and 6to4 forms), at save time and again at every delivery. The request then goes to the address that was checked, with the host name kept for `Host` and TLS, so a name cannot change its answer in between. No redirects, 5 s limit, answers never read or kept. `TRAEFIK_CONTROL_WEBHOOK_ALLOW_PRIVATE=true` lifts it for operators who trust the network. Names are only looked up when saving, never while the form validates (see below), and every lookup is time-bounded. - **Delivery:** background tasks under a supervisor (max 100), retried after 5 and 30 s, outcome recorded on the channel; `Delivery.deliver/2` never raises. Best effort, no persistent queue: an attempt in flight is lost on restart (documented). A burst alerts the first 20 events; a flapping target alerts on every change. - **Node monitor** (`Nodes.Monitor`): every 30 s compares `last_seen_at` with `TRAEFIK_CONTROL_NODE_OFFLINE_AFTER` (default 180, min 90, 0 = off). Presence changes are claimed with conditional updates, so with several instances exactly one alerts (tested with four concurrent checks). Nodes whose state was unknown settle silently. - **Target alerts** come from `Health.ingest/3` (down, back up, or first seen down). - **Email:** new dependency `gen_smtp` 1.3.0 (approved; BSD-2, current, pulls in `ranch`) for Swoosh's SMTP adapter. `TRAEFIK_CONTROL_SMTP_*` / `TRAEFIK_CONTROL_MAIL_FROM`; the server certificate is always verified unless TLS is `none`. Without an SMTP host, email channels cannot be added. Plain text only. - Migration: `alert_channels` and `nodes.presence`, portable (the array column round-trips on both backends). ## Bugs found by running it for real - Validating the form on every keystroke resolved the host name each time; in a container with a slow resolver this froze the page. DNS now happens only on save. - `username: nil` in the SMTP config makes Swoosh raise even with `auth: :never`; credentials are now passed only when given. - Swoosh decides its MIME encoding at compile time: a `_build` made before `gen_smtp` was added fails every send with `BadMapError` until `mix deps.compile swoosh --force` (documented in the README; a clean build is fine). An exception in the mail adapter used to crash the LiveView and the background task before the failure was recorded; deliveries now never raise. - A tampered, non-UUID channel id crashed the page on PostgreSQL only (SQLite tolerates it); malformed ids are now "not found". ## Verification - `mix format --check-formatted`, `mix compile --warnings-as-errors`, `mix credo --strict` (no issues), `mix dialyzer` (0 errors). - `mix test`: 1060 passed on SQLite and on PostgreSQL. The webhook tests run against a real local HTTP server (signatures verified, retries, redirects refused, timeouts, private destinations refused at send time, the Host header kept); the node monitor tests include the concurrent claim. - Migration round trip with data on both backends. - Live smoke on the compose stack with a real SMTP server (Mailpit) and a webhook receiver: a private destination is refused with the default settings; with it allowed, a webhook and an email channel were added in the browser, **Test** delivered to both, and a webhook signature was verified independently with `openssl`. Then real incidents: stopping `whoami` alerted `target.down` on both channels, restarting it `target.up` (a healthy first sighting alerted nothing); stopping the node's Traefik alerted `node.offline` about 2.5 minutes after its last poll, restarting it `node.online`. Each event arrived once per channel. The audit log named the host and no URL, token or address. The dev database was restored afterwards. ## Not covered - TLS verification of an HTTPS webhook through the pinned address (host name kept via Mint's `hostname` option) is not exercised against a real TLS server, and SMTP with STARTTLS or `ssl` against a real certificate was not run (Mailpit ran with TLS `none`).

ryan.mcgee added 1 commit

2026-10-14 17:32:34 +00:00

feat(alerts): alert by webhook and email d398624587

- per-organization channels (signed webhook, email recipient) with
  the events they want: node offline/online, target down/up
- webhook secret generated, shown once, stored encrypted; requests
  carry an HMAC-SHA256 signature over timestamp and body
- destinations that are private or local are refused when saved and
  at every delivery, and the request is pinned to the checked
  address; no redirects, 5 s limit, answers never read
- background delivery with retries; a delivery never raises
- node monitor claims presence changes with conditional updates, so
  one instance alerts; health changes alert from Health.ingest
- /orgs/:org/alerts (admins), audited without destinations; not in
  the API
- SMTP via gen_smtp with certificate verification

ryan.mcgee changed title from x to feat(alerts): alert by webhook and email

2026-10-14 17:32:48 +00:00

ryan.mcgee referenced this pull request

2026-10-14 18:09:57 +00:00

feat(auth): sign in with OpenID Connect providers #14

ryan.mcgee changed target branch from feat/health-maintenance to main

2026-10-15 18:07:57 +00:00

ryan.mcgee added 1 commit

2026-10-15 18:08:01 +00:00

merge main into feat/alerting fddce51ede

ryan.mcgee merged commit 9d284b1442 into main

2026-10-15 18:08:03 +00:00