feat(kubernetes): add probes and pod clustering #34

Merged

ryan.mcgee merged 1 commit from feat/kubernetes into main

2026-10-21 13:54:01 +00:00

ryan.mcgee commented

2026-10-21 13:45:23 +00:00

Owner

Summary

The images had no health endpoints (prod force_ssl redirects a probe to the pod address), no way to cluster pods, and an always-on Erlang distribution with a cookie published in the image.
A review of the process state showed the application is already safe with several replicas (throttling, target health and the audit log are in the database; the node monitor claims each presence change atomically; pruning is idempotent), so no job needed a cluster-wide singleton. The one gap is PubSub, which stays per pod until pods cluster.

Changes

TraefikControlWeb.HealthPlug, mounted first in the endpoint: GET/HEAD /healthz is 200 while the process serves; /readyz runs SELECT 1 (2 s timeout) and is 200 or 503. Excluded from force_ssl in config/prod.exs, not logged, no session.
rel/env.sh.eex: distribution only with both RELEASE_COOKIE and POD_IP (node traefik_control@$POD_IP, port DIST_PORT, default 9100, applied only to start/daemon so the rpc helper VM does not collide); otherwise RELEASE_DISTRIBUTION=none. Behaviour change: images used to start distribution with the baked-in cookie; now bin/traefik_control rpc/remote need the two variables.
Containerfile: copy rel/, RELEASE_TMP=/tmp for a read-only root filesystem.
scripts/propose-release.sh and README: rel/ counts as shipped. New Kubernetes section and the new variables in the Configuration table.
Tests for the plug.

Verification

Release image (postgres) in podman, probed from a sibling container over plain HTTP to the pod IP: /healthz 200, /readyz 200, /users/log-in 301, zero probe lines in the log; with the database stopped /readyz 503 and /healthz 200; recovers to 200.
Two containers (--read-only --tmpfs /tmp --cap-drop ALL, fixed IPs, shared DNS alias): each lists the other in Node.list(); a PubSub broadcast from the peer reaches a local subscriber; a third container without a cookie boots, serves, refuses rpc and does not join; stopping one pod leaves the other ready with an empty peer list. A first run found the rpc port collision, fixed above.
SQLite image: SIGTERM received - shutting down, exit code 0.
SQLite and PostgreSQL: mix format --check-formatted, compile --warnings-as-errors, credo --strict (found an alias suggestion, fixed), mix test 1515 passed on both, migration rollback/migrate round trip on PostgreSQL, mix dialyzer clean; shellcheck clean.
Not run: a real Kubernetes cluster. k3s does not start in rootless podman on this host (kubelet cannot create kubepods in the cgroup v2 tree), so probes, rolling updates and the chart are covered by the podman runs above and by rendering the chart and validating it with kubeconform (see the chart repository).

Release

This is a feat, so the weekly job proposes a minor bump (1.1.0 from 1.0.0). The chart pins that version.

## Summary - The images had no health endpoints (prod `force_ssl` redirects a probe to the pod address), no way to cluster pods, and an always-on Erlang distribution with a cookie published in the image. - A review of the process state showed the application is already safe with several replicas (throttling, target health and the audit log are in the database; the node monitor claims each presence change atomically; pruning is idempotent), so no job needed a cluster-wide singleton. The one gap is PubSub, which stays per pod until pods cluster. ## Changes - `TraefikControlWeb.HealthPlug`, mounted first in the endpoint: `GET/HEAD /healthz` is `200` while the process serves; `/readyz` runs `SELECT 1` (2 s timeout) and is `200` or `503`. Excluded from `force_ssl` in `config/prod.exs`, not logged, no session. - `rel/env.sh.eex`: distribution only with both `RELEASE_COOKIE` and `POD_IP` (node `traefik_control@$POD_IP`, port `DIST_PORT`, default 9100, applied only to `start`/`daemon` so the `rpc` helper VM does not collide); otherwise `RELEASE_DISTRIBUTION=none`. **Behaviour change:** images used to start distribution with the baked-in cookie; now `bin/traefik_control rpc`/`remote` need the two variables. - `Containerfile`: copy `rel/`, `RELEASE_TMP=/tmp` for a read-only root filesystem. - `scripts/propose-release.sh` and README: `rel/` counts as shipped. New Kubernetes section and the new variables in the Configuration table. - Tests for the plug. ## Verification - Release image (postgres) in podman, probed from a sibling container over plain HTTP to the pod IP: `/healthz` 200, `/readyz` 200, `/users/log-in` 301, zero probe lines in the log; with the database stopped `/readyz` 503 and `/healthz` 200; recovers to 200. - Two containers (`--read-only --tmpfs /tmp --cap-drop ALL`, fixed IPs, shared DNS alias): each lists the other in `Node.list()`; a PubSub broadcast from the peer reaches a local subscriber; a third container without a cookie boots, serves, refuses `rpc` and does not join; stopping one pod leaves the other ready with an empty peer list. A first run found the `rpc` port collision, fixed above. - SQLite image: `SIGTERM received - shutting down`, exit code 0. - SQLite and PostgreSQL: `mix format --check-formatted`, `compile --warnings-as-errors`, `credo --strict` (found an alias suggestion, fixed), `mix test` 1515 passed on both, migration rollback/migrate round trip on PostgreSQL, `mix dialyzer` clean; `shellcheck` clean. - Not run: a real Kubernetes cluster. k3s does not start in rootless podman on this host (kubelet cannot create `kubepods` in the cgroup v2 tree), so probes, rolling updates and the chart are covered by the podman runs above and by rendering the chart and validating it with kubeconform (see the chart repository). ## Release This is a `feat`, so the weekly job proposes a minor bump (1.1.0 from 1.0.0). The chart pins that version.

ryan.mcgee added 1 commit

2026-10-21 13:45:23 +00:00

feat(kubernetes): add probes and pod clustering

CI / format, compile, credo, dialyzer (pull_request) Successful in 1m40s

Details

CI / tests (SQLite) (pull_request) Successful in 2m18s

Details

CI / dependency audit (pull_request) Successful in 39s

Details

CI / tests (PostgreSQL) (pull_request) Successful in 2m13s

Details

Development images / images (push) Successful in 4m43s

Details

fc4e565e97

- answer /healthz (liveness) and /readyz (database) before the
  rest of the endpoint, outside the https redirect and the logs
- start Erlang distribution only when RELEASE_COOKIE and POD_IP
  are set, so pods can join through DNS_CLUSTER_QUERY and share
  PubSub; otherwise it is off instead of using the image's cookie
- keep the fixed distribution port off rpc and remote helpers
- run on a read-only root filesystem (RELEASE_TMP=/tmp)
- count rel/ as shipped for weekly releases
- document probes, replicas and clustering

ryan.mcgee referenced this pull request from White_Ghost_Cyber_Systems/helm

2026-10-21 13:46:33 +00:00

feat(traefik-control): add traefik-control chart #1

ryan.mcgee merged commit a377feb70c into main

2026-10-21 13:54:01 +00:00

ryan.mcgee deleted branch feat/kubernetes

2026-10-21 13:54:01 +00:00

ryan.mcgee referenced this pull request from a commit

2026-10-21 13:54:02 +00:00

feat(kubernetes): add probes and pod clustering (#34)