White_Ghost_Cyber_Systems/traefik-control
3
0
Fork 0
Code Issues 1 Pull requests 1 Projects Releases 4 Packages 2 Wiki Activity Actions

test: stop alert and discovery tests flaking under load #25

Merged
ryan.mcgee merged 1 commit from fix/test-webhook-timeouts into main 2026-10-15 21:30:55 +00:00
Conversation 0 Commits 1 Files changed 5 +82 -37
ryan.mcgee commented 2026-10-15 21:23:52 +00:00
Owner
Copy link

Summary

  • The PostgreSQL job of the push run on main (run 7) failed once: AlertsDeliveryTest "sends each event to each channel that wants it" got the email but not the webhook within ExUnit's default 100 ms (log read from the runner). It passed on 4 local reruns and on the PR run, so it is a timing flake of the test, not of the code.
  • Reproducing under CPU load (24 busy loops) also showed the older SQLite Database busy failures in about half of the runs of AlertsDeliveryTest, and later in other modules.

Changes

  • alerts_delivery_test.exs, alert_live_test.exs: webhook and email deliveries made by other processes over real HTTP are awaited for up to 2 seconds.
  • DataCase: before the sandbox owner stops, wait (one 5 second deadline) for the alert and mail task supervisors' children. A task still writing when its owner went away left SQLite holding a write lock; the next test's first insert failed with Database busy.
  • config/database.exs: SQLite busy_timeout: 15_000 in the test configuration only.
  • discovery_test.exs (from #23): no more sleeps and time windows. TTL and retry waits are driven with lifetimes of zero and the key-refresh limit by its lower bound only.

Verification

  • Under 24 busy loops, alerts_delivery_test.exs six runs in a row: before, about half of the runs had a failure; with the drain, 6 of 6 passed. A loaded full-suite run with the busy timeout had no Database busy in about 2600 tests.
  • The changed tests under 8 busy loops: 96, 39 and 39 passed.
  • format, compile with warnings as errors, credo strict, dialyzer: clean. mix test: 1508 passed on SQLite and on PostgreSQL.

Not resolved

  • One full SQLite run during this work (1505 of 1506) failed with a test I could not identify, and I could not reproduce it in 10 further runs. If CI shows a failure, its log names the test.
## Summary - The PostgreSQL job of the push run on `main` (run 7) failed once: `AlertsDeliveryTest` "sends each event to each channel that wants it" got the email but not the webhook within ExUnit's default 100 ms (log read from the runner). It passed on 4 local reruns and on the PR run, so it is a timing flake of the test, not of the code. - Reproducing under CPU load (24 busy loops) also showed the older SQLite `Database busy` failures in about half of the runs of `AlertsDeliveryTest`, and later in other modules. ## Changes - `alerts_delivery_test.exs`, `alert_live_test.exs`: webhook and email deliveries made by other processes over real HTTP are awaited for up to 2 seconds. - `DataCase`: before the sandbox owner stops, wait (one 5 second deadline) for the alert and mail task supervisors' children. A task still writing when its owner went away left SQLite holding a write lock; the next test's first insert failed with `Database busy`. - `config/database.exs`: SQLite `busy_timeout: 15_000` in the test configuration only. - `discovery_test.exs` (from #23): no more sleeps and time windows. TTL and retry waits are driven with lifetimes of zero and the key-refresh limit by its lower bound only. ## Verification - Under 24 busy loops, `alerts_delivery_test.exs` six runs in a row: before, about half of the runs had a failure; with the drain, 6 of 6 passed. A loaded full-suite run with the busy timeout had no `Database busy` in about 2600 tests. - The changed tests under 8 busy loops: 96, 39 and 39 passed. - format, compile with warnings as errors, credo strict, dialyzer: clean. `mix test`: 1508 passed on SQLite and on PostgreSQL. ## Not resolved - One full SQLite run during this work (1505 of 1506) failed with a test I could not identify, and I could not reproduce it in 10 further runs. If CI shows a failure, its log names the test.
test: stop alert and discovery tests flaking under load
All checks were successful
CI / format, compile, credo, dialyzer (pull_request) Successful in 1m21s
Details
CI / tests (PostgreSQL) (pull_request) Successful in 3m57s
Details
CI / tests (SQLite) (pull_request) Successful in 2m13s
Details
CI / dependency audit (pull_request) Successful in 41s
Details
c3a536a961 
- wait up to 2 seconds for webhook deliveries made over real HTTP
  instead of ExUnit's 100 ms default (failed the PostgreSQL run of
  main once on the CI runner)
- let background tasks (alert deliveries, mail) finish before the
  sandbox owner stops, so SQLite is not left holding a write lock
  that fails the next test with Database busy
- raise the SQLite busy_timeout for tests to 15 seconds
- make the discovery cache tests independent of the clock
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
White_Ghost_Cyber_Systems/traefik-control!25
No description provided.