simon/jolt
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test/ci: make clojure-test-suite per-file deadline configurable (JOLT_SUITE_TIMEOUT)

Browse source 
The 6s per-file deadline + baseline=3981 were tuned to a fast dev machine; a
slower CI runner could time out a sub-second finite file, dropping total-pass
below baseline and flaking CI red. The deadline is now an env var (default 6
locally); CI sets it to 20s for headroom.

Characterized the 6 timeouts: identical pass/timeout counts at 6s and 20s, so all
6 are genuinely-infinite hangs (killed at any deadline) and the 227 finite files
finish well under 6s. So no baseline margin is needed — a generous CI deadline
removes the flake risk while preserving full regression sensitivity.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yogthos 2026-06-08 19:13:30 -04:00
parent 3e52c90532
commit 597f76a753
2 changed files with 13 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

11
test/integration/clojure-test-suite-test.janet
View file

@ -41,9 +41,14 @@
(def baseline-pass 3981)
# A file is "clean" when it ran with zero failures AND zero errors.
(def baseline-clean-files 66)
# Per-file wall-clock budget (seconds). Normal files finish in well under 1s;
# this only fires on infinite-sequence hangs.
(def per-file-timeout 6)
# Per-file wall-clock budget (seconds). Normal files finish in well under 1s, so
# this normally only fires on genuinely-infinite-sequence hangs. It's an env var
# (JOLT_SUITE_TIMEOUT) so CI — whose runners are slower than a dev machine — can
# give slow-but-finite files generous headroom without timing them out (which
# would drop total-pass below the baseline and flake CI red). Default 6 locally.
(def per-file-timeout
(let [e (os/getenv "JOLT_SUITE_TIMEOUT")]
(or (and e (scan-number e)) 6)))
(defn- walk [dir acc]
(each e (os/dir dir)