simon/jolt
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test/ci: make clojure-test-suite per-file deadline configurable (JOLT_SUITE_TIMEOUT)

Browse source 
The 6s per-file deadline + baseline=3981 were tuned to a fast dev machine; a
slower CI runner could time out a sub-second finite file, dropping total-pass
below baseline and flaking CI red. The deadline is now an env var (default 6
locally); CI sets it to 20s for headroom.

Characterized the 6 timeouts: identical pass/timeout counts at 6s and 20s, so all
6 are genuinely-infinite hangs (killed at any deadline) and the 227 finite files
finish well under 6s. So no baseline margin is needed — a generous CI deadline
removes the flake risk while preserving full regression sensitivity.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Yogthos 2026-06-08 19:13:30 -04:00
parent 3e52c90532
commit 597f76a753
2 changed files with 13 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

5
.github/workflows/tests.yml vendored
View file

@ -10,6 +10,11 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
env: env:
JANET_VERSION: v1.41.2 # bump to match the version Jolt is developed against JANET_VERSION: v1.41.2 # bump to match the version Jolt is developed against
# Per-file deadline for the clojure-test-suite battery. Finite files finish
# in well under 1s; the genuinely-infinite ones get killed at any deadline.
# A generous value gives slow CI runners headroom so a sub-second file
# spiking doesn't time out and drop total-pass below the baseline.
JOLT_SUITE_TIMEOUT: "20"
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:

11
test/integration/clojure-test-suite-test.janet
View file

@ -41,9 +41,14 @@
(def baseline-pass 3981) (def baseline-pass 3981)
# A file is "clean" when it ran with zero failures AND zero errors. # A file is "clean" when it ran with zero failures AND zero errors.
(def baseline-clean-files 66) (def baseline-clean-files 66)
# Per-file wall-clock budget (seconds). Normal files finish in well under 1s; # Per-file wall-clock budget (seconds). Normal files finish in well under 1s, so
# this only fires on infinite-sequence hangs. # this normally only fires on genuinely-infinite-sequence hangs. It's an env var
(def per-file-timeout 6) # (JOLT_SUITE_TIMEOUT) so CI — whose runners are slower than a dev machine — can
# give slow-but-finite files generous headroom without timing them out (which
# would drop total-pass below the baseline and flake CI red). Default 6 locally.
(def per-file-timeout
(let [e (os/getenv "JOLT_SUITE_TIMEOUT")]
(or (and e (scan-number e)) 6)))
(defn- walk [dir acc] (defn- walk [dir acc]
(each e (os/dir dir) (each e (os/dir dir)