Conformance inc1: JVM-certify the corpus against reference Clojure
The corpus carried hand-written :expected values — a regression suite but a weak spec (it checked jolt against its authors, not against Clojure). certify.clj runs every corpus row's :actual and :expected through reference JVM Clojure 1.12.5 (fresh user ns per case, output/stdin sunk, 5s per-case watchdog) and compares with =. Result: of ~2487 vanilla-certifiable rows, 2416 match real Clojure exactly. The 71 divergences are all classified in known-divergences.edn — mostly deliberate jolt-specific/host-model deltas (all-double numerics, snapshot concurrency, no-JVM host model, jolt reader features, printer, strictness), plus 4 genuine bugs filed as beads (jolt-l8e8 ex-message, jolt-hc35 munge, jolt-pqio print-nil, jolt-2507 bounded-count). certify-test.janet gates it: skips without clojure on PATH, else fails only on a NEW (unclassified) divergence or stale allowlist entry; flaky timing-dependent cases (future-cancel) tolerated either way. Full gate 155 files 0 failed.
This commit is contained in:
parent
d0fce540ed
commit
b520eefa7a
4 changed files with 528 additions and 0 deletions
57
test/conformance/README.md
Normal file
57
test/conformance/README.md
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
# Conformance: certifying the corpus against reference Clojure
|
||||
|
||||
The corpus (`test/chez/corpus.edn`) is jolt's host-neutral behavioral suite — one
|
||||
row per case: `{:suite :label :expected :actual}`, where `:actual` is a Clojure
|
||||
source expression and `:expected` its result (or `:throws`). Runtime harnesses
|
||||
(`test/chez/run-corpus-prelude.janet`, `run-corpus-zero-janet.janet`) replay it on
|
||||
each host and compare by value-equality.
|
||||
|
||||
Historically every `:expected` was **hand-written by jolt developers**. That makes
|
||||
the corpus a fine regression suite but a weak *specification*: it certifies jolt
|
||||
against its authors' beliefs, not against Clojure. This directory closes that gap.
|
||||
|
||||
## What's here
|
||||
|
||||
- **`certify.clj`** — runs every corpus row's `:actual` and `:expected` through
|
||||
**reference JVM Clojure** (each in a fresh `user` namespace, output/stdin sunk, a
|
||||
5s per-case watchdog) and compares with Clojure's `=`. It buckets each row:
|
||||
- `certified` / `certified-throws` — jolt's `:expected` matches real Clojure
|
||||
- `divergent` — both evaluate but jolt's `:expected` disagrees with Clojure
|
||||
- `throws-mismatch` — jolt and Clojure disagree on whether it throws
|
||||
- `jvm-error` — `:actual` isn't runnable on vanilla Clojure (host-coupled /
|
||||
jolt-specific) — informational, not certifiable
|
||||
- `read-error` / `timeout` — won't read on the JVM reader, or ran too long
|
||||
|
||||
- **`known-divergences.edn`** — every current divergence, classified. Most are
|
||||
**deliberate** jolt-specific or host-model deltas (see `:legend`): the all-double
|
||||
numeric model, snapshot-heap concurrency, the no-JVM host model, jolt reader
|
||||
features, the jolt printer, intentional strictness. A few are genuine **`:bug`**
|
||||
entries with a tracked bead. These categories become the `:features` flags in
|
||||
conformance inc3.
|
||||
|
||||
- **`certify-test.janet`** — gate wrapper. Skips cleanly when `clojure` (JVM) is
|
||||
not installed; otherwise runs `certify.clj` and fails the build on a **NEW**
|
||||
(unclassified) divergence or a **stale** allowlist entry. Flaky entries (JVM
|
||||
result is timing-dependent, e.g. `future-cancel`) are tolerated either way.
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
clojure -M test/conformance/certify.clj # gate (exit≠0 on new/stale)
|
||||
clojure -M test/conformance/certify.clj test/chez/corpus.edn --edn /tmp/report.edn # full machine-readable report
|
||||
janet test/conformance/certify-test.janet # the gate wrapper
|
||||
```
|
||||
|
||||
## Current state
|
||||
|
||||
Of ~2487 vanilla-certifiable rows, **>2410 match reference Clojure exactly**; the
|
||||
~70 divergences are all classified (deliberate deltas + 4 tracked bugs). The corpus
|
||||
is trustworthy as a spec, with the host-specific deltas made explicit rather than
|
||||
hidden.
|
||||
|
||||
## Adding / changing cases
|
||||
|
||||
When you add corpus rows or change behavior, re-run the certifier. A NEW divergence
|
||||
means either a real bug (file it, tag the allowlist entry `:bug` + `:bead`) or a
|
||||
deliberate delta (classify it). A stale entry means a divergence was fixed — remove
|
||||
it from `known-divergences.edn`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue