simon/jolt
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
jolt/test/conformance
History
Yogthos 635bbbcc27 Conformance inc2: fold the 355 inline conformance cases into the corpus 
The corpus only ever saw test/spec/*-spec.janet; the 355 hand-written cases in
test/integration/conformance-test.janet (inline Janet, the lazy-seq / IFn /
destructuring / transducer essentials) were invisible to it and to any non-Janet
runtime. extract-corpus.janet now also pulls that (def cases ...) vector, deduped
by :actual, organized into 41 'conformance / <section>' suites recovered from the
file's ### headers. Corpus 2658 -> 2919 rows (+261 unique).

JVM certification: only 1 new divergence ((/ 2) => 0.5 vs 1/2, the all-double
numeric model) — classified. Chez gates: +1 known host gap (instance? Atom, atom
class identity, Phase 4) allowlisted in both runners; parity rose 2295 -> 2533 on
both, floors raised. Janet gate 155 files 0 failed; certifier green (0 new/stale).

Deferred: 41 non-literal core-async spec rows ((a "src") async-harness wrapper)
need harness context the corpus format doesn't carry — left for inc3.
2026-06-20 10:10:32 -04:00
..
certify-test.janet Conformance inc1: JVM-certify the corpus against reference Clojure 2026-06-20 09:38:31 -04:00
certify.clj Conformance inc1: JVM-certify the corpus against reference Clojure 2026-06-20 09:38:31 -04:00
known-divergences.edn Conformance inc2: fold the 355 inline conformance cases into the corpus 2026-06-20 10:10:32 -04:00
README.md Conformance inc1: JVM-certify the corpus against reference Clojure 2026-06-20 09:38:31 -04:00

README.md

Conformance: certifying the corpus against reference Clojure

The corpus (test/chez/corpus.edn) is jolt's host-neutral behavioral suite — one row per case: {:suite :label :expected :actual}, where :actual is a Clojure source expression and :expected its result (or :throws). Runtime harnesses (test/chez/run-corpus-prelude.janet, run-corpus-zero-janet.janet) replay it on each host and compare by value-equality.

Historically every :expected was hand-written by jolt developers. That makes the corpus a fine regression suite but a weak specification: it certifies jolt against its authors' beliefs, not against Clojure. This directory closes that gap.

What's here

  • certify.clj — runs every corpus row's :actual and :expected through reference JVM Clojure (each in a fresh user namespace, output/stdin sunk, a 5s per-case watchdog) and compares with Clojure's =. It buckets each row:

    • certified / certified-throws — jolt's :expected matches real Clojure
    • divergent — both evaluate but jolt's :expected disagrees with Clojure
    • throws-mismatch — jolt and Clojure disagree on whether it throws
    • jvm-error — :actual isn't runnable on vanilla Clojure (host-coupled / jolt-specific) — informational, not certifiable
    • read-error / timeout — won't read on the JVM reader, or ran too long

  • known-divergences.edn — every current divergence, classified. Most are deliberate jolt-specific or host-model deltas (see :legend): the all-double numeric model, snapshot-heap concurrency, the no-JVM host model, jolt reader features, the jolt printer, intentional strictness. A few are genuine :bug entries with a tracked bead. These categories become the :features flags in conformance inc3.

  • certify-test.janet — gate wrapper. Skips cleanly when clojure (JVM) is not installed; otherwise runs certify.clj and fails the build on a NEW (unclassified) divergence or a stale allowlist entry. Flaky entries (JVM result is timing-dependent, e.g. future-cancel) are tolerated either way.

Running

clojure -M test/conformance/certify.clj                      # gate (exit≠0 on new/stale)
clojure -M test/conformance/certify.clj test/chez/corpus.edn --edn /tmp/report.edn  # full machine-readable report
janet test/conformance/certify-test.janet                    # the gate wrapper

Current state

Of ~2487 vanilla-certifiable rows, >2410 match reference Clojure exactly; the ~70 divergences are all classified (deliberate deltas + 4 tracked bugs). The corpus is trustworthy as a spec, with the host-specific deltas made explicit rather than hidden.

Adding / changing cases

When you add corpus rows or change behavior, re-run the certifier. A NEW divergence means either a real bug (file it, tag the allowlist entry :bug + :bead) or a deliberate delta (classify it). A stale entry means a divergence was fixed — remove it from known-divergences.edn.