simon/jolt
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
jolt/test/conformance
History
Yogthos da775802d6 Source the conformance corpus from JVM Clojure; retire the prelude gate 
corpus.edn :expected is now the value reference JVM Clojure produces, set by the
new test/conformance/regen-corpus.clj (one JVM process, per-row thread watchdog).
167 rows moved to the JVM value: ratios (/ 1 2)=>1/2, doubles (double 3)=>3.0,
shared-heap concurrency (the future/pmap/agent cases), clojure.math doubles. The
JVM is the spec; jolt is measured against it.

known-divergences.edn shrinks to the rows whose JVM value is an opaque host object
that can't round-trip to source (Java arrays, transients, atoms, beans, proxies,
chunks all print as #object[..@addr]) plus (fn* foo) and a few racy concurrency
cases (:flaky). The zero-janet gate's allowlist becomes the set of host gaps vs the
JVM spec (no Class/array/BigDecimal, :jolt reader, jolt's own printing).

Math/clojure.math sqrt/pow/floor/trig now return doubles (Chez returns exact for
exact args, e.g. (sqrt 9)=>3); JVM always returns a double.

extract-corpus.janet no longer writes corpus.edn unless asked (the test runner
imported it and was silently overwriting the JVM corpus with the spec sources'
placeholder answers). The prelude parity gate is deleted — the zero-janet spine +
certify.clj are the oracles.

zero-janet 2678 (0 new divergences), certify 0 new / 0 stale, emit-test 330/330.
2026-06-21 01:45:04 -04:00
..
certify-test.janet Conformance inc1: JVM-certify the corpus against reference Clojure 2026-06-20 09:38:31 -04:00
certify.clj Conformance inc3: promote the corpus to a documented portable spec 2026-06-20 10:21:09 -04:00
known-divergences.edn Source the conformance corpus from JVM Clojure; retire the prelude gate 2026-06-21 01:45:04 -04:00
profile.edn Fix 4 clojure.core bugs surfaced by JVM certification 2026-06-20 11:06:33 -04:00
README.md Conformance inc3: promote the corpus to a documented portable spec 2026-06-20 10:21:09 -04:00
regen-corpus.clj Source the conformance corpus from JVM Clojure; retire the prelude gate 2026-06-21 01:45:04 -04:00
SPEC.md Source the conformance corpus from JVM Clojure; retire the prelude gate 2026-06-21 01:45:04 -04:00

README.md

Conformance: certifying the corpus against reference Clojure

See SPEC.md for the full host-neutral language-spec contract: the corpus schema, conformance levels, the feature profile, and how to host jolt on a new runtime. This README covers the certification tooling specifically.

The corpus (test/chez/corpus.edn) is jolt's host-neutral behavioral suite — one row per case: {:suite :label :expected :actual}, where :actual is a Clojure source expression and :expected its result (or :throws). Runtime harnesses (test/chez/run-corpus-prelude.janet, run-corpus-zero-janet.janet) replay it on each host and compare by value-equality.

Historically every :expected was hand-written by jolt developers. That makes the corpus a fine regression suite but a weak specification: it certifies jolt against its authors' beliefs, not against Clojure. This directory closes that gap.

What's here

  • certify.clj — runs every corpus row's :actual and :expected through reference JVM Clojure (each in a fresh user namespace, output/stdin sunk, a 5s per-case watchdog) and compares with Clojure's =. It buckets each row:

    • certified / certified-throws — jolt's :expected matches real Clojure
    • divergent — both evaluate but jolt's :expected disagrees with Clojure
    • throws-mismatch — jolt and Clojure disagree on whether it throws
    • jvm-error — :actual isn't runnable on vanilla Clojure (host-coupled / jolt-specific) — informational, not certifiable
    • read-error / timeout — won't read on the JVM reader, or ran too long

  • known-divergences.edn — every current divergence, classified. Most are deliberate jolt-specific or host-model deltas (see :legend): the all-double numeric model, snapshot-heap concurrency, the no-JVM host model, jolt reader features, the jolt printer, intentional strictness. A few are genuine :bug entries with a tracked bead. These categories become the :features flags in conformance inc3.

  • certify-test.janet — gate wrapper. Skips cleanly when clojure (JVM) is not installed; otherwise runs certify.clj and fails the build on a NEW (unclassified) divergence or a stale allowlist entry. Flaky entries (JVM result is timing-dependent, e.g. future-cancel) are tolerated either way.

Running

clojure -M test/conformance/certify.clj                      # gate (exit≠0 on new/stale)
clojure -M test/conformance/certify.clj test/chez/corpus.edn --edn /tmp/report.edn  # full machine-readable report
janet test/conformance/certify-test.janet                    # the gate wrapper

Current state

Of ~2487 vanilla-certifiable rows, >2410 match reference Clojure exactly; the ~70 divergences are all classified (deliberate deltas + 4 tracked bugs). The corpus is trustworthy as a spec, with the host-specific deltas made explicit rather than hidden.

Adding / changing cases

When you add corpus rows or change behavior, re-run the certifier. A NEW divergence means either a real bug (file it, tag the allowlist entry :bug + :bead) or a deliberate delta (classify it). A stale entry means a divergence was fixed — remove it from known-divergences.edn.