simon/jolt
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
jolt/test/conformance
History
Yogthos 6abbea3835 Fix 4 clojure.core bugs surfaced by JVM certification 
The corpus certifier (test/conformance) flagged four cases where jolt's
hand-written :expected matched a real defect rather than Clojure. Fixed in the
jolt-core overlay, corrected the spec :expected, re-certified against JVM Clojure:

- ex-message: returns nil for a non-throwable (dropped the lenient string branch);
  still returns the message for ex-info. (jolt-l8e8)
- munge: preserves the argument's type — a symbol munges to a symbol, not a string.
  (jolt-hc35)
- print: (print nil) emits "nil", not "" (top-level nil guard; str yields "").
  (jolt-pqio)
- bounded-count: uses the counted? fast path (full count), else counts up to n via
  seq — was (min n (count coll)), wrong for counted colls. Added an uncounted-coll
  spec case. (jolt-2507)

Removed the 4 :bug entries from known-divergences.edn (now certified), regenerated
corpus + profile, re-minted the Chez bootstrap seed (clojure.core changed). Gates:
Janet 155/0, JVM certify clean, both Chez corpus gates 2534 (floors raised),
bootstrap 6/6, fixpoint intact.
2026-06-20 11:06:33 -04:00
..
certify-test.janet Conformance inc1: JVM-certify the corpus against reference Clojure 2026-06-20 09:38:31 -04:00
certify.clj Conformance inc3: promote the corpus to a documented portable spec 2026-06-20 10:21:09 -04:00
known-divergences.edn Fix 4 clojure.core bugs surfaced by JVM certification 2026-06-20 11:06:33 -04:00
profile.edn Fix 4 clojure.core bugs surfaced by JVM certification 2026-06-20 11:06:33 -04:00
README.md Conformance inc3: promote the corpus to a documented portable spec 2026-06-20 10:21:09 -04:00
SPEC.md Conformance inc3: promote the corpus to a documented portable spec 2026-06-20 10:21:09 -04:00

README.md

Conformance: certifying the corpus against reference Clojure

See SPEC.md for the full host-neutral language-spec contract: the corpus schema, conformance levels, the feature profile, and how to host jolt on a new runtime. This README covers the certification tooling specifically.

The corpus (test/chez/corpus.edn) is jolt's host-neutral behavioral suite — one row per case: {:suite :label :expected :actual}, where :actual is a Clojure source expression and :expected its result (or :throws). Runtime harnesses (test/chez/run-corpus-prelude.janet, run-corpus-zero-janet.janet) replay it on each host and compare by value-equality.

Historically every :expected was hand-written by jolt developers. That makes the corpus a fine regression suite but a weak specification: it certifies jolt against its authors' beliefs, not against Clojure. This directory closes that gap.

What's here

  • certify.clj — runs every corpus row's :actual and :expected through reference JVM Clojure (each in a fresh user namespace, output/stdin sunk, a 5s per-case watchdog) and compares with Clojure's =. It buckets each row:

    • certified / certified-throws — jolt's :expected matches real Clojure
    • divergent — both evaluate but jolt's :expected disagrees with Clojure
    • throws-mismatch — jolt and Clojure disagree on whether it throws
    • jvm-error — :actual isn't runnable on vanilla Clojure (host-coupled / jolt-specific) — informational, not certifiable
    • read-error / timeout — won't read on the JVM reader, or ran too long

  • known-divergences.edn — every current divergence, classified. Most are deliberate jolt-specific or host-model deltas (see :legend): the all-double numeric model, snapshot-heap concurrency, the no-JVM host model, jolt reader features, the jolt printer, intentional strictness. A few are genuine :bug entries with a tracked bead. These categories become the :features flags in conformance inc3.

  • certify-test.janet — gate wrapper. Skips cleanly when clojure (JVM) is not installed; otherwise runs certify.clj and fails the build on a NEW (unclassified) divergence or a stale allowlist entry. Flaky entries (JVM result is timing-dependent, e.g. future-cancel) are tolerated either way.

Running

clojure -M test/conformance/certify.clj                      # gate (exit≠0 on new/stale)
clojure -M test/conformance/certify.clj test/chez/corpus.edn --edn /tmp/report.edn  # full machine-readable report
janet test/conformance/certify-test.janet                    # the gate wrapper

Current state

Of ~2487 vanilla-certifiable rows, >2410 match reference Clojure exactly; the ~70 divergences are all classified (deliberate deltas + 4 tracked bugs). The corpus is trustworthy as a spec, with the host-specific deltas made explicit rather than hidden.

Adding / changing cases

When you add corpus rows or change behavior, re-run the certifier. A NEW divergence means either a real bug (file it, tag the allowlist entry :bug + :bead) or a deliberate delta (classify it). A stale entry means a divergence was fixed — remove it from known-divergences.edn.