simon/jolt

Yogthos f54e99cc08 Conformance inc3: promote the corpus to a documented portable spec

Makes the host-neutral corpus a first-class language specification with
conformance levels, not just a regression suite.

- [suite label] is now a unique, stable case id (extract-corpus disambiguates
  duplicate labels with ' (N)' — one collision existed).
- certify.clj --profile emits test/conformance/profile.edn: every non-portable
  case classified by the host feature it requires (numerics/double-only,
  concurrency/snapshot, host/jvm-interop, host/arrays, host/janet,
  async/core-async, runtime/eval, reader/jolt, printer/jolt, strictness/jolt,
  impl/representation, bug). 2670 of 2919 cases are portable (pass on any faithful
  Clojure); 249 are feature-gated.
- SPEC.md documents the contract: row schema, the JVM oracle, conformance levels,
  the feature vocabulary, and a worked new-runtime harness — so hosting jolt
  elsewhere and proving it correct is read-one-file mechanical.

Janet gate 155 files 0 failed; certify + zero-janet gates green.

2026-06-20 10:21:09 -04:00

3.2 KiB

Raw Blame History

Conformance: certifying the corpus against reference Clojure

See SPEC.md for the full host-neutral language-spec contract: the corpus schema, conformance levels, the feature profile, and how to host jolt on a new runtime. This README covers the certification tooling specifically.

The corpus (test/chez/corpus.edn) is jolt's host-neutral behavioral suite — one row per case: {:suite :label :expected :actual}, where :actual is a Clojure source expression and :expected its result (or :throws). Runtime harnesses (test/chez/run-corpus-prelude.janet, run-corpus-zero-janet.janet) replay it on each host and compare by value-equality.

Historically every :expected was hand-written by jolt developers. That makes the corpus a fine regression suite but a weak specification: it certifies jolt against its authors' beliefs, not against Clojure. This directory closes that gap.

What's here

certify.clj — runs every corpus row's :actual and :expected through reference JVM Clojure (each in a fresh user namespace, output/stdin sunk, a 5s per-case watchdog) and compares with Clojure's =. It buckets each row:
- certified / certified-throws — jolt's :expected matches real Clojure
- divergent — both evaluate but jolt's :expected disagrees with Clojure
- throws-mismatch — jolt and Clojure disagree on whether it throws
- jvm-error — :actual isn't runnable on vanilla Clojure (host-coupled / jolt-specific) — informational, not certifiable
- read-error / timeout — won't read on the JVM reader, or ran too long
known-divergences.edn — every current divergence, classified. Most are deliberate jolt-specific or host-model deltas (see :legend): the all-double numeric model, snapshot-heap concurrency, the no-JVM host model, jolt reader features, the jolt printer, intentional strictness. A few are genuine :bug entries with a tracked bead. These categories become the :features flags in conformance inc3.
certify-test.janet — gate wrapper. Skips cleanly when clojure (JVM) is not installed; otherwise runs certify.clj and fails the build on a NEW (unclassified) divergence or a stale allowlist entry. Flaky entries (JVM result is timing-dependent, e.g. future-cancel) are tolerated either way.

Running

clojure -M test/conformance/certify.clj                      # gate (exit≠0 on new/stale)
clojure -M test/conformance/certify.clj test/chez/corpus.edn --edn /tmp/report.edn  # full machine-readable report
janet test/conformance/certify-test.janet                    # the gate wrapper

Current state

Of ~2487 vanilla-certifiable rows, >2410 match reference Clojure exactly; the ~70 divergences are all classified (deliberate deltas + 4 tracked bugs). The corpus is trustworthy as a spec, with the host-specific deltas made explicit rather than hidden.

Adding / changing cases

When you add corpus rows or change behavior, re-run the certifier. A NEW divergence means either a real bug (file it, tag the allowlist entry :bug + :bead) or a deliberate delta (classify it). A stale entry means a divergence was fixed — remove it from known-divergences.edn.

3.2 KiB Raw Blame History