jolt/test/conformance/SPEC.md

The jolt conformance spec

This directory defines jolt's behavior as a host-neutral, executable language specification: a data file of cases, certified against reference Clojure, with a feature profile that lets any runtime declare a conformance level. The goal is to make hosting jolt on a new runtime (and proving it correct) a mechanical exercise: read one data file, run each case, compare, report.

The artifacts

File	Role	Generated by
test/chez/corpus.edn	The spec. ~2900 cases of {:suite :label :expected :actual}, :expected sourced from reference JVM Clojure.	test/conformance/regen-corpus.clj
test/conformance/profile.edn	Per-case feature classification — which non-portable cases need which host capability.	certify.clj --profile
test/conformance/known-divergences.edn	The few rows whose JVM value is an opaque host object that can't round-trip to readable source (Java arrays/transients/atoms/beans/proxies print as #object[..@addr]), so the corpus keeps jolt's value.	regen-corpus.clj leftovers, hand-checked
test/conformance/regen-corpus.clj	Sources every :expected from reference JVM Clojure in one process.	—
test/conformance/certify.clj	Certifies :expected against reference JVM Clojure; gates on new/stale divergences; emits the profile.	—

corpus.edn is JVM-sourced: regen-corpus.clj evaluates each case's :actual on reference JVM Clojure and writes the JVM value as :expected. The case list (the :actual strings) comes from test/spec/*-spec.janet via extract-corpus.janet, but the answers are the JVM's. corpus.edn is the canonical contract: it is what every runtime consumes and what certify.clj certifies.

Row schema

{:suite    "numbers / arithmetic"      ; grouping; "<suite> :: <label>" is the case id
 :label    "integer add"               ; unique within a suite
 :actual   "(+ 1 2)"                    ; Clojure source to evaluate
 :expected "3"}                         ; Clojure source whose value it must equal,
                                        ; or the keyword :throws

• [:suite :label] is the canonical, unique case id (the generator disambiguates duplicate labels with (N)).
• Comparison is value-equality (=), never string/printed-form — so map/set iteration order never matters.
• :expected :throws asserts evaluating :actual raises.

The oracle: reference JVM Clojure

Historically every :expected was hand-written. certify.clj removes that weakness: it evaluates every :actual (and :expected) on JVM Clojure in a fresh user namespace and checks jolt's :expected against what real Clojure produces. Of ~2740 vanilla-certifiable rows, >2660 match reference Clojure exactly. The rest are classified (see below) — none are silently wrong.

clojure -M test/conformance/certify.clj                                  # gate
clojure -M test/conformance/certify.clj test/chez/corpus.edn --edn r.edn # + report
clojure -M test/conformance/certify.clj test/chez/corpus.edn --profile test/conformance/profile.edn

The gate fails only on a new (unclassified) divergence or a stale allowlist entry; flaky timing-dependent cases (future-cancel) are tolerated.

Conformance levels & the feature profile

Not every case is portable: some assume a host capability jolt has on one runtime but not another (Java interop, real threads, BigDecimal). profile.edn classifies each non-portable case by the feature it requires. Cases not in the profile are portable — they must pass on any faithful Clojure.

A runtime's conformance level = portable cases + the feature families it implements. Current profile (≈2670 portable, ≈249 non-portable):

Feature	Meaning
:numerics/double-only	all-double numeric model — no Ratio/BigDecimal/float; (/ 1 2) ⇒ 0.5
:concurrency/snapshot	isolated-heap futures/agents/pmap — captured atoms are snapshotted, not shared
:host/jvm-interop	Java classes / instance? on host classes / proxy / bean / definterface
:host/arrays	Java arrays (into-array, int-array, …)
:host/janet	Janet host interop (janet.*)
:async/core-async	clojure.core.async channels/go
:runtime/eval	runtime eval / load-string
:reader/jolt	jolt reader features (#?(:jolt …)) + syntax-quote literal collapse
:printer/jolt	jolt's rendering of transients/atoms/print-method overrides
:strictness/jolt	intentionally stricter (throws on odd assoc! args, etc.)
:impl/representation	representation detail (e.g. syntax-quote yields a list?, not a Cons)
:bug	a known defect (tracked bead) — not a host difference

Hosting jolt on a new runtime

1. Implement the reader + analyzer + a backend for your runtime (see the Chez port under host/chez/ for a worked example).

2. Write a ~30-line harness that, for each corpus row, evaluates :actual and :expected and compares by value-equality (skip :throws rows to an expect-raises check). Pseudocode:

   (doseq [{:keys [suite label actual expected]} (read-edn "test/chez/corpus.edn")]
     (let [feats (profile-features [suite label])]   ; from profile.edn
       (when (subset? feats my-implemented-features)  ; only cases I claim to support
         (record! [suite label]
                  (if (= :throws expected)
                    (raises? actual)
                    (value= (eval actual) (eval expected)))))))
   

3. Run it. Your conformance level is the set of feature families with no failures. Portable-only is the floor; each feature you implement raises it.

The two reference harnesses already do exactly this on Chez: test/chez/run-corpus-prelude.janet (Janet analyzer → Chez runtime) and test/chez/run-corpus-zero-janet.janet (Chez analyzer → Chez runtime), both with a regression floor.

Maintaining the spec

• Add/change cases: edit test/spec/*-spec.janet or conformance-test.janet, then janet test/chez/extract-corpus.janet to regenerate corpus.edn.
• Re-certify: clojure -M test/conformance/certify.clj. A new divergence is either a real bug (file it, mark the allowlist entry :bug + :bead) or a deliberate delta (classify it in known-divergences.edn).
• Refresh the profile: re-run with --profile test/conformance/profile.edn.
• Re-floor the runtime gates when parity rises (run-corpus-*.janet).