jolt/test/conformance/SPEC.md

The jolt conformance spec

This directory defines jolt's behavior as a host-neutral, executable language specification: a data file of cases, certified against reference Clojure, with a feature profile that lets any runtime declare a conformance level. The goal is to make hosting jolt on a new runtime (and proving it correct) a mechanical exercise: read one data file, run each case, compare, report.

The artifacts

File	Role	Generated by
test/chez/corpus.edn	The spec. ~2900 cases of {:suite :label :expected :actual}, :expected sourced from reference JVM Clojure.	test/conformance/regen-corpus.clj
test/conformance/profile.edn	Per-case feature classification — which non-portable cases need which host capability.	certify.clj --profile
test/conformance/known-divergences.edn	The few rows whose JVM value is an opaque host object that can't round-trip to readable source (Java arrays/transients/atoms/beans/proxies print as #object[..@addr]), so the corpus keeps jolt's value.	regen-corpus.clj leftovers, hand-checked
test/conformance/regen-corpus.clj	Sources every :expected from reference JVM Clojure in one process.	—
test/conformance/certify.clj	Certifies :expected against reference JVM Clojure; gates on new/stale divergences; emits the profile.	—

corpus.edn is JVM-sourced: regen-corpus.clj evaluates each case's :actual on reference JVM Clojure and writes the JVM value as :expected. corpus.edn is the canonical, frozen contract: it is what every runtime consumes, what certify.clj certifies, and where new cases are authored directly.

Row schema

{:suite    "numbers / arithmetic"      ; grouping; "<suite> :: <label>" is the case id
 :label    "integer add"               ; unique within a suite
 :actual   "(+ 1 2)"                    ; Clojure source to evaluate
 :expected "3"}                         ; Clojure source whose value it must equal,
                                        ; or the keyword :throws

• [:suite :label] is the canonical, unique case id (the generator disambiguates duplicate labels with (N)).
• Comparison is value-equality (=), never string/printed-form — so map/set iteration order never matters.
• Because comparison is =, a type or laziness difference is invisible to a plain value row: (= [0 1] '(0 1)) is true, so a fn returning a vector where Clojure returns a seq still passes. Pin those explicitly — container/element type with a predicate row ((seq? …), (vector? …), (every? seq? …)), and laziness with a (take n (… (range))) row over an infinite source (it hangs, not just diverges, if the fn isn't lazy). The seq / lazy over infinite suite does both.
• :expected :throws asserts evaluating :actual raises.

The oracle: reference JVM Clojure

Historically every :expected was hand-written. certify.clj removes that weakness: it evaluates every :actual (and :expected) on JVM Clojure in a fresh user namespace and checks jolt's :expected against what real Clojure produces. Of ~2740 vanilla-certifiable rows, >2730 match reference Clojure exactly. The rest are classified (see below) — none are silently wrong.

clojure -M test/conformance/certify.clj                                  # gate
clojure -M test/conformance/certify.clj test/chez/corpus.edn --edn r.edn # + report
clojure -M test/conformance/certify.clj test/chez/corpus.edn --profile test/conformance/profile.edn

The gate fails only on a new (unclassified) divergence or a stale allowlist entry; flaky timing-dependent cases (future-cancel) are tolerated.

Conformance levels & the feature profile

Not every case is portable: some assume a host capability jolt has on one runtime but not another (Java interop, real threads, BigDecimal). profile.edn classifies each non-portable case by the feature it requires. Cases not in the profile are portable — they must pass on any faithful Clojure.

A runtime's conformance level = portable cases + the feature families it implements. Current profile (≈2735 portable, ≈167 non-portable):

Feature	Meaning
:numerics/double-only	all-double numeric model — no Ratio/BigDecimal/float; (/ 1 2) ⇒ 0.5
:concurrency/snapshot	isolated-heap futures/agents/pmap — captured atoms are snapshotted, not shared
:host/jvm-interop	Java classes / instance? on host classes / proxy / bean / definterface
:host/arrays	Java arrays (into-array, int-array, …)
:async/core-async	clojure.core.async channels/go
:runtime/eval	runtime eval / load-string
:reader/jolt	jolt reader features (#?(:jolt …)) + syntax-quote literal collapse
:printer/jolt	jolt's rendering of transients/atoms/print-method overrides
:strictness/jolt	intentionally stricter (throws on odd assoc! args, etc.)
:impl/representation	representation detail (e.g. syntax-quote yields a list?, not a Cons)
:bug	a known defect (tracked bead) — not a host difference

Seq semantics

Values alone don't pin laziness — an eager map and a lazy map return the same elements. The spec certifies seq semantics by reducing them to values with a side-effect counter, so the corpus catches a laziness regression the value comparison would miss.

Laziness (certified — jolt matches JVM). The whole producer family (map/filter/remove/take/drop/concat/take-while/drop-while/mapcat/ partition/partition-all/partition-by/keep/keep-indexed/map-indexed/ distinct/interpose/interleave/take-nth/reductions/tree-seq/replace) is lazy at construction: building over a side-effecting source realizes zero elements (lazy / family is lazy at construction). Realization order is left-to-right, take/nth/drop realize exactly as far as demanded, a lazy seq memoizes (realize-once across walks), and next realizes head + one lookahead while rest realizes only the head (lazy / realization order & count, lazy / realization is memoized, lazy / realization timing). A lazy result is clojure.lang.LazySeq.

Accepted divergences. jolt is a simpler, finer-grained superset of JVM seq behavior; two classes diverge by representation, never by value, and are allowlisted in known-divergences.edn:

• :seq-type-model (seq-type-model / … suite, jolt-aei7) — jolt reifies every seq as PersistentList (eager) or LazySeq (deferred). JVM has a specialized class per producer (Cons, Iterate, LongRange, Repeat, Cycle, PersistentVector$ChunkedSeq, StringSeq, KeySeq/ValSeq, RSeq, ArraySeq, SubVector), so (class …) differs. instance? clojure.lang.ISeq/Sequential and all values/laziness are correct.
• :chunking-model (chunking-model / … suite, jolt-mm6v) — jolt seqs are unchunked: forcing one element realizes one, where JVM realizes a ~32-element chunk; mapcat/dedupe realize 0 at construction where JVM forces the first chunk. Strictly finer-grained laziness, decided after the chunk fast path (jolt-j9dz) was made O(n).

Hosting jolt on a new runtime

1. Implement the reader + analyzer + a backend for your runtime (see the Chez port under host/chez/ for a worked example).

2. Write a ~30-line harness that, for each corpus row, evaluates :actual and :expected and compares by value-equality (skip :throws rows to an expect-raises check). Pseudocode:

   (doseq [{:keys [suite label actual expected]} (read-edn "test/chez/corpus.edn")]
     (let [feats (profile-features [suite label])]   ; from profile.edn
       (when (subset? feats my-implemented-features)  ; only cases I claim to support
         (record! [suite label]
                  (if (= :throws expected)
                    (raises? actual)
                    (value= (eval actual) (eval expected)))))))
   

3. Run it. Your conformance level is the set of feature families with no failures. Portable-only is the floor; each feature you implement raises it.

The reference harness does exactly this on Chez: host/chez/run-corpus.ss (the analyzer runs on Chez → Chez runtime), with a regression floor. Run it via make corpus.

Maintaining the spec

• Add/change cases: edit test/chez/corpus.edn directly, then re-source the answers with regen-corpus.clj.
• Re-certify: clojure -M test/conformance/certify.clj. A new divergence is either a real bug (file it, mark the allowlist entry :bug + :bead) or a deliberate delta (classify it in known-divergences.edn).
• Refresh the profile: re-run with --profile test/conformance/profile.edn.
• Re-floor the runtime gate when parity rises (host/chez/run-corpus.ss).