Conformance inc3: promote the corpus to a documented portable spec

Makes the host-neutral corpus a first-class language specification with
conformance levels, not just a regression suite.

- [suite label] is now a unique, stable case id (extract-corpus disambiguates
  duplicate labels with ' (N)' — one collision existed).
- certify.clj --profile emits test/conformance/profile.edn: every non-portable
  case classified by the host feature it requires (numerics/double-only,
  concurrency/snapshot, host/jvm-interop, host/arrays, host/janet,
  async/core-async, runtime/eval, reader/jolt, printer/jolt, strictness/jolt,
  impl/representation, bug). 2670 of 2919 cases are portable (pass on any faithful
  Clojure); 249 are feature-gated.
- SPEC.md documents the contract: row schema, the JVM oracle, conformance levels,
  the feature vocabulary, and a worked new-runtime harness — so hosting jolt
  elsewhere and proving it correct is read-one-file mechanical.

Janet gate 155 files 0 failed; certify + zero-janet gates green.

test/chez/corpus.edn

@@ -2897,7 +2897,7 @@
   {:suite "conformance / regex (capturing groups, backtracking, flags, lookahead)" :label "optional group" :expected "[\"1.2.3\" \"1\" \"2\" \"3\" nil]" :actual "(re-find #\"(\\d+)\\.(\\d+)\\.(\\d+)(?:-([a-z]+))?\" \"1.2.3\")"}
   {:suite "conformance / regex (capturing groups, backtracking, flags, lookahead)" :label "alternation" :expected "\"dog\"" :actual "(re-find #\"cat|dog\" \"a dog cat\")"}
   {:suite "conformance / regex (capturing groups, backtracking, flags, lookahead)" :label "str/replace $1" :expected "\"he[ll]o\"" :actual "(do (require (quote [clojure.string :as s])) (s/replace \"hello\" #\"(l+)\" \"[$1]\"))"}
-  {:suite "conformance / regex (capturing groups, backtracking, flags, lookahead)" :label "str/replace regex" :expected "\"X-X\"" :actual "(do (require (quote [clojure.string :as s])) (s/replace \"a-b\" #\"[a-z]\" \"X\"))"}
+  {:suite "conformance / regex (capturing groups, backtracking, flags, lookahead)" :label "str/replace regex (2)" :expected "\"X-X\"" :actual "(do (require (quote [clojure.string :as s])) (s/replace \"a-b\" #\"[a-z]\" \"X\"))"}
   {:suite "conformance / map literals evaluate their values" :label "map literal var" :expected "{:k 5}" :actual "(let [x 5] {:k x})"}
   {:suite "conformance / map literals evaluate their values" :label "map literal nested" :expected "{:a {:b 2}}" :actual "(let [y 2] {:a {:b y}})"}
   {:suite "conformance / map literals evaluate their values" :label "map literal keyfn" :expected "{:x 1}" :actual "(let [k :x] {k 1})"}

test/chez/extract-corpus.janet

@@ -99,12 +99,20 @@
                            :label label :expected expected :actual actual}))))))
 (printf "folded %d unique conformance cases (deduped by :actual)" conf-added)
 
-# emit EDN-and-Janet-valid corpus
+# emit EDN-and-Janet-valid corpus. [suite label] is the canonical case id, so make
+# it unique: a duplicate label within a suite gets " (N)" appended (jolt-3447 — the
+# conformance fold can repeat a label, e.g. two "str/replace regex" rows). Rows are
+# immutable structs, so disambiguate the label here at emit time.
+(def label-seen @{})
 (def out @"[\n")
 (each row all
+  (def k (string (row :suite) "\x00" (row :label)))
+  (def n (get label-seen k))
+  (put label-seen k (if n (+ n 1) 1))
+  (def label (if n (string (row :label) " (" (+ n 1) ")") (row :label)))
   (buffer/push out
     (string "  {:suite " (edn-str (row :suite))
-            " :label " (edn-str (row :label))
+            " :label " (edn-str label)
             " :expected " (if (keyword? (row :expected)) ":throws" (edn-str (row :expected)))
             " :actual " (edn-str (row :actual)) "}\n")))
 (buffer/push out "]\n")

test/conformance/README.md

@@ -1,5 +1,10 @@
 # Conformance: certifying the corpus against reference Clojure
 
+> **See [SPEC.md](SPEC.md)** for the full host-neutral language-spec contract: the
+> corpus schema, conformance levels, the feature profile, and how to host jolt on a
+> new runtime. This README covers the certification tooling specifically.
+
+
 The corpus (`test/chez/corpus.edn`) is jolt's host-neutral behavioral suite — one
 row per case: `{:suite :label :expected :actual}`, where `:actual` is a Clojure
 source expression and `:expected` its result (or `:throws`). Runtime harnesses

test/conformance/SPEC.md

@@ -0,0 +1,115 @@
+# The jolt conformance spec
+
+This directory defines jolt's behavior as a **host-neutral, executable language
+specification**: a data file of cases, certified against reference Clojure, with a
+feature profile that lets any runtime declare a conformance *level*. The goal is to
+make hosting jolt on a new runtime (and proving it correct) a mechanical exercise:
+read one data file, run each case, compare, report.
+
+## The artifacts
+
+| File | Role | Generated by |
+|------|------|--------------|
+| `test/chez/corpus.edn` | **The spec.** ~2900 cases of `{:suite :label :expected :actual}`. | `test/chez/extract-corpus.janet` |
+| `test/conformance/profile.edn` | Per-case **feature classification** — which non-portable cases need which host capability. | `certify.clj --profile` |
+| `test/conformance/known-divergences.edn` | Curated allowlist of cases whose `:expected` deliberately differs from JVM Clojure (+ tracked bugs). | hand-maintained |
+| `test/conformance/certify.clj` | Certifies `:expected` against reference **JVM Clojure**; gates on new/stale divergences; emits the profile. | — |
+
+The corpus is *generated* from `test/spec/*-spec.janet` and the inline cases in
+`test/integration/conformance-test.janet` — those are the authoring convenience.
+**`corpus.edn` is the canonical contract**: it is what every runtime consumes, and
+what `certify.clj` certifies. A new runtime never needs to read Janet.
+
+## Row schema
+
+```edn
+{:suite    "numbers / arithmetic"      ; grouping; "<suite> :: <label>" is the case id
+ :label    "integer add"               ; unique within a suite
+ :actual   "(+ 1 2)"                    ; Clojure source to evaluate
+ :expected "3"}                         ; Clojure source whose value it must equal,
+                                        ; or the keyword :throws
+```
+
+- `[:suite :label]` is the **canonical, unique case id** (the generator
+  disambiguates duplicate labels with ` (N)`).
+- Comparison is **value-equality** (`=`), never string/printed-form — so map/set
+  iteration order never matters.
+- `:expected :throws` asserts evaluating `:actual` raises.
+
+## The oracle: reference JVM Clojure
+
+Historically every `:expected` was hand-written. `certify.clj` removes that
+weakness: it evaluates every `:actual` (and `:expected`) on **JVM Clojure** in a
+fresh `user` namespace and checks jolt's `:expected` against what real Clojure
+produces. Of ~2740 vanilla-certifiable rows, **>2660 match reference Clojure
+exactly**. The rest are classified (see below) — none are silently wrong.
+
+```sh
+clojure -M test/conformance/certify.clj                                  # gate
+clojure -M test/conformance/certify.clj test/chez/corpus.edn --edn r.edn # + report
+clojure -M test/conformance/certify.clj test/chez/corpus.edn --profile test/conformance/profile.edn
+```
+
+The gate fails only on a **new** (unclassified) divergence or a **stale**
+allowlist entry; flaky timing-dependent cases (`future-cancel`) are tolerated.
+
+## Conformance levels & the feature profile
+
+Not every case is portable: some assume a host capability jolt has on one runtime
+but not another (Java interop, real threads, BigDecimal). `profile.edn` classifies
+each **non-portable** case by the feature it requires. Cases *not* in the profile
+are **portable** — they must pass on any faithful Clojure.
+
+A runtime's **conformance level** = portable cases + the feature families it
+implements. Current profile (≈2670 portable, ≈249 non-portable):
+
+| Feature | Meaning |
+|---------|---------|
+| `:numerics/double-only` | all-double numeric model — no Ratio/BigDecimal/float; `(/ 1 2)` ⇒ `0.5` |
+| `:concurrency/snapshot` | isolated-heap futures/agents/pmap — captured atoms are snapshotted, not shared |
+| `:host/jvm-interop` | Java classes / `instance?` on host classes / proxy / bean / definterface |
+| `:host/arrays` | Java arrays (`into-array`, `int-array`, …) |
+| `:host/janet` | Janet host interop (`janet.*`) |
+| `:async/core-async` | `clojure.core.async` channels/`go` |
+| `:runtime/eval` | runtime `eval` / `load-string` |
+| `:reader/jolt` | jolt reader features (`#?(:jolt …)`) + syntax-quote literal collapse |
+| `:printer/jolt` | jolt's rendering of transients/atoms/`print-method` overrides |
+| `:strictness/jolt` | intentionally stricter (throws on odd `assoc!` args, etc.) |
+| `:impl/representation` | representation detail (e.g. syntax-quote yields a `list?`, not a `Cons`) |
+| `:bug` | a *known defect* (tracked bead) — not a host difference |
+
+## Hosting jolt on a new runtime
+
+1. Implement the reader + analyzer + a backend for your runtime (see the Chez port
+   under `host/chez/` for a worked example).
+2. Write a ~30-line harness that, for each corpus row, evaluates `:actual` and
+   `:expected` and compares by value-equality (skip `:throws` rows to an
+   expect-raises check). Pseudocode:
+
+   ```
+   (doseq [{:keys [suite label actual expected]} (read-edn "test/chez/corpus.edn")]
+     (let [feats (profile-features [suite label])]   ; from profile.edn
+       (when (subset? feats my-implemented-features)  ; only cases I claim to support
+         (record! [suite label]
+                  (if (= :throws expected)
+                    (raises? actual)
+                    (value= (eval actual) (eval expected)))))))
+   ```
+
+3. Run it. Your **conformance level** is the set of feature families with no
+   failures. Portable-only is the floor; each feature you implement raises it.
+
+The two reference harnesses already do exactly this on Chez:
+`test/chez/run-corpus-prelude.janet` (Janet analyzer → Chez runtime) and
+`test/chez/run-corpus-zero-janet.janet` (Chez analyzer → Chez runtime), both with a
+regression floor.
+
+## Maintaining the spec
+
+- **Add/change cases**: edit `test/spec/*-spec.janet` or `conformance-test.janet`,
+  then `janet test/chez/extract-corpus.janet` to regenerate `corpus.edn`.
+- **Re-certify**: `clojure -M test/conformance/certify.clj`. A new divergence is
+  either a real bug (file it, mark the allowlist entry `:bug` + `:bead`) or a
+  deliberate delta (classify it in `known-divergences.edn`).
+- **Refresh the profile**: re-run with `--profile test/conformance/profile.edn`.
+- **Re-floor the runtime gates** when parity rises (`run-corpus-*.janet`).

test/conformance/certify.clj

@@ -130,6 +130,51 @@
            :detail (str "jolt-expected=" (pr-str (second e))
                         " JVM-result=" (pr-str (second a)))})))))
 
+(def profile-out
+  (let [args (vec *command-line-args*)
+        i (.indexOf args "--profile")]
+    (when (and (>= i 0) (< (inc i) (count args))) (nth args (inc i)))))
+
+;; Allowlist category -> conformance feature (for divergent / throws-mismatch rows
+;; whose nature a human classified in known-divergences.edn).
+(def category->feature
+  {:numeric-model :numerics/double-only
+   :concurrency-model :concurrency/snapshot
+   :reader-model :reader/jolt
+   :printer-model :printer/jolt
+   :strictness :strictness/jolt
+   :impl-detail :impl/representation
+   :host-model :host/jvm-interop
+   :bug :bug})
+(def allow-category
+  (into {} (map (fn [e] [[(:suite e) (:label e)] (:category e)]) allowlist-entries)))
+
+;; Coarse feature for a row the JVM couldn't certify (jvm-error) — by scanning the
+;; source for what host capability it exercises. Defaults to generic host interop.
+(defn jvm-error-feature [actual]
+  (let [a (str/lower-case actual)]
+    (cond
+      (re-find #"janet" a) :host/janet
+      (re-find #"chan|>!|<!|go-loop|\(go |alts!|core\.async" a) :async/core-async
+      (re-find #"load-string|\beval\b|read-string.*eval" a) :runtime/eval
+      (re-find #"-array|into-array|to-array|aget|aset|aclone|alength" a) :host/arrays
+      (re-find #"proxy|reify|gen-class|definterface|deftype.*:|\.\.|bean" a) :host/jvm-interop
+      (re-find #"class|forname|instance\?|cast|\.getclass" a) :host/jvm-interop
+      :else :host/jvm-interop)))
+
+;; The full conformance feature(s) a row requires (empty = portable). Certified
+;; rows are portable; everything else gets a feature so a runtime knows what host
+;; capability the case assumes.
+(defn row-features [bucket suite label actual]
+  (case bucket
+    (:certified :certified-throws) []
+    (:divergent :throws-mismatch) [(category->feature (allow-category [suite label] :host/jvm-interop)
+                                                       :host/jvm-interop)]
+    :read-error [:reader/jolt]
+    :timeout [:perf/unbounded]
+    :jvm-error [(jvm-error-feature actual)]
+    [:host/jvm-interop]))
+
 (defn -main [& _]
   (let [corpus (edn/read-string (slurp corpus-path))
         results (mapv (fn [row] (assoc (classify row) :row row)) corpus)
@@ -170,6 +215,34 @@
       (def new-divergences news)
       (def stale-entries stale))
 
+    ;; Conformance profile: every NON-portable case keyed by [suite label] -> its
+    ;; feature(s) + bucket. Certified (portable) cases are omitted — they are the
+    ;; baseline every runtime must pass. A runtime computes its conformance LEVEL by
+    ;; subtracting the features it doesn't implement. Written when --profile is given.
+    (when profile-out
+      (let [entries (->> results
+                         (remove #(#{:certified :certified-throws} (:bucket %)))
+                         (map (fn [{:keys [bucket row]}]
+                                (let [{:keys [suite label actual]} row]
+                                  {:suite suite :label label :bucket bucket
+                                   :features (row-features bucket suite label actual)})))
+                         (sort-by (juxt :suite :label)) vec)
+            feat-counts (->> entries (mapcat :features) frequencies (into (sorted-map)))]
+        (spit profile-out
+              (with-out-str
+                (pp/pprint
+                  {:doc (str "Conformance profile for test/chez/corpus.edn, generated by certify.clj. "
+                             "Each entry is a NON-portable case (keyed by [suite label]) and the host "
+                             "feature(s) it requires. Cases NOT listed are portable — they pass on any "
+                             "faithful Clojure. A runtime's conformance LEVEL = portable + the feature "
+                             "families it implements. See SPEC.md.")
+                   :clojure-version (clojure-version)
+                   :portable-count (+ (cnt :certified) (cnt :certified-throws))
+                   :non-portable-count (count entries)
+                   :feature-counts feat-counts
+                   :entries entries})))
+        (println (format "\nwrote conformance profile (%d non-portable cases) to %s" (count entries) profile-out))))
+
     ;; Full per-row divergence detail goes to the --edn report (for triage); the
     ;; console stays quiet about KNOWN divergences (the NEW/STALE sections above are
     ;; what matters for the gate).