From f54e99cc08aa151d1d2daafa252c593865181fe0 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Sat, 20 Jun 2026 10:21:09 -0400 Subject: [PATCH] Conformance inc3: promote the corpus to a documented portable spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Makes the host-neutral corpus a first-class language specification with conformance levels, not just a regression suite. - [suite label] is now a unique, stable case id (extract-corpus disambiguates duplicate labels with ' (N)' — one collision existed). - certify.clj --profile emits test/conformance/profile.edn: every non-portable case classified by the host feature it requires (numerics/double-only, concurrency/snapshot, host/jvm-interop, host/arrays, host/janet, async/core-async, runtime/eval, reader/jolt, printer/jolt, strictness/jolt, impl/representation, bug). 2670 of 2919 cases are portable (pass on any faithful Clojure); 249 are feature-gated. - SPEC.md documents the contract: row schema, the JVM oracle, conformance levels, the feature vocabulary, and a worked new-runtime harness — so hosting jolt elsewhere and proving it correct is read-one-file mechanical. Janet gate 155 files 0 failed; certify + zero-janet gates green. --- test/chez/corpus.edn | 2 +- test/chez/extract-corpus.janet | 12 +- test/conformance/README.md | 5 + test/conformance/SPEC.md | 115 ++++ test/conformance/certify.clj | 73 +++ test/conformance/profile.edn | 1019 ++++++++++++++++++++++++++++++++ 6 files changed, 1223 insertions(+), 3 deletions(-) create mode 100644 test/conformance/SPEC.md create mode 100644 test/conformance/profile.edn diff --git a/test/chez/corpus.edn b/test/chez/corpus.edn index bebb3c5..9f47b3f 100644 --- a/test/chez/corpus.edn +++ b/test/chez/corpus.edn @@ -2897,7 +2897,7 @@ {:suite "conformance / regex (capturing groups, backtracking, flags, lookahead)" :label "optional group" :expected "[\"1.2.3\" \"1\" \"2\" \"3\" nil]" :actual "(re-find #\"(\\d+)\\.(\\d+)\\.(\\d+)(?:-([a-z]+))?\" \"1.2.3\")"} {:suite "conformance / regex (capturing groups, backtracking, flags, lookahead)" :label "alternation" :expected "\"dog\"" :actual "(re-find #\"cat|dog\" \"a dog cat\")"} {:suite "conformance / regex (capturing groups, backtracking, flags, lookahead)" :label "str/replace $1" :expected "\"he[ll]o\"" :actual "(do (require (quote [clojure.string :as s])) (s/replace \"hello\" #\"(l+)\" \"[$1]\"))"} - {:suite "conformance / regex (capturing groups, backtracking, flags, lookahead)" :label "str/replace regex" :expected "\"X-X\"" :actual "(do (require (quote [clojure.string :as s])) (s/replace \"a-b\" #\"[a-z]\" \"X\"))"} + {:suite "conformance / regex (capturing groups, backtracking, flags, lookahead)" :label "str/replace regex (2)" :expected "\"X-X\"" :actual "(do (require (quote [clojure.string :as s])) (s/replace \"a-b\" #\"[a-z]\" \"X\"))"} {:suite "conformance / map literals evaluate their values" :label "map literal var" :expected "{:k 5}" :actual "(let [x 5] {:k x})"} {:suite "conformance / map literals evaluate their values" :label "map literal nested" :expected "{:a {:b 2}}" :actual "(let [y 2] {:a {:b y}})"} {:suite "conformance / map literals evaluate their values" :label "map literal keyfn" :expected "{:x 1}" :actual "(let [k :x] {k 1})"} diff --git a/test/chez/extract-corpus.janet b/test/chez/extract-corpus.janet index e9d8f35..9fc52b0 100644 --- a/test/chez/extract-corpus.janet +++ b/test/chez/extract-corpus.janet @@ -99,12 +99,20 @@ :label label :expected expected :actual actual})))))) (printf "folded %d unique conformance cases (deduped by :actual)" conf-added) -# emit EDN-and-Janet-valid corpus +# emit EDN-and-Janet-valid corpus. [suite label] is the canonical case id, so make +# it unique: a duplicate label within a suite gets " (N)" appended (jolt-3447 — the +# conformance fold can repeat a label, e.g. two "str/replace regex" rows). Rows are +# immutable structs, so disambiguate the label here at emit time. +(def label-seen @{}) (def out @"[\n") (each row all + (def k (string (row :suite) "\x00" (row :label))) + (def n (get label-seen k)) + (put label-seen k (if n (+ n 1) 1)) + (def label (if n (string (row :label) " (" (+ n 1) ")") (row :label))) (buffer/push out (string " {:suite " (edn-str (row :suite)) - " :label " (edn-str (row :label)) + " :label " (edn-str label) " :expected " (if (keyword? (row :expected)) ":throws" (edn-str (row :expected))) " :actual " (edn-str (row :actual)) "}\n"))) (buffer/push out "]\n") diff --git a/test/conformance/README.md b/test/conformance/README.md index 614e560..6d4e6ac 100644 --- a/test/conformance/README.md +++ b/test/conformance/README.md @@ -1,5 +1,10 @@ # Conformance: certifying the corpus against reference Clojure +> **See [SPEC.md](SPEC.md)** for the full host-neutral language-spec contract: the +> corpus schema, conformance levels, the feature profile, and how to host jolt on a +> new runtime. This README covers the certification tooling specifically. + + The corpus (`test/chez/corpus.edn`) is jolt's host-neutral behavioral suite — one row per case: `{:suite :label :expected :actual}`, where `:actual` is a Clojure source expression and `:expected` its result (or `:throws`). Runtime harnesses diff --git a/test/conformance/SPEC.md b/test/conformance/SPEC.md new file mode 100644 index 0000000..95b0154 --- /dev/null +++ b/test/conformance/SPEC.md @@ -0,0 +1,115 @@ +# The jolt conformance spec + +This directory defines jolt's behavior as a **host-neutral, executable language +specification**: a data file of cases, certified against reference Clojure, with a +feature profile that lets any runtime declare a conformance *level*. The goal is to +make hosting jolt on a new runtime (and proving it correct) a mechanical exercise: +read one data file, run each case, compare, report. + +## The artifacts + +| File | Role | Generated by | +|------|------|--------------| +| `test/chez/corpus.edn` | **The spec.** ~2900 cases of `{:suite :label :expected :actual}`. | `test/chez/extract-corpus.janet` | +| `test/conformance/profile.edn` | Per-case **feature classification** — which non-portable cases need which host capability. | `certify.clj --profile` | +| `test/conformance/known-divergences.edn` | Curated allowlist of cases whose `:expected` deliberately differs from JVM Clojure (+ tracked bugs). | hand-maintained | +| `test/conformance/certify.clj` | Certifies `:expected` against reference **JVM Clojure**; gates on new/stale divergences; emits the profile. | — | + +The corpus is *generated* from `test/spec/*-spec.janet` and the inline cases in +`test/integration/conformance-test.janet` — those are the authoring convenience. +**`corpus.edn` is the canonical contract**: it is what every runtime consumes, and +what `certify.clj` certifies. A new runtime never needs to read Janet. + +## Row schema + +```edn +{:suite "numbers / arithmetic" ; grouping; " ::