Source the conformance corpus from JVM Clojure; retire the prelude gate
corpus.edn :expected is now the value reference JVM Clojure produces, set by the new test/conformance/regen-corpus.clj (one JVM process, per-row thread watchdog). 167 rows moved to the JVM value: ratios (/ 1 2)=>1/2, doubles (double 3)=>3.0, shared-heap concurrency (the future/pmap/agent cases), clojure.math doubles. The JVM is the spec; jolt is measured against it. known-divergences.edn shrinks to the rows whose JVM value is an opaque host object that can't round-trip to source (Java arrays, transients, atoms, beans, proxies, chunks all print as #object[..@addr]) plus (fn* foo) and a few racy concurrency cases (:flaky). The zero-janet gate's allowlist becomes the set of host gaps vs the JVM spec (no Class/array/BigDecimal, :jolt reader, jolt's own printing). Math/clojure.math sqrt/pow/floor/trig now return doubles (Chez returns exact for exact args, e.g. (sqrt 9)=>3); JVM always returns a double. extract-corpus.janet no longer writes corpus.edn unless asked (the test runner imported it and was silently overwriting the JVM corpus with the spec sources' placeholder answers). The prelude parity gate is deleted — the zero-janet spine + certify.clj are the oracles. zero-janet 2678 (0 new divergences), certify 0 new / 0 stale, emit-test 330/330.
This commit is contained in:
Yogthos
parent
467ad75ff7
commit
da775802d6
13 changed files with 435 additions and 823 deletions
|
|
@ -10,15 +10,17 @@ read one data file, run each case, compare, report.
|
|||
|
||||
| File | Role | Generated by |
|
||||
|------|------|--------------|
|
||||
| `test/chez/corpus.edn` | **The spec.** ~2900 cases of `{:suite :label :expected :actual}`. | `test/chez/extract-corpus.janet` |
|
||||
| `test/chez/corpus.edn` | **The spec.** ~2900 cases of `{:suite :label :expected :actual}`, `:expected` **sourced from reference JVM Clojure**. | `test/conformance/regen-corpus.clj` |
|
||||
| `test/conformance/profile.edn` | Per-case **feature classification** — which non-portable cases need which host capability. | `certify.clj --profile` |
|
||||
| `test/conformance/known-divergences.edn` | Curated allowlist of cases whose `:expected` deliberately differs from JVM Clojure (+ tracked bugs). | hand-maintained |
|
||||
| `test/conformance/known-divergences.edn` | The few rows whose JVM value is an opaque host object that can't round-trip to readable source (Java arrays/transients/atoms/beans/proxies print as `#object[..@addr]`), so the corpus keeps jolt's value. | `regen-corpus.clj` leftovers, hand-checked |
|
||||
| `test/conformance/regen-corpus.clj` | Sources every `:expected` from reference **JVM Clojure** in one process. | — |
|
||||
| `test/conformance/certify.clj` | Certifies `:expected` against reference **JVM Clojure**; gates on new/stale divergences; emits the profile. | — |
|
||||
|
||||
The corpus is *generated* from `test/spec/*-spec.janet` and the inline cases in
|
||||
`test/integration/conformance-test.janet` — those are the authoring convenience.
|
||||
**`corpus.edn` is the canonical contract**: it is what every runtime consumes, and
|
||||
what `certify.clj` certifies. A new runtime never needs to read Janet.
|
||||
`corpus.edn` is **JVM-sourced**: `regen-corpus.clj` evaluates each case's `:actual`
|
||||
on reference JVM Clojure and writes the JVM value as `:expected`. The case *list*
|
||||
(the `:actual` strings) comes from `test/spec/*-spec.janet` via `extract-corpus.janet`,
|
||||
but the *answers* are the JVM's. **`corpus.edn` is the canonical contract**: it is
|
||||
what every runtime consumes and what `certify.clj` certifies.
|
||||
|
||||
## Row schema
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
{:doc
|
||||
"Known divergences of test/chez/corpus.edn :expected from JVM Clojure, classified. Most are deliberate jolt-specific or host-model differences (-> :features in conformance inc3). :bug entries are genuine and tracked. The certifier (certify.clj) gates on NEW (unlisted) divergences only. Keyed by [suite label].",
|
||||
"Rows of test/chez/corpus.edn whose :expected differs from reference JVM Clojure. The corpus is JVM-sourced (regen-corpus.clj), so this list is only the rows whose JVM value is an opaque host object that cannot round-trip to readable source — Java arrays, transients, atoms, beans, proxies, and chunks print as #object[..@addr] with a per-run identity — plus the (fn* foo) strictness case and a few racy concurrency cases (:flaky). For those the corpus keeps jolt's value. certify.clj gates on NEW (unlisted) divergences and STALE entries. Keyed by [suite label].",
|
||||
:legend
|
||||
{:numeric-model
|
||||
"jolt is all-double: no Ratio/BigDecimal/float; (/ 1 2)=>0.5, 3.0 prints 3",
|
||||
|
|
@ -16,67 +16,36 @@
|
|||
:impl-detail
|
||||
"representation detail: syntax-quote yields a list? (JVM yields a Cons)"},
|
||||
:entries
|
||||
[{:suite "clojure.core / futures — predicates",
|
||||
[{:suite "forms / fn",
|
||||
:label "no param vector",
|
||||
:category :strictness}
|
||||
;; racy: agent send / future-cancel observe state that the action thread may or
|
||||
;; may not have reached yet, so the JVM value is nondeterministic run-to-run.
|
||||
{:suite "state / agents (synchronous shim)", :label "send applies",
|
||||
:category :concurrency-model, :flaky true}
|
||||
{:suite "state / agents (synchronous shim)", :label "send-off applies",
|
||||
:category :concurrency-model, :flaky true}
|
||||
{:suite "clojure.core / futures — predicates",
|
||||
:label "cancel an in-flight future returns true",
|
||||
:category :concurrency-model,
|
||||
:flaky true}
|
||||
{:suite "clojure.core / futures — snapshot (copy) semantics",
|
||||
:label "captured atom is snapshotted, not shared",
|
||||
:category :concurrency-model}
|
||||
{:suite "clojure.core / pmap family",
|
||||
:label "snapshot semantics",
|
||||
:category :concurrency-model}
|
||||
{:suite "state / agents (synchronous shim)",
|
||||
:label "send applies",
|
||||
:category :concurrency-model}
|
||||
{:suite "state / agents (synchronous shim)",
|
||||
:label "send-off applies",
|
||||
:category :concurrency-model}
|
||||
{:suite "core / extenders",
|
||||
:label "lists extended type",
|
||||
:category :host-model}
|
||||
{:suite "core / extenders",
|
||||
:label "seq of tags",
|
||||
:category :host-model}
|
||||
{:suite "core / with-open",
|
||||
:label "close on throw",
|
||||
:category :host-model}
|
||||
{:suite "host-interop / class tokens & readers",
|
||||
:label "class name evaluates to canonical string",
|
||||
:category :host-model}
|
||||
{:suite "host-interop / exception + HashMap shims",
|
||||
:label "getMessage on a thrown string",
|
||||
:category :host-model}
|
||||
{:suite "io / *in* + with-in-str + read-line",
|
||||
:label "*in* is bound",
|
||||
:category :host-model}
|
||||
{:suite "metadata / def metadata",
|
||||
:label "^Type tag on var",
|
||||
:category :host-model}
|
||||
:category :concurrency-model, :flaky true}
|
||||
{:suite "clojure.core / futures — predicates",
|
||||
:label "future-cancelled? after cancel",
|
||||
:category :concurrency-model, :flaky true}
|
||||
{:suite "io / cold tagged types via print-method",
|
||||
:label "atom override fires nested",
|
||||
:category :reader-model}
|
||||
{:suite "io / cold tagged types via print-method",
|
||||
:label "transient map",
|
||||
:category :printer-model}
|
||||
{:suite "io / cold tagged types via print-method",
|
||||
:label "transient vector",
|
||||
:category :printer-model}
|
||||
{:suite "metadata / type hints",
|
||||
:label "symbol hint -> :tag",
|
||||
:category :host-model}
|
||||
{:suite "predicates / compare, type, any? (stage 3)",
|
||||
:label "type of record",
|
||||
:category :host-model}
|
||||
{:suite "predicates / tagged-value (Phase 4)",
|
||||
:label "chunked-seq? always false",
|
||||
:category :host-model}
|
||||
{:suite "untested / JVM-shape stubs (documented jolt behavior)",
|
||||
:label "bean is the map",
|
||||
:category :host-model}
|
||||
{:suite "untested / JVM-shape stubs (documented jolt behavior)",
|
||||
:label "class keyword",
|
||||
:category :host-model}
|
||||
{:suite "untested / JVM-shape stubs (documented jolt behavior)",
|
||||
:label "class number",
|
||||
:category :host-model}
|
||||
{:suite "untested / JVM-shape stubs (documented jolt behavior)",
|
||||
:label "class string",
|
||||
:category :host-model}
|
||||
{:suite "untested / JVM-shape stubs (documented jolt behavior)",
|
||||
:label "definterface defines",
|
||||
:category :host-model}
|
||||
{:suite "untested / JVM-shape stubs (documented jolt behavior)",
|
||||
:label "proxy resolves nil",
|
||||
:category :host-model}
|
||||
|
|
@ -106,119 +75,4 @@
|
|||
:category :host-model}
|
||||
{:suite "untested / chunk family (eager equivalents) + cat",
|
||||
:label "chunk round-trip",
|
||||
:category :host-model}
|
||||
{:suite "untested / ns + REPL machinery",
|
||||
:label "*in* bound",
|
||||
:category :host-model}
|
||||
{:suite "untested / ns + REPL machinery",
|
||||
:label "ns-imports empty user",
|
||||
:category :host-model}
|
||||
{:suite "untested / unchecked-* are plain ops",
|
||||
:label "unchecked-char",
|
||||
:category :host-model}
|
||||
{:suite "macros / defmacro",
|
||||
:label "macroexpand-1",
|
||||
:category :impl-detail}
|
||||
{:suite "clojure.core / leaf batch (complement fnil munge etc.)",
|
||||
:label "bigdec",
|
||||
:category :numeric-model}
|
||||
{:suite
|
||||
"conformance / native-op parity (compile emits janet ops at guarded arities)",
|
||||
:label "native unary div",
|
||||
:category :numeric-model}
|
||||
{:suite "numbers / arithmetic",
|
||||
:label "divide to fraction",
|
||||
:category :numeric-model}
|
||||
{:suite "numbers / literal syntax",
|
||||
:label "bigdec int M",
|
||||
:category :numeric-model}
|
||||
{:suite "numbers / literal syntax",
|
||||
:label "bigdec suffix M",
|
||||
:category :numeric-model}
|
||||
{:suite "numbers / literal syntax",
|
||||
:label "neg ratio",
|
||||
:category :numeric-model}
|
||||
{:suite "numbers / literal syntax",
|
||||
:label "ratio -> double",
|
||||
:category :numeric-model}
|
||||
{:suite "numbers / literal syntax",
|
||||
:label "ratio 3/4",
|
||||
:category :numeric-model}
|
||||
{:suite "numbers / printing of inf & nan",
|
||||
:label "inf inside coll",
|
||||
:category :numeric-model}
|
||||
{:suite "numbers / printing of inf & nan",
|
||||
:label "pr-str Infinity",
|
||||
:category :numeric-model}
|
||||
{:suite "reader / scalar literals",
|
||||
:label "bigdec suffix M",
|
||||
:category :numeric-model}
|
||||
{:suite "reader / scalar literals",
|
||||
:label "ratio -> double",
|
||||
:category :numeric-model}
|
||||
{:suite "untested / primed + division + bit ops",
|
||||
:label "/ ratio-as-double",
|
||||
:category :numeric-model}
|
||||
{:suite "untested / typed coercion views",
|
||||
:label "double",
|
||||
:category :numeric-model}
|
||||
{:suite "untested / typed coercion views",
|
||||
:label "float",
|
||||
:category :numeric-model}
|
||||
{:suite "untested / unchecked-* are plain ops",
|
||||
:label "unchecked-double",
|
||||
:category :numeric-model}
|
||||
{:suite "untested / unchecked-* are plain ops",
|
||||
:label "unchecked-float",
|
||||
:category :numeric-model}
|
||||
{:suite "io / cold tagged types via print-method",
|
||||
:label "transient map",
|
||||
:category :printer-model}
|
||||
{:suite "io / cold tagged types via print-method",
|
||||
:label "transient vector",
|
||||
:category :printer-model}
|
||||
{:suite "io / print-method multimethod",
|
||||
:label "defmethod fires through prn",
|
||||
:category :printer-model}
|
||||
{:suite "io / print-method multimethod",
|
||||
:label "defmethod overrides a record, top level",
|
||||
:category :printer-model}
|
||||
{:suite "io / cold tagged types via print-method",
|
||||
:label "atom override fires nested",
|
||||
:category :reader-model}
|
||||
{:suite "io / print-method multimethod",
|
||||
:label "defmethod fires nested in a map",
|
||||
:category :reader-model}
|
||||
{:suite "maps / literal construction",
|
||||
:label "source order through syntax-quote",
|
||||
:category :reader-model}
|
||||
{:suite "reader / dispatch & sugar",
|
||||
:label "reader cond :jolt",
|
||||
:category :reader-model}
|
||||
{:suite "reader / dispatch & sugar",
|
||||
:label "reader cond no match",
|
||||
:category :reader-model}
|
||||
{:suite "reader / dispatch & sugar",
|
||||
:label "reader cond splice",
|
||||
:category :reader-model}
|
||||
{:suite "reader / dispatch & sugar",
|
||||
:label "reader cond splice no match",
|
||||
:category :reader-model}
|
||||
{:suite "reader / dispatch & sugar",
|
||||
:label "reader conditional",
|
||||
:category :reader-model}
|
||||
{:suite "reader / syntax-quote literal collapse (spec 2.4 S25)",
|
||||
:label "bool nested",
|
||||
:category :reader-model}
|
||||
{:suite "reader / syntax-quote literal collapse (spec 2.4 S25)",
|
||||
:label "nil nested",
|
||||
:category :reader-model}
|
||||
{:suite "forms / fn",
|
||||
:label "no param vector",
|
||||
:category :strictness}
|
||||
{:suite "transient / assoc! odd args throw",
|
||||
:label "map dangling key",
|
||||
:category :strictness}
|
||||
{:suite "transient / assoc! odd args throw",
|
||||
:label "vector dangling",
|
||||
:category :strictness}]}
|
||||
:category :host-model}]}
|
||||
|
|
|
|||
121
test/conformance/regen-corpus.clj
Normal file
121
test/conformance/regen-corpus.clj
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
;; regen-corpus.clj — set test/chez/corpus.edn :expected to what reference JVM
|
||||
;; Clojure produces, making the JVM the source of truth for the spec.
|
||||
;;
|
||||
;; Runs in ONE JVM process: a single `clojure -M` invocation reads the corpus and
|
||||
;; evaluates every row's :actual once, in-process. The per-row watchdog is a thread
|
||||
;; (future) with a wall-clock deadline — not a new JVM per case.
|
||||
;;
|
||||
;; For each row:
|
||||
;; - JVM evaluates :actual to a value -> :expected := a self-evaluating source
|
||||
;; string for that value (lists/lazy-seqs are vectorized at every nesting depth,
|
||||
;; matching the corpus convention; jolt='s cross-type sequential equality makes
|
||||
;; a vector :expected match a seq :actual). Only adopted if the rendered string
|
||||
;; round-trips (re-reads+evals back to an = value) — else the existing :expected
|
||||
;; is kept.
|
||||
;; - JVM throws AND the row already expected :throws -> :throws (unchanged).
|
||||
;; - Otherwise (JVM can't run the case: jolt-specific host interop / reader, a
|
||||
;; timeout, or a throw where jolt has a value) -> the existing :expected is kept.
|
||||
;; These are the non-portable rows (profile-tagged) — JVM is not the oracle for
|
||||
;; them.
|
||||
;;
|
||||
;; Run from the repo root:
|
||||
;; clojure -M test/conformance/regen-corpus.clj ; rewrites corpus.edn
|
||||
;; clojure -M test/conformance/regen-corpus.clj --dry ; report only, no write
|
||||
(ns regen-corpus
|
||||
(:require [clojure.edn :as edn]
|
||||
[clojure.string :as str]
|
||||
[clojure.walk :as walk]))
|
||||
|
||||
(def corpus-path "test/chez/corpus.edn")
|
||||
(def dry? (some #{"--dry"} *command-line-args*))
|
||||
|
||||
;; --- isolated JVM evaluation (same model as certify.clj) ---------------------
|
||||
(defn read-program [src]
|
||||
(read-string {:read-cond :allow} (str "(do " src ")")))
|
||||
|
||||
(defn eval-isolated [src]
|
||||
(let [sink (java.io.StringWriter.)
|
||||
empty-in (java.io.PushbackReader. (java.io.StringReader. ""))]
|
||||
(try
|
||||
(remove-ns 'user)
|
||||
(let [the-ns (create-ns 'user)]
|
||||
(binding [*ns* the-ns *out* sink *err* sink *in* empty-in]
|
||||
(clojure.core/refer-clojure)
|
||||
;; make the stdlib namespaces a corpus case may reference by qualified name
|
||||
;; resolvable (clojure.math/floor etc.) so they evaluate to the real JVM
|
||||
;; value instead of throwing Unable-to-resolve (-> kept Janet value).
|
||||
(doseq [n '[clojure.string clojure.set clojure.walk clojure.edn
|
||||
clojure.math clojure.pprint]]
|
||||
(try (require n) (catch Throwable _ nil)))
|
||||
(let [form (try (read-program src)
|
||||
(catch Throwable t (throw (ex-info "read" {::read t}))))]
|
||||
[:ok (eval form)])))
|
||||
(catch clojure.lang.ExceptionInfo e
|
||||
(if (::read (ex-data e)) [:read-error (::read (ex-data e))] [:throw e]))
|
||||
(catch Throwable t [:throw t]))))
|
||||
|
||||
(def ^:const case-timeout-ms 5000)
|
||||
|
||||
;; --- rendering a JVM value to a self-evaluating :expected source string ------
|
||||
;; Vectorize every list/lazy-seq (at any nesting) so the rendered form is
|
||||
;; self-evaluating (no bare call forms) and jolt= to the seq it represents.
|
||||
(defn vectorize [v]
|
||||
(walk/postwalk (fn [x] (if (and (seq? x) (not (vector? x))) (vec x) x)) v))
|
||||
|
||||
(defn render [v] (binding [*print-length* nil *print-level* nil] (pr-str (vectorize v))))
|
||||
|
||||
;; Adopt the JVM value only if its rendered form round-trips back to an = value on
|
||||
;; the JVM (guards against records / tagged literals / opaque objects that pr-str
|
||||
;; can't reproduce as readable source).
|
||||
(defn round-trips? [s v]
|
||||
(try
|
||||
(let [rt (binding [*ns* (create-ns 'user)] (eval (read-string {:read-cond :allow} s)))]
|
||||
(= rt v))
|
||||
(catch Throwable _ false)))
|
||||
|
||||
;; Evaluate AND render inside one watchdog'd future: a row whose :actual returns an
|
||||
;; unforced infinite lazy seq evaluates fast (not a timeout) but render/postwalk
|
||||
;; would then realize it forever — so rendering must be inside the deadline too.
|
||||
;; Returns one of [:value s] / [:throw] / [:read-error] / [:timeout] / [:unrenderable].
|
||||
(defn eval+render [src]
|
||||
(let [f (future
|
||||
(try
|
||||
(let [r (eval-isolated src)]
|
||||
(if (= (first r) :ok)
|
||||
(let [v (second r) s (render v)]
|
||||
(if (round-trips? s v) [:value s] [:unrenderable]))
|
||||
[(first r)]))
|
||||
(catch Throwable _ [:unrenderable])))
|
||||
res (deref f case-timeout-ms ::timeout)]
|
||||
(if (= res ::timeout) (do (future-cancel f) [:timeout]) res)))
|
||||
|
||||
(defn regen-row [row]
|
||||
(let [{:keys [expected actual]} row
|
||||
r (eval+render actual)]
|
||||
(case (first r)
|
||||
:value (let [s (second r)] [(assoc row :expected s) (if (= s expected) :same :updated)])
|
||||
:throw (if (= expected :throws) [row :throws] [row :kept])
|
||||
;; read-error / timeout / unrenderable -> JVM is not the oracle; keep existing.
|
||||
[row :kept])))
|
||||
|
||||
;; --- corpus writer (preserves the one-row-per-line layout) -------------------
|
||||
(defn row-str [{:keys [suite label expected actual]}]
|
||||
(str " {:suite " (pr-str suite)
|
||||
" :label " (pr-str label)
|
||||
" :expected " (if (= expected :throws) ":throws" (pr-str expected))
|
||||
" :actual " (pr-str actual) "}"))
|
||||
|
||||
(defn -main [& _]
|
||||
(let [corpus (edn/read-string (slurp corpus-path))
|
||||
n (count corpus)
|
||||
results (mapv regen-row corpus)
|
||||
rows (mapv first results)
|
||||
tally (frequencies (map second results))]
|
||||
(println (format "Regenerated %d rows from JVM Clojure %s" n (clojure-version)))
|
||||
(doseq [k [:updated :same :throws :kept :unrenderable]]
|
||||
(println (format " %-13s %5d" (name k) (get tally k 0))))
|
||||
(when-not dry?
|
||||
(spit corpus-path (str "[\n" (str/join "\n" (map row-str rows)) "\n]\n"))
|
||||
(println (format "\nwrote %s" corpus-path)))))
|
||||
|
||||
(apply -main *command-line-args*)
|
||||
Loading…
Add table
Add a link
Reference in a new issue