da775802d6
corpus.edn :expected is now the value reference JVM Clojure produces, set by the new test/conformance/regen-corpus.clj (one JVM process, per-row thread watchdog). 167 rows moved to the JVM value: ratios (/ 1 2)=>1/2, doubles (double 3)=>3.0, shared-heap concurrency (the future/pmap/agent cases), clojure.math doubles. The JVM is the spec; jolt is measured against it. known-divergences.edn shrinks to the rows whose JVM value is an opaque host object that can't round-trip to source (Java arrays, transients, atoms, beans, proxies, chunks all print as #object[..@addr]) plus (fn* foo) and a few racy concurrency cases (:flaky). The zero-janet gate's allowlist becomes the set of host gaps vs the JVM spec (no Class/array/BigDecimal, :jolt reader, jolt's own printing). Math/clojure.math sqrt/pow/floor/trig now return doubles (Chez returns exact for exact args, e.g. (sqrt 9)=>3); JVM always returns a double. extract-corpus.janet no longer writes corpus.edn unless asked (the test runner imported it and was silently overwriting the JVM corpus with the spec sources' placeholder answers). The prelude parity gate is deleted — the zero-janet spine + certify.clj are the oracles. zero-janet 2678 (0 new divergences), certify 0 new / 0 stale, emit-test 330/330.
117 lines
6.5 KiB
Markdown
117 lines
6.5 KiB
Markdown
# The jolt conformance spec
|
|
|
|
This directory defines jolt's behavior as a **host-neutral, executable language
|
|
specification**: a data file of cases, certified against reference Clojure, with a
|
|
feature profile that lets any runtime declare a conformance *level*. The goal is to
|
|
make hosting jolt on a new runtime (and proving it correct) a mechanical exercise:
|
|
read one data file, run each case, compare, report.
|
|
|
|
## The artifacts
|
|
|
|
| File | Role | Generated by |
|
|
|------|------|--------------|
|
|
| `test/chez/corpus.edn` | **The spec.** ~2900 cases of `{:suite :label :expected :actual}`, `:expected` **sourced from reference JVM Clojure**. | `test/conformance/regen-corpus.clj` |
|
|
| `test/conformance/profile.edn` | Per-case **feature classification** — which non-portable cases need which host capability. | `certify.clj --profile` |
|
|
| `test/conformance/known-divergences.edn` | The few rows whose JVM value is an opaque host object that can't round-trip to readable source (Java arrays/transients/atoms/beans/proxies print as `#object[..@addr]`), so the corpus keeps jolt's value. | `regen-corpus.clj` leftovers, hand-checked |
|
|
| `test/conformance/regen-corpus.clj` | Sources every `:expected` from reference **JVM Clojure** in one process. | — |
|
|
| `test/conformance/certify.clj` | Certifies `:expected` against reference **JVM Clojure**; gates on new/stale divergences; emits the profile. | — |
|
|
|
|
`corpus.edn` is **JVM-sourced**: `regen-corpus.clj` evaluates each case's `:actual`
|
|
on reference JVM Clojure and writes the JVM value as `:expected`. The case *list*
|
|
(the `:actual` strings) comes from `test/spec/*-spec.janet` via `extract-corpus.janet`,
|
|
but the *answers* are the JVM's. **`corpus.edn` is the canonical contract**: it is
|
|
what every runtime consumes and what `certify.clj` certifies.
|
|
|
|
## Row schema
|
|
|
|
```edn
|
|
{:suite "numbers / arithmetic" ; grouping; "<suite> :: <label>" is the case id
|
|
:label "integer add" ; unique within a suite
|
|
:actual "(+ 1 2)" ; Clojure source to evaluate
|
|
:expected "3"} ; Clojure source whose value it must equal,
|
|
; or the keyword :throws
|
|
```
|
|
|
|
- `[:suite :label]` is the **canonical, unique case id** (the generator
|
|
disambiguates duplicate labels with ` (N)`).
|
|
- Comparison is **value-equality** (`=`), never string/printed-form — so map/set
|
|
iteration order never matters.
|
|
- `:expected :throws` asserts evaluating `:actual` raises.
|
|
|
|
## The oracle: reference JVM Clojure
|
|
|
|
Historically every `:expected` was hand-written. `certify.clj` removes that
|
|
weakness: it evaluates every `:actual` (and `:expected`) on **JVM Clojure** in a
|
|
fresh `user` namespace and checks jolt's `:expected` against what real Clojure
|
|
produces. Of ~2740 vanilla-certifiable rows, **>2660 match reference Clojure
|
|
exactly**. The rest are classified (see below) — none are silently wrong.
|
|
|
|
```sh
|
|
clojure -M test/conformance/certify.clj # gate
|
|
clojure -M test/conformance/certify.clj test/chez/corpus.edn --edn r.edn # + report
|
|
clojure -M test/conformance/certify.clj test/chez/corpus.edn --profile test/conformance/profile.edn
|
|
```
|
|
|
|
The gate fails only on a **new** (unclassified) divergence or a **stale**
|
|
allowlist entry; flaky timing-dependent cases (`future-cancel`) are tolerated.
|
|
|
|
## Conformance levels & the feature profile
|
|
|
|
Not every case is portable: some assume a host capability jolt has on one runtime
|
|
but not another (Java interop, real threads, BigDecimal). `profile.edn` classifies
|
|
each **non-portable** case by the feature it requires. Cases *not* in the profile
|
|
are **portable** — they must pass on any faithful Clojure.
|
|
|
|
A runtime's **conformance level** = portable cases + the feature families it
|
|
implements. Current profile (≈2670 portable, ≈249 non-portable):
|
|
|
|
| Feature | Meaning |
|
|
|---------|---------|
|
|
| `:numerics/double-only` | all-double numeric model — no Ratio/BigDecimal/float; `(/ 1 2)` ⇒ `0.5` |
|
|
| `:concurrency/snapshot` | isolated-heap futures/agents/pmap — captured atoms are snapshotted, not shared |
|
|
| `:host/jvm-interop` | Java classes / `instance?` on host classes / proxy / bean / definterface |
|
|
| `:host/arrays` | Java arrays (`into-array`, `int-array`, …) |
|
|
| `:host/janet` | Janet host interop (`janet.*`) |
|
|
| `:async/core-async` | `clojure.core.async` channels/`go` |
|
|
| `:runtime/eval` | runtime `eval` / `load-string` |
|
|
| `:reader/jolt` | jolt reader features (`#?(:jolt …)`) + syntax-quote literal collapse |
|
|
| `:printer/jolt` | jolt's rendering of transients/atoms/`print-method` overrides |
|
|
| `:strictness/jolt` | intentionally stricter (throws on odd `assoc!` args, etc.) |
|
|
| `:impl/representation` | representation detail (e.g. syntax-quote yields a `list?`, not a `Cons`) |
|
|
| `:bug` | a *known defect* (tracked bead) — not a host difference |
|
|
|
|
## Hosting jolt on a new runtime
|
|
|
|
1. Implement the reader + analyzer + a backend for your runtime (see the Chez port
|
|
under `host/chez/` for a worked example).
|
|
2. Write a ~30-line harness that, for each corpus row, evaluates `:actual` and
|
|
`:expected` and compares by value-equality (skip `:throws` rows to an
|
|
expect-raises check). Pseudocode:
|
|
|
|
```
|
|
(doseq [{:keys [suite label actual expected]} (read-edn "test/chez/corpus.edn")]
|
|
(let [feats (profile-features [suite label])] ; from profile.edn
|
|
(when (subset? feats my-implemented-features) ; only cases I claim to support
|
|
(record! [suite label]
|
|
(if (= :throws expected)
|
|
(raises? actual)
|
|
(value= (eval actual) (eval expected)))))))
|
|
```
|
|
|
|
3. Run it. Your **conformance level** is the set of feature families with no
|
|
failures. Portable-only is the floor; each feature you implement raises it.
|
|
|
|
The two reference harnesses already do exactly this on Chez:
|
|
`test/chez/run-corpus-prelude.janet` (Janet analyzer → Chez runtime) and
|
|
`test/chez/run-corpus-zero-janet.janet` (Chez analyzer → Chez runtime), both with a
|
|
regression floor.
|
|
|
|
## Maintaining the spec
|
|
|
|
- **Add/change cases**: edit `test/spec/*-spec.janet` or `conformance-test.janet`,
|
|
then `janet test/chez/extract-corpus.janet` to regenerate `corpus.edn`.
|
|
- **Re-certify**: `clojure -M test/conformance/certify.clj`. A new divergence is
|
|
either a real bug (file it, mark the allowlist entry `:bug` + `:bead`) or a
|
|
deliberate delta (classify it in `known-divergences.edn`).
|
|
- **Refresh the profile**: re-run with `--profile test/conformance/profile.edn`.
|
|
- **Re-floor the runtime gates** when parity rises (`run-corpus-*.janet`).
|