From 750ce0571688961ee276da9e19ade3fe7a5d4aad Mon Sep 17 00:00:00 2001 From: Yogthos Date: Sun, 21 Jun 2026 11:34:48 -0400 Subject: [PATCH] Docs + CI for the Chez-only substrate Rewrite the README, CLAUDE.md build/architecture sections, test/chez/README, and conformance SPEC for the Janet-free world: bin/joltc + make test, the self-hosting bootstrap, the frozen JVM-sourced corpus. CI installs Chez + JDK/ Clojure and runs 'make test' (was Janet/jpm). jolt-cf1q.6 --- .github/workflows/tests.yml | 61 +++---- CLAUDE.md | 93 +++++------ README.md | 306 +++++++++--------------------------- test/chez/README.md | 75 ++++----- test/conformance/SPEC.md | 20 ++- 5 files changed, 169 insertions(+), 386 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 05fd2bb..f2f699d 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -1,6 +1,7 @@ name: tests -# Run the full test suite (jpm test) on every push and pull request. +# Run the gate (make test) on every push and pull request. Chez is the sole +# substrate; the JVM is used only as the conformance oracle (certify). on: push: pull_request: @@ -8,53 +9,27 @@ on: jobs: test: runs-on: ubuntu-latest - env: - JANET_VERSION: v1.41.2 # bump to match the version Jolt is developed against - # Per-file deadline for the clojure-test-suite battery. Finite files finish - # in well under 1s; the genuinely-infinite ones get killed at any deadline. - # A generous value gives slow CI runners headroom so a sub-second file - # spiking doesn't time out and drop total-pass below the baseline. - JOLT_SUITE_TIMEOUT: "20" steps: - uses: actions/checkout@v5 with: - # Submodules: vendor/sci (SCI bootstrap/runtime tests) and - # vendor/clojure-test-suite (the cross-dialect conformance battery, - # asserted against a baseline by clojure-test-suite-test.janet). - submodules: recursive + submodules: recursive # vendor/irregex, used by the Chez regex shim - - name: Cache Janet build - id: cache-janet - uses: actions/cache@v5 - with: - path: /tmp/janet - key: janet-${{ env.JANET_VERSION }}-${{ runner.os }} - - - name: Build Janet - if: steps.cache-janet.outputs.cache-hit != 'true' + - name: Install Chez Scheme run: | - git clone --depth 1 --branch "$JANET_VERSION" https://github.com/janet-lang/janet.git /tmp/janet - make -C /tmp/janet + sudo apt-get update + sudo apt-get install -y chezscheme + # the gate invokes `chez`; Debian/Ubuntu installs it as `scheme` + if ! command -v chez >/dev/null 2>&1; then + sudo ln -s "$(command -v scheme)" /usr/local/bin/chez + fi + chez --version || true - - name: Install Janet - run: sudo make -C /tmp/janet install - - - name: Install jpm + - name: Install JDK + Clojure (certify oracle) run: | - git clone --depth 1 https://github.com/janet-lang/jpm.git /tmp/jpm - # bootstrap.janet resolves jpm/cli.janet relative to the cwd, so it - # must run from inside the jpm checkout. - cd /tmp/jpm - sudo janet bootstrap.janet + sudo apt-get install -y default-jdk rlwrap + curl -L -O https://github.com/clojure/brew-install/releases/latest/download/linux-install.sh + sudo bash linux-install.sh + clojure --version - - name: Janet version - run: janet -v - - - name: Build executable - run: jpm build - - - name: Run tests - # Parallel runner (same file set as `jpm test`, across CPU workers). - # The executable is built in the previous step, so binary-spawning tests - # don't skip. - run: janet run-tests.janet + - name: Gate + run: make test diff --git a/CLAUDE.md b/CLAUDE.md index aa36b0b..5f05eba 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -53,75 +53,64 @@ bd close # Complete work ## Build & Test -```bash -jpm build # build/jolt (one binary; ctx baked at build time) -jpm build; janet run-tests.janet # FULL gate, PARALLEL (across CPU workers) — build first so binary-spawning tests don't skip; JOLT_TEST_JOBS overrides worker count, -v shows all output -jpm test # FULL gate, serial (same file set; slower — recursive over test/) -janet test/spec/.janet # one spec file -janet test/integration/conformance-test.janet # 3-mode conformance (interpret/compile/self-host) -JOLT_BENCH=1 janet test/bench/core-bench.janet # bench (opt-in; skipped in the gate) — back-to-back vs main, never absolute -``` - -**Run the gate with a REAL exit code.** `jpm test | grep ...` reports grep's -exit, not jpm's — this once shipped masked spec failures. Correct form: +No build step — `bin/joltc` runs off the checked-in seed (`host/chez/seed/`). +The gate is pure Chez (+ Clojure for the JVM oracle); no Janet. ```bash -jpm test > /tmp/gate.out 2>&1; echo "EXIT: $?" -grep -E "non-zero exit|All tests" /tmp/gate.out +bin/joltc -e EXPR # run a Clojure expression on Chez +make test # FULL gate (self-host + corpus + unit + smoke + certify) +make corpus # conformance corpus vs the JVM-sourced spec (floor 2678) +make unit # host-specific unit cases (test/chez/unit.edn) +make selfhost # bootstrap fixpoint (rebuild == checked-in seed) +make certify # JVM oracle (skips if clojure absent) +chez --script host/chez/run-corpus.ss # the corpus gate directly; JOLT_CORPUS_LIMIT=N for a fast stride +make remint # re-mint the seed after a seed-source change ``` -The literal `All tests passed.` line must be present. CI (.github/workflows/ -tests.yml) runs the same gate on every push/PR. +**Re-mint after changing a seed source.** The reader (`host/chez/reader.ss`), the +analyzer/IR/backend (`jolt-core/jolt/*.clj`), or the `clojure.core` overlay +(`jolt-core/clojure/core/*.clj`) are baked into the seed — change one and run +`make remint` (iterates `host/chez/bootstrap.ss` to a byte-fixpoint) or `make +selfhost` fails. Runtime-only `host/chez/*.ss` shims do NOT need a re-mint. -`jpm build` output goes STALE silently — `rm -rf build && jpm clean` before -trusting the binary, or test from source (authoritative). +**Run the gate with a REAL exit code.** `make test > /tmp/gate.out 2>&1; echo +"EXIT: $?"` — the final `OK: all gates passed` line must be present. CI +(`.github/workflows/tests.yml`) runs `make test` on every push/PR. ## Architecture Overview -Clojure on Janet. A shrinking Janet seed (`src/jolt/*.janet`: reader, value -layer, vars/ns, evaluator, the self-hosted pipeline's back end) hosts a -Clojure overlay (`jolt-core/`): the analyzer/IR (`jolt-core/jolt/`) and -`clojure.core` in dependency-ordered tiers (`jolt-core/clojure/core/NN-*.clj`, -loaded in order: 00-syntax, 00-kernel (bootstrap-compiled), 10-seq, 20-coll, -25-sorted, 30-macros, 40-lazy, 50-io). Compile is the default path (analyzer --> IR -> Janet bytecode, hybrid with interpreter fallback); `JOLT_INTERPRET=1` -forces the tree-walking interpreter, `JOLT_INTERPRET_MACROS=1` additionally -keeps macro expanders interpreted (the pure oracle). `api/init-cached` serves -a disk-cached ctx image (~5ms vs ~2.4s); the cache key fingerprints sources + -env knobs — add any NEW ctx-shaping env var to `image-cache-path` in -api.janet or tests will see stale language behavior. +Clojure on Chez Scheme — the sole substrate, no Janet. A small Chez runtime +(`host/chez/*.ss`: value model, persistent collections, seqs, vars/ns, host +interop) hosts a portable Clojure overlay (`jolt-core/`): the +reader/analyzer/IR/backend (`jolt-core/jolt/`) and `clojure.core` in +dependency-ordered tiers (`jolt-core/clojure/core/NN-*.clj`, loaded in order: +00-syntax, 00-kernel, 10-seq, 20-coll, 25-sorted, 30-macros, 40-lazy, 50-io). +The stdlib namespaces (`clojure.string`/`set`/`walk`/`edn`/`pprint`/…) are +portable Clojure under `src/jolt/clojure/`. + +`bin/joltc` (`host/chez/cli.ss`) loads the checked-in seed +(`host/chez/seed/{prelude,image}.ss`) + the spine and compiles+evals on Chez +(read → analyze → IR → emit → eval). `host/chez/bootstrap.ss` rebuilds that seed +from source on pure Chez; the build is a self-hosting fixpoint (a rebuild +reproduces the checked-in seed byte-for-byte — `make selfhost`). The correctness +oracle is the JVM-sourced conformance corpus (`test/chez/corpus.edn`, +`test/conformance/`). Issue tracking and design notes live in beads (`bd prime`, `bd memories`). ## Conventions & Patterns -Porting seed fns to the overlay (the jolt-tzo shrink ladder) — traps that have -each bitten at least once: - -- **Verify leaf-ness first**: grep ALL `src/jolt/*.janet` for the `core-X` - name (defn + core-bindings entry only), and check that tiers loading - EARLIER than the target tier don't call it. Nothing the analyzer/ir use may - move below the kernel tier. -- **Delete the seed defn + binding in the same change.** A leftover stub - breaks direct-linked self-recursion: the overlay fn's recursive call binds - to the STUB's root at compile time (line-seq once truncated after one - element this way). - **A tier may only use macros from tiers that load before it.** Compile mode - expands macros at tier LOAD; the interpreter expands lazily — so an - if-let (30-macros) inside a 20-coll fn passes every interpreted test and - breaks compiled init. + expands macros at tier LOAD, so an `if-let` (30-macros) inside a 20-coll fn + breaks compiled init even though it passes when expanded lazily. Same ordering + for expander-called fns (empty?/keys/vals live in 00-syntax). - **Never read your own wrapper's fields with `get`** in attached-ops values (sorted colls): `get` on the wrapper IS the dispatched lookup and recurses forever. Use `jolt.host/ref-get`. - **Map literals with `:jolt/type` as a key** parse as tagged reader forms — don't tag overlay value maps in source. -- **Expander-called fns live in 00-syntax** (empty?/keys/vals): expansion - first happens during the kernel-tier compile, before later tiers exist. - Early defns and expanders are interpreted during init and recompiled by the - staged passes (recompile-defns!/recompile-macros!) once the analyzer is - alive. - **Fix latent bugs to match Clojure** rather than preserving them, with a - regression spec row. Canonical Clojure definitions are preferred verbatim. -- **Gate every batch**: conformance x3 modes, suite >= baseline - (clojure-test-suite-test.janet — raise the baseline when it rises), full - jpm test with a real exit code, bench back-to-back vs main. + regression case. Match the JVM (or provide a superset); the JVM-sourced corpus + is the contract. +- **Gate every change**: `make test` with a real exit code (self-host fixpoint, + corpus floor, unit, cli smoke, certify). Re-mint if a seed source changed. diff --git a/README.md b/README.md index e9de902..82693ed 100644 --- a/README.md +++ b/README.md @@ -2,266 +2,106 @@ [![tests](https://github.com/jolt-lang/jolt/actions/workflows/tests.yml/badge.svg)](https://github.com/jolt-lang/jolt/actions/workflows/tests.yml) -A Clojure implementation on top of [Janet](https://janet-lang.org). Jolt reads Clojure source and, by default, compiles each form to native Janet bytecode — falling back to a tree-walking interpreter for forms the compiler doesn't handle, so results always match the interpreter. It ships a Clojure-compatible standard library. The goal is a Janet-hosted [SCI](https://github.com/borkdude/sci)-style runtime with a minimal bootstrap. +A Clojure implementation on [Chez Scheme](https://cisco.github.io/ChezScheme/). +Jolt reads Clojure source, analyzes it to a host-neutral IR, emits Scheme, and +runs it on Chez. The compiler is self-hosted: it is written in Clojure +(`jolt-core/`) and compiles itself. It ships a Clojure-compatible standard library. + +## Requirements + +Only [Chez Scheme](https://cisco.github.io/ChezScheme/) (the gate invokes it as +`chez`). The conformance gate additionally uses Clojure on the JVM as an oracle, +but running jolt does not. ## Build +There is no build step. The bootstrap seed (`host/chez/seed/{prelude,image}.ss`) +is checked in, so a fresh clone runs immediately: + ```bash -git clone https://github.com/jolt-lang/jolt.git +git clone --recurse-submodules https://github.com/jolt-lang/jolt.git cd jolt -git submodule update --init # pulls vendor/sci and vendor/clojure-test-suite -jpm build # builds build/jolt (one binary) +bin/joltc -e '(+ 1 2)' # => 3 ``` -Requires `jpm` and a recent Janet (CI-tested against 1.41). See -[docs/building-and-deps.md](docs/building-and-deps.md) for build details, the -`jpm clean` caveat, how namespaces are resolved (`JOLT_PATH`), and pulling -Clojure libraries from a `deps.edn` (resolved by `jolt` itself — `jolt -M:…`, -`jolt run`, `jolt path`). +After changing a compiler source — the reader (`host/chez/reader.ss`), the +analyzer/IR/backend (`jolt-core/jolt/*.clj`), or the `clojure.core` overlay +(`jolt-core/clojure/core/*.clj`) — re-mint the seed: + +```bash +make remint # iterates host/chez/bootstrap.ss to a byte-fixpoint +``` ## Run +```bash +bin/joltc -e EXPR # evaluate a Clojure expression and print the result ``` -build/jolt # start a REPL -build/jolt file.clj [args] # run a file (binds *command-line-args* and *file*) -build/jolt -e EXPR [args] # evaluate EXPR and print the result -build/jolt -m NS [args] # require NS and call its -main -build/jolt nrepl-server [addr] # start an nREPL server ([host:]port, default 7888) -build/jolt --version # print the version -build/jolt -h | --help # help -``` - -The REPL accumulates multi-line forms until they balance: - -``` -user=> (defn fib [n] (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2))))) -#'user/fib -user=> (map fib (range 10)) -(0 1 1 2 3 5 8 13 21 34) -``` - -Running a file evaluates its top-level forms: - -``` -$ echo '(println "hello" (* 6 7))' > hello.clj -$ build/jolt hello.clj -hello 42 -``` - -## Use as a library - -```janet -(use jolt/api) - -(def ctx (init)) -(eval-string ctx "(+ 1 2)") # → 3 -(eval-string ctx "(map inc [1 2 3])") # → (2 3 4) ; a lazy seq, like Clojure -``` - -`(init)` returns a context with `clojure.core` loaded. Each context is isolated; use separate contexts for separate environments. - -### Evaluation pipeline: interpreted and compiled - -Every form passes through one router (`loader/eval-toplevel`) that decides *per -form* whether to tree-walk it or compile it to Janet bytecode. The shipped -runtime **compiles by default**; set `JOLT_INTERPRET=1` to force the interpreter. - -**Hybrid, always correct.** The compiler is incomplete by design: a form it can't -compile correctly throws `jolt/uncompilable`, and the router falls back to the -tree-walking interpreter (`eval-form`) for that form. So the result *always* -matches the interpreter — compilation is a transparent speedup, never a semantic -change. Only the compile step is guarded; runtime errors in compiled code -propagate normally (no double-evaluation, no hidden errors). - -What compiles: `def`/`defn`, multi-arity / named / variadic fns, `recur` (in -`loop` and directly in `fn`), `let`/`if`/`do`/`try`/`throw`/`quote`, map and -vector literals, and calls. What falls back to the interpreter: context-modifying -and definitional forms (`ns`, `defmacro`, `deftype`, `defprotocol`, -`defmulti`/`defmethod`, `reify`, `require`, `binding`, …), destructuring, regex -literals, and the handful of interpreter-only special forms. - -**Live redefinition.** Compiled global references deref through Jolt **var cells** -(Janet early-binds plain symbols, which would freeze redefinition), so redefining -a `def`/`defn` at the REPL is visible to already-compiled callers — Clojure's var -model. Hot numeric primitives (`+ - * < > <= >=`) emit native Janet ops, and -calls compile to direct Janet calls. - -```janet -(def ctx (init {:compile? true})) -(eval-string ctx "(defn fib [n] (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2)))))") -(eval-string ctx "(fib 30)") ; → 832040, native Janet bytecode -``` - -For compute-heavy code the compiled path is dramatically faster than tree-walking, -at native Janet speed. - -**Validated at parity.** The conformance suite passes 258/258 under *all three* -execution paths — interpreter, compiler, and the self-hosted compiler -(`conformance-test.janet` runs all three in CI) — and the full clojure-test-suite -matches its baseline across ~4.6k assertions — evidence the hybrid path doesn't -diverge. - -**AOT.** `aot.janet` marshals a compiled namespace to a Janet bytecode image -(`save-ns`) and loads it back into a fresh context (`load-ns-image`), skipping -parse/analyze/emit/compile on reload. Core fns are referenced by name against the -baked-in runtime; only user bytecode and var cells are serialized. - -## Host interop - -Jolt exposes CLJS-style host interop through `.` on any Janet table or struct — a field holding a function is called with the receiver as the first argument: - -```clojure -(def obj {:greet (fn [self name] (str "Hello " name))}) -(. obj greet "Alice") ; → "Hello Alice" -(.-greet obj) ; field access (reader sugar for (. obj :greet)) -``` - -### The `janet` interop bridge - -The whole Janet standard library is reachable from Clojure through an explicit -`janet` namespace segment, which marks every crossing into host code (where -Clojure semantics no longer hold): - -```clojure -(janet.os/clock) ; → a Janet module fn: os/clock -(janet.string/join ["a" "b"] ",") ; → janet `string/join` (NB: takes a Janet - ; tuple, not a Jolt vector — convert first) -(janet/slurp "deps.edn") ; → a Janet root builtin: slurp -(janet/type [1 2]) ; → :table -``` - -The rule is `janet/` for a Janet root binding and `janet./` -for a module binding. Because the boundary is explicit, you can tell at the call -site that a form drops into the host — and that values cross the boundary as -their Janet representations (a Jolt vector is a Janet table, etc.), so a Janet -function expecting a tuple needs an explicit conversion. The `jolt.interop`, -`jolt.shell`, and `jolt.http` namespaces are thin Clojure wrappers built on this. - -This bridge is what makes networking (and everything else in Janet's stdlib) -available to ordinary Clojure — for example, `jolt.nrepl` (below) is plain -Clojure over `janet.net/*`. It also reaches any **jpm-installed module**: -the first reference to `janet./` requires the module from the -janet module path and caches its bindings — so after `jpm install spork`, -`janet.spork.http/*` just works. The `jolt.http` client and the Ring adapter -in [examples/ring-app](https://github.com/jolt-lang/examples/tree/main/ring-app) -are built on spork/http this way, and a project can declare the requirement -in `deps.edn` with a `:jpm/module` coordinate (see docs/tools-deps.md). - -```clojure -(require '[jolt.interop :as j]) -(j/janet-type [1 2]) ; → :tuple -(j/janet-table-keys {:a 1 :b 2}) ; → [:b :a] -``` - -## nREPL - -Jolt ships an [nREPL](https://nrepl.org) server and client (`jolt.nrepl`), -written in Clojure on top of the `janet.net/*` bridge. Start a server from the -CLI — it writes `.nrepl-port` so editors (CIDER, Calva, …) auto-connect: ```bash -jolt nrepl-server # listen on 127.0.0.1:7888, write .nrepl-port -jolt nrepl-server 12345 # choose a port -jolt nrepl-server 0.0.0.0:12345 # choose host and port (alias: nrepl) +$ bin/joltc -e '(->> (range 10) (filter even?) (map (fn [x] (* x x))) (reduce +))' +120 +$ bin/joltc -e '(/ 1 2)' +1/2 ``` -In a project with a `deps.edn`, `jolt nrepl-server` auto-resolves it, so the -server starts with the project and its dependencies on the path — connect your -editor, then `(require 'your.app)`. (Equivalently, add `:aliases {:nrepl -{:main-opts ["nrepl-server"]}}` and run `jolt -M:nrepl`.) +## Architecture -Supported ops: `clone`, `describe`, `eval`, `load-file`, `close`, `ls-sessions`, -`interrupt` (acknowledged; an in-flight eval can't actually be interrupted), and -`eldoc`. `eval` streams `out`, reports the current `ns`, evaluates each form in -the message, and returns an `eval-error` status (the session stays usable) on -failure. One Jolt runtime backs the server and sessions share it, so `def`s -persist across a connection like a normal dev REPL. +A small Chez runtime (`host/chez/*.ss`: value model, persistent collections, seqs, +vars/namespaces, host interop) hosts a portable Clojure overlay (`jolt-core/`): the +reader/analyzer/IR/backend (`jolt-core/jolt/`) and `clojure.core` in +dependency-ordered tiers (`jolt-core/clojure/core/NN-*.clj`). The stdlib namespaces +(`clojure.string`/`set`/`walk`/`edn`/`pprint`/…) are portable Clojure under +`src/jolt/clojure/`. -It's also usable as a library — embed a server, or drive another nREPL as a -client: - -```clojure -(require '[jolt.nrepl :as nrepl]) -(def server (nrepl/start-server! {:port 7888})) -;; ... later ... -(nrepl/stop-server! server) - -(def c (nrepl/connect {:port 7888})) -(def session (nrepl/client-clone c)) -(nrepl/client-eval c "(+ 1 2)" session) ; → responses incl. {"value" "3"} -(nrepl/client-close c) -``` +`bin/joltc` loads the checked-in seed and the spine, then compiles and evaluates on +Chez (read → analyze → IR → emit → eval). `host/chez/bootstrap.ss` rebuilds that +seed from source on pure Chez; the build is a self-hosting fixpoint (a rebuild +reproduces the checked-in seed byte-for-byte). ## Differences from Clojure -Jolt targets Clojure semantics but runs on Janet, not the JVM. The notable divergences: +Jolt targets Clojure semantics but runs on Chez, not the JVM. -- **Host platform.** No JVM and no Java interop — `import`, `gen-class`, `proxy` of Java classes, and `java.*` are unavailable. `instance?` recognizes a small set of built-in types (`clojure.lang.Atom`, `Number`, `String`, …). -- **Numbers.** Janet integers and doubles. `(/ 1 3)` is `0.3333…` and large products lose precision. No ratios or `BigDecimal` (`ratio?` is always false, `bigdec` falls back to a double); `bigint`/`biginteger` use Janet's 64-bit `int/s64`, not arbitrary precision. The reader still accepts Clojure's numeric literal syntaxes — the BigInt/BigDecimal suffixes (`42N`, `1.5M`), ratios (`1/2`), radixed integers (`2r1010`, `16rFF`), and exponents (`1e3`) — but reads them as plain Janet numbers (a ratio becomes its double quotient). The auto-promoting `+'`/`-'`/`*'`/`inc'`/`dec'` are aliases for the plain ops, since Janet numbers don't overflow. `quot`/`rem`/`mod` follow Clojure's sign rules. The symbolic values `##Inf`/`##-Inf`/`##NaN` read, and `infinite?`/`NaN?` work. Janet represents an integer and an integer-valued double identically, so `1` and `1.0` are indistinguishable: `(float?/double? 1.0)` is `false` and `(int? 1.0)` is `true` — `float?`/`double?` are true only for values with a fractional part or `##Inf`/`##NaN`. -- **Collections.** By default Jolt uses immutable persistent data structures: vectors are 32-way branching tries (structural-sharing persistent vectors with O(log₃₂ n) `conj`/`assoc`/`nth`), lists are persistent singly-linked cons cells (O(1) `conj`/`cons` prepend with structural sharing), and maps/sets are persistent hash structures. Value equality and sequence operations are Clojure-compatible, but hash-map/hash-set iteration order is unspecified and differs from Clojure — use `sorted-map`/`sorted-set` when order matters. -- **Mutable build mode.** Jolt can be compiled to use fast Janet-native *mutable* collections instead, via a build-time flag: `JOLT_MUTABLE=1 jpm build` (default `jpm build` is immutable). In mutable mode vectors and lists share one mutable array representation (so `conj` mutates in place and appends, and `vector?`/`list?` no longer distinguish them) — a performance/looseness trade-off. The default immutable build has full Clojure value semantics. -- **Concurrency / STM.** No refs, `dosync`, agents, or `send`; `locking` evaluates its body without real locking. Atoms, volatiles, promises, and delays are supported. -- **Futures.** `future` runs its body on a *real* OS thread (Janet's `ev/thread`), so it can use a second core for CPU-bound work — unlike the cooperatively-scheduled `go` blocks. `deref`/`@` parks until the result is ready (with the optional `(deref f timeout-ms timeout-val)` arity); `future?`, `future-done?`, `realized?`, `future-cancel`, and `future-cancelled?` are supported. Two important divergences from the JVM: (1) **snapshot semantics** — Janet threads have separate heaps, so the body and the state it closes over are *copied* to the worker thread and only the return value is copied back; mutating a captured atom does not propagate to the parent (communicate via the return value). (2) **no thread interruption** — Janet OS threads can't be cancelled mid-run, so `future-cancel` marks the *future* cancelled (deref then throws and the predicates flip) but the underlying computation still runs to completion in the background. As on the JVM, a live future thread keeps the process alive until it finishes (the JVM's non-daemon future pool behaves the same). -- **core.async.** `clojure.core.async` runs on Janet fibers and channels (`chan`, `go`, `go-loop`, `!`/`!!`, `close!`, `alts!`, `timeout`, `put!`/`take!`, `buffer`/`dropping-buffer`/`sliding-buffer`, and channel transducers via `(chan n xform)`). Because Janet fibers are stackful coroutines, a `go` block is just its body run in a fiber — no CPS/state-machine rewrite — so `!` work *anywhere*, including inside `try`, nested `fn`s, and loops (positions Clojure's `go` macro forbids). Go blocks are cooperatively scheduled on one OS thread, so parking (`…)`), and Unicode property classes (`\p{Lu}`). -- **Arrays.** Java-style arrays map onto Janet's native types: `byte-array` is a Janet buffer (contiguous, C-backed); `object-array`/`int-array`/`double-array`/etc. are Janet arrays. `aget`/`aset`/`alength`/`aclone` work over both. -- **Transients.** `transient`/`conj!`/`assoc!`/`dissoc!`/`disj!`/`pop!`/`persistent!` are real mutable scratch collections backed by Janet's native arrays and tables (vectors → arrays, maps/sets → tables), so building a collection with them avoids the per-step copying of the persistent path (notably for maps/sets). `persistent!` freezes back to a persistent value. -- **Not implemented.** JVM reflection, `proxy`, and the `clojure.repl`/`clojure.template` namespaces. +- **Host platform.** No JVM and no Java interop — `import`, `gen-class`, `proxy` of + Java classes, and `java.*` are unavailable. A class token resolves to a name; a + small set of host classes is recognized for `instance?`. +- **Numbers.** The full Scheme numeric tower, matching the JVM: exact integers and + bignums, exact ratios (`(/ 1 2)` ⇒ `1/2`), and flonum doubles. `=` is + category-aware (`(= 3 3.0)` ⇒ `false`); `==` is value-equality (`(== 3 3.0)` ⇒ + `true`). `integer?`/`int?` are exact integers, `float?`/`double?` are flonums, + `ratio?` is an exact non-integer. No `BigDecimal` (`decimal?` is always false). +- **Concurrency.** `future`/`promise`/`agent`/`pmap` run on real OS threads over a + **shared heap**, matching JVM semantics (not isolated-heap snapshots). Atoms use a + per-atom mutex with JVM-style CAS. `clojure.core.async` provides blocking channels + and `go`/`!`/`alts!`/`timeout`. +- **Regex.** Backed by [irregex](https://github.com/ashinn/irregex) (vendored), + PCRE/Java-style patterns. +- **Collections.** Immutable persistent vectors (32-way tries), cons lists, and HAMT + maps/sets. Hash-map/hash-set iteration order is unspecified — use + `sorted-map`/`sorted-set` when order matters. Transients are real mutable scratch + collections. -Supported and Clojure-compatible: chars as a distinct type, lazy/infinite sequences, transducers, destructuring, multimethods with hierarchies, protocols/records (`deftype`/`defrecord`/`reify`/`extend-protocol`), metadata, namespaces, and the reader (`#()`, `#_`, `#?`, tagged literals, `#"…"`). +Supported and Clojure-compatible: lazy/infinite sequences, transducers, +destructuring, multimethods with hierarchies, protocols/records +(`deftype`/`defrecord`/`reify`/`extend-protocol`), metadata, namespaces, runtime +`eval`/`load-string`/`defmacro`, and the reader (`#()`, `#_`, `#?`, tagged literals, +`#"…"`). ## Test -``` -jpm test # full suite (recurses test/) -janet test/spec/sequences-spec.janet # a single spec -janet test/integration/conformance-test.janet +```bash +make test # the full gate (no Janet) +make corpus # conformance corpus vs the JVM-sourced spec +make unit # host-specific unit cases +make selfhost # bootstrap fixpoint (rebuild == checked-in seed) +make smoke # bin/joltc CLI smoke +make certify # JVM oracle (skips if clojure is absent) ``` -Tests are organized in three layers: - -- **`test/spec/`** — the contract. Black-box, behavior-defining tables (one file - per public API area) that collectively pin down Jolt's defined behavior. This - is the authoritative description of what Jolt promises. -- **`test/integration/`** — cross-cutting and regression batteries: the Clojure - conformance suite (run in all three execution modes), SCI bootstrap/runtime - loading, jank conformance, the cross-dialect - [clojure-test-suite](https://github.com/jank-lang/clojure-test-suite) (a git - submodule at `vendor/clojure-test-suite`, run via a minimal `clojure.test` shim - and baseline-guarded), compile-mode tests, the library API, and a broad - systematic-coverage net. -- **`test/unit/`** — white-box tests for individual components (reader, - evaluator, types, persistent collections, regex, compiler). - -`test/support/harness.janet` provides the shared `defspec` table runner (cases -are `["label" expected actual]`, compared with Jolt's own `=`) plus -`expect=`/`expect-throws` for unit tests. - -The syntactic half of the contract — the surface syntax the reader accepts — is -specified as an EBNF grammar in [`docs/grammar.ebnf`](docs/grammar.ebnf), with -Jolt-vs-Clojure deviations noted inline. `test/spec/reader-syntax-spec.janet` -exercises it. - -### clojure-test-suite conformance - -The [clojure-test-suite](https://github.com/jank-lang/clojure-test-suite) battery -(vendored as a git submodule) runs ~3980 assertions green. Jolt validates its -arguments like Clojure — arithmetic on non-numbers, comparisons against `nil`, -out-of-range indices, malformed `conj!`/`assoc!`/`merge`, non-seqable -`first`/`seq`/`vec`, and lazy transformers (`map`/`filter`/…) realized over a -non-seqable all throw. The lazy seq fns return seqs (not vectors), so -`seq?`/`vector?`/`sequential?` of their results match Clojure. The assertions -that remain failing are accounted for by the platform/design differences above, -not by missing behavior: - -- **No bignum/ratio/BigDecimal** — `bigint`/`numerator`/`denominator`/`bigdec`, - the `big-int?`/auto-promotion checks, and the `2N`/`1/2`/`1.0M` literals read - but don't carry those exact types. -- **Integer/float identity** — Janet represents `1` and `1.0` identically, so - `quot`/`rem`/`mod`'s `double?`/`int?` result-type assertions and many - `float?`/`double?` cases can't distinguish them (`(str 0.0)` is `"0"`). -- **64-bit integers / Unicode** — `bit-and` etc. on full-width 64-bit constants - lose precision (doubles), and `subs`/`count` work on bytes, not code points. +The conformance corpus (`test/chez/corpus.edn`) is a host-neutral language spec +whose expected values are sourced from reference JVM Clojure. See +[test/conformance/SPEC.md](test/conformance/SPEC.md). ## License diff --git a/test/chez/README.md b/test/chez/README.md index 478bdf8..db51aad 100644 --- a/test/chez/README.md +++ b/test/chez/README.md @@ -1,60 +1,41 @@ # Chez test harness -The correctness gate for jolt on Chez (epic jolt-cf1q). Correctness is judged -against the JVM-sourced conformance spec — no Janet decides pass/fail. The spec -itself lives in `test/conformance/` (see its `SPEC.md`); this directory holds the -Chez runners and unit tests. +The correctness gate for jolt. Pure Chez (+ Clojure for the JVM oracle), no Janet. +Correctness is judged against the JVM-sourced conformance spec; the spec itself +lives in `test/conformance/` (see its `SPEC.md`). Run the whole gate with `make +test` from the repo root. ## The spec corpus -- `corpus.edn` — the contract: ~2920 rows `{:suite :label :expected :actual}`, - with `:expected` sourced from reference JVM Clojure. Valid as both EDN and Janet - data. Generated by `test/conformance/regen-corpus.clj` (the answers) over the - case list from `extract-corpus.janet` (the `:actual` strings, pulled from - `test/spec/*.janet` and `conformance-test.janet`). -- `extract-corpus.janet` — re-derives the case list when spec rows change. Writes - only when `JOLT_EXTRACT_CORPUS_OUT` is set, so a gate run never clobbers the - JVM-sourced `corpus.edn`. After re-deriving, re-source the answers with - `regen-corpus.clj`. +`corpus.edn` is the contract: ~2920 rows `{:suite :label :expected :actual}`, with +`:expected` sourced from reference JVM Clojure by `test/conformance/regen-corpus.clj`. +It is frozen (the canonical source) — add or change cases here, then re-source the +answers with `regen-corpus.clj` and re-certify with `test/conformance/certify.clj`. -## The standing gate +## The gate runners (`host/chez/`) -`run-corpus-zero-janet.janet` runs every corpus case through the zero-Janet -spine: read → analyze → IR → emit → eval, all on Chez (the analyzer is -`jolt.analyzer` cross-compiled to Scheme over `host-contract.ss`). Each result is -compared by value-equality against the JVM `:expected`. A `known-fail` allowlist -covers cases jolt can't match because Chez has no JVM host (Java classes, arrays, -`BigDecimal`, opaque host-object printers, …); the gate fails only on a NEW -divergence or if the pass count drops below the floor. +- `run-corpus.ss` — runs every corpus case through the spine (read → analyze → IR → + emit → eval, all on Chez), comparing each result by value-equality against the JVM + `:expected`. A `known-fail` allowlist covers cases jolt can't match because Chez has + no JVM host (Java classes, arrays, `BigDecimal`, opaque host-object printers, …); + the gate fails only on a NEW divergence or if the pass count drops below the floor. - JOLT_CHEZ_ZEROJANET_CORPUS=1 janet test/chez/run-corpus-zero-janet.janet - JOLT_CORPUS_LIMIT=200 … # every-Nth stride, fast iteration + chez --script host/chez/run-corpus.ss + JOLT_CORPUS_LIMIT=200 … # every-Nth stride, fast iteration + JOLT_CHEZ_ZJ_FLOOR=N … # override the floor (default 2678) -Floor: 2678 (`JOLT_CHEZ_ZJ_FLOOR` overrides). Raise it as host gaps close. +- `run-unit.ss` — host-specific unit cases (`test/chez/unit.edn`) that aren't in the + JVM-portable corpus: dot-forms, java statics, io, reader, walk, vars/namespaces, + refs. Each `:expr` is evaluated in-process and its printed value compared to a baked + `:expected` (`:throws` asserts a raise). -## Unit tests +- `selfcheck.sh` — self-host fixpoint: `bootstrap.ss` rebuild byte-equals the + checked-in seed (`host/chez/seed/`). +- `smoke.sh` — real `bin/joltc -e` CLI smoke. -`_*.janet` drive `bin/joltc` (the pure-Chez CLI; `JOLT_BIN` overrides) one -subprocess per case and compare the last stdout line against a baked expected -value. `:throws` asserts a non-zero exit. Each file targets one area — -`_type`/`_class`/`_seqpred` (taxonomy + predicates), `_str`/`_strns`/`_reader` -(strings + reader), `_io`/`_ioreader`/`_insttime`/`_javastatic` (host interop), -`_dynbind`/`_var_meta`/`_ns` (vars + namespaces), `_atomwatch`/`_walk`/`_dotform` -/`_stdlib` (refs, walk, dot-forms, stdlib). Run one with -`janet test/chez/_type.janet`. +## Other Chez tests -## Self-host gates +- `values-test.ss` — the value model (nil/truthiness/collections). `make values`. +- `bench-chez.ss` — compute bench through the pipeline (opt-in; not in the gate). -- `spine-test.janet` — the zero-Janet compile/eval spine end to end. -- `cli-test.janet` — `bin/joltc -e` behavior. -- `bootstrap-test.janet` — `host/chez/bootstrap.ss` rebuilds the prelude + image - from source on Chez and matches the checked-in seed (`host/chez/seed/`). -- `fixpoint-test.janet` — the on-Chez compiler reproduces itself (stage2 == stage3, - prelude pstage3 == pstage4). - -## Bench - -`bench-pipeline.janet` (opt-in via `JOLT_CHEZ_BENCH=1`) times fib + mandelbrot -through the real pipeline against the spike ceiling. Skipped in the gate. - -All Chez runners skip cleanly when `chez` is not on PATH. +All runners assume `chez` on PATH. diff --git a/test/conformance/SPEC.md b/test/conformance/SPEC.md index 36e9e4f..33ca5a4 100644 --- a/test/conformance/SPEC.md +++ b/test/conformance/SPEC.md @@ -17,10 +17,9 @@ read one data file, run each case, compare, report. | `test/conformance/certify.clj` | Certifies `:expected` against reference **JVM Clojure**; gates on new/stale divergences; emits the profile. | — | `corpus.edn` is **JVM-sourced**: `regen-corpus.clj` evaluates each case's `:actual` -on reference JVM Clojure and writes the JVM value as `:expected`. The case *list* -(the `:actual` strings) comes from `test/spec/*-spec.janet` via `extract-corpus.janet`, -but the *answers* are the JVM's. **`corpus.edn` is the canonical contract**: it is -what every runtime consumes and what `certify.clj` certifies. +on reference JVM Clojure and writes the JVM value as `:expected`. **`corpus.edn` is +the canonical, frozen contract**: it is what every runtime consumes, what +`certify.clj` certifies, and where new cases are authored directly. ## Row schema @@ -101,17 +100,16 @@ implements. Current profile (≈2670 portable, ≈249 non-portable): 3. Run it. Your **conformance level** is the set of feature families with no failures. Portable-only is the floor; each feature you implement raises it. -The two reference harnesses already do exactly this on Chez: -`test/chez/run-corpus-prelude.janet` (Janet analyzer → Chez runtime) and -`test/chez/run-corpus-zero-janet.janet` (Chez analyzer → Chez runtime), both with a -regression floor. +The reference harness does exactly this on Chez: `host/chez/run-corpus.ss` (the +analyzer runs on Chez → Chez runtime), with a regression floor. Run it via `make +corpus`. ## Maintaining the spec -- **Add/change cases**: edit `test/spec/*-spec.janet` or `conformance-test.janet`, - then `janet test/chez/extract-corpus.janet` to regenerate `corpus.edn`. +- **Add/change cases**: edit `test/chez/corpus.edn` directly, then re-source the + answers with `regen-corpus.clj`. - **Re-certify**: `clojure -M test/conformance/certify.clj`. A new divergence is either a real bug (file it, mark the allowlist entry `:bug` + `:bead`) or a deliberate delta (classify it in `known-divergences.edn`). - **Refresh the profile**: re-run with `--profile test/conformance/profile.edn`. -- **Re-floor the runtime gates** when parity rises (`run-corpus-*.janet`). +- **Re-floor the runtime gate** when parity rises (`host/chez/run-corpus.ss`).