From 48d39ecd5adc7144194333b8cda5c7889c18ec66 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Wed, 17 Jun 2026 12:31:21 -0400 Subject: [PATCH] Chez port plan + beads epic (jolt-cf1q) Phased plan for re-hosting jolt's substrate on Chez Scheme, organized around two north stars: minimal host shim (push everything possible into self-hosted jolt-core, drop the tree-walking interpreter) and the spec/conformance corpus as the host-neutral correctness contract. Closes obsolete Janet-backend/cgen beads superseded by the native substrate. --- docs/chez-port-plan.md | 134 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 docs/chez-port-plan.md diff --git a/docs/chez-port-plan.md b/docs/chez-port-plan.md new file mode 100644 index 0000000..96ba68d --- /dev/null +++ b/docs/chez-port-plan.md @@ -0,0 +1,134 @@ +# Re-hosting Jolt on Chez Scheme — phased plan + +Decision (2026-06-17): port jolt's runtime substrate from Janet to Chez Scheme +(cisco/ChezScheme). The spike (`spike/chez/RESULTS.md`) validated the thesis: +the compute-substrate ceiling is ~12-47x faster than Janet (mandelbrot +166->13.4ms matching jolt's own C-codegen result; fib 246->5.2ms), Chez's +native compiler reaches that at runtime with the REPL intact, size stays +single-digit MB, and the one regression (memory baseline ~2.5x) is opt-in +tunable via WPO + stripped boot + AOT-under-petite. + +This plan is built around two north stars beyond raw speed: + +1. **Minimal host shim.** Every line that stays in Scheme is a line we failed to + self-host. The port is the forcing function for jolt-tzo/uqi/lcn: shrink the + Scheme seed to the irreducible primitive set, push everything else into + portable Clojure (`jolt-core/`). +2. **Tests are the contract.** The spec/conformance corpus is host-neutral data + (`[name expected-clj actual-clj]` triples compared via jolt's own `=`). It is + the acceptance gate for "the port is correct" — Chez-jolt must pass the same + corpus the Janet host passes, with no regression to the clojure-test-suite + baseline. + +## The minimal host shim (target end-state) + +What MUST be Scheme (the irreducible primitive layer the self-hosted core rests +on) vs what MOVES into portable Clojure: + +### Stays in Scheme (the shim) +- **Value primitives that can't bottom out in Clojure without circularity:** + the `nil` sentinel (distinct from `#f` and `'()` — the classic Lisp-on-Lisp + trap), keyword/symbol records (Clojure symbols carry ns + meta), char/string + bridging. NOTE: Chez's **numeric tower is a windfall** — int/float/ratio/ + bignum are native, so jolt gets exact ratios + bignums for free (Janet lacked + them). +- **Mutable cell / box + array primitives** the self-hosted RT builds on (var + roots, transients, HAMT node arrays). +- **A type-tag / record primitive** for deftype/protocol dispatch (Chez records). +- **`host/compile`** — eval a backend-emitted Scheme form to a procedure. On Chez + this is literally `eval`/`compile`. Trivial. This is the whole backend + host-dependency. +- **FFI / interop bridge** (foreign-procedure) for host interop calls. +- **Persistent-collection hot nodes** — ONLY IF the Clojure-on-Chez version + (Phase 0c) doesn't hold perf. Open question, decided by measurement. + +### Moves into portable Clojure (`jolt-core/`) +- **The reader** (text -> forms). CLJS self-hosts its reader; ours can too. ~33KB + of Janet leaves the host. Not hot. +- **Analyzer + IR + passes** — already portable Clojure. No change. +- **The backend emitter** — its LOGIC becomes Clojure that emits Scheme forms as + data; only `host/compile` crosses the seam. +- **macros + clojure.core** — finish the jolt-uqi/tzo migration (most already + Clojure). +- **Protocol/multimethod dispatch logic** — over the host tag primitive. +- **Persistent collections** — candidate (Phase 0c), perf-permitting. + +### Dropped entirely +- **The tree-walking interpreter** (`eval_base/eval_runtime/eval_special/ + eval_resolve`, ~140KB Janet). On Chez, native `compile` is always present and + cheap, so the compile-only path can cover every form — no `jolt/uncompilable` + fallback needed. The interpreter's role as the correctness *oracle* transfers + to the spec corpus + JVM Clojure (the real reference), which is strictly + better. This is the single largest shim reduction and the biggest open risk; + Phase 1 validates that compile-only is total before we commit to the drop. + +## Test contract strategy + +- **Corpus is the contract.** The 44 `test/spec/*.janet` files, the conformance + cases, `clojure-test-suite`, and `clojure-stdlib-suite` are host-neutral: pure + Clojure source + expected values. Extract the triples into a runner that can + target an arbitrary `jolt` binary (subprocess at the Clojure boundary). +- **Parity gate.** Chez-jolt must pass the same corpus as Janet-jolt; the + clojure-test-suite baseline is the bar (raise it when it rises, never lower). +- **Oracle shift.** Today's 3-mode conformance (interpret/compile/self-host) + loses the "interpret" leg when the interpreter is dropped; the golden expected + values + JVM Clojure become the oracle. Keep the frozen expected values. +- **Dual-run during migration.** Run BOTH hosts against the corpus until Chez + reaches parity, then retire the Janet host. + +## Phases (-> beads epic) + +**Phase 0 — Foundations & contract harness** (de-risk; no jolt pipeline yet) +- 0a. Chez RT value model: nil sentinel, keyword/symbol records, numeric-tower + mapping, `=`/hashing. Resolve the nil/`'()`/`#f` representation up front. +- 0b. Host-neutral test-contract runner: extract the spec/conformance corpus to + drive an arbitrary jolt binary; stand up the parity-gate machinery. +- 0c. Persistent-collection perf experiment: HAMT/PV in Clojure-on-Chez vs + Scheme-native — the data that decides what stays shim vs self-hosted. + +**Phase 1 — Minimal Chez kernel + real-pipeline bootstrap** +- Scheme shim (value layer, var/ns cells, `host/compile`, cenv impl). +- `jolt.backend` Scheme-emit target for the IR the analyzer already produces. +- Bootstrap jolt-core (ir/analyzer) on Chez; compile + run `(+ 1 2)` -> fib -> + mandelbrot through the REAL pipeline. Gate: compute benches run end-to-end and + hit ~the spike ceiling; confirm compile-only is total (no fallback needed). + +**Phase 2 — clojure.core to spec parity** +- Bring up persistent collections (per 0c) + seq/coll/print/refs/io tiers over + the Chez RT. Gate: spec + conformance + clojure-test-suite parity with the + Janet baseline. + +**Phase 3 — Self-host expansion (shrink the shim)** +- Move the reader into jolt-core. Continue core-* leaf migration (jolt-uqi/ded/ + tzo, now targeting Chez). Drop the tree-walking interpreter. Gate: shim equals + the documented minimal set; parity holds. + +**Phase 4 — Deployment & optimization modes** (the "optimize specific cases" lever) +- Wire `JOLT_WHOLE_PROGRAM`/direct-link to emit specialized Scheme (fl*/fx*), + feeding Chez `compile-whole-program`. `jolt build`: WPO + strip-fasl + + AOT-under-petite + heap tuning -> small fast binary (jolt-0w9u reframed). + Rebuild fibers/async on call/cc + threads. Gate: full bench suite incl. + collections/binary-trees (the GC axes); size + memory measured vs spike + baseline. + +**Phase 5 — Retire the Janet host** +- Chez parity + perf confirmed -> remove `src/jolt/*.janet` seed. jolt-core + unchanged. Janet becomes a historical/alternate host proving portability. + +## Beads reconciliation + +- **Closed (obsolete — Janet bytecode-VM / cgen / Janet-dispatch mechanisms Chez + replaces wholesale):** jolt-ffn (epic, already concluded flat), jolt-5vsp.1, + jolt-qx70, jolt-l1l4 (cgen), jolt-cm7t, jolt-fw2 (Janet dispatch substrate), + jolt-pria (Janet ctx cold-build startup). +- **Recommend close (confirm) — Janet constant-factor passes Chez's JIT/GC/WPO + subsume:** jolt-826, jolt-27w, jolt-t6r, jolt-8flj, jolt-3ko, jolt-t34, + jolt-u1f. (jolt-ffn's own STATUS says the gap is "generic-runtime overhead… + the JVM erases via JIT + inline caching + unboxing" — exactly Chez's job.) +- **Reframed under the Chez epic:** jolt-0w9u + .1 -> Phase 4 deploy mode + + closed-world audit; jolt-1r86 -> Phase 0b/4 bench validation; jolt-lcn / + jolt-uqi / jolt-tzo / jolt-ded / jolt-brh / jolt-7dl -> Phase 2/3 self-hosting, + now targeting Chez; jolt-5vsp (foundational-runtime epic) -> parent, the Chez + port is its realization. +- **Keep host-agnostic:** deps beads (jolt-x4o, jolt-xkd, jolt-pnje, jolt-vley); + correctness bug jolt-jk23.