jolt/docs/chez-port-plan.md

Re-hosting Jolt on Chez Scheme — phased plan

Decision (2026-06-17): port jolt's runtime substrate from Janet to Chez Scheme (cisco/ChezScheme). The spike (spike/chez/RESULTS.md) validated the thesis: the compute-substrate ceiling is ~12-47x faster than Janet (mandelbrot 166->13.4ms matching jolt's own C-codegen result; fib 246->5.2ms), Chez's native compiler reaches that at runtime with the REPL intact, size stays single-digit MB, and the one regression (memory baseline ~2.5x) is opt-in tunable via WPO + stripped boot + AOT-under-petite.

This plan is built around two north stars beyond raw speed:

1. Zero Janet — Chez is the sole substrate (revised 2026-06-18). The goal is not a minimal Janet shim that coexists with Chez; it is to rip Janet out entirely and rely on Chez going forward. Two things must move off Janet: the runtime (a hand-written Chez Scheme RT replaces the Janet value layer / vars / evaluator) and the compiler. The compiler is the subtle part: the analyzer/IR are already portable Clojure, but today they execute on the Janet host, and the IR->Scheme emitter + driver are Janet code (host/chez/ emit.janet), so the current clojure.core prelude is a Janet cross-compile. The end state requires Chez-jolt to run the analyzer itself and the emitter to become portable Clojure — so Chez-jolt compiles its own clojure.core AND the analyzer from source, with no Janet in the loop (the bootstrap fixpoint). Then both src/jolt/*.janet and host/chez/*.janet are deleted. Every line that stays in Scheme is hand-written Chez RT, not Janet; the forcing function for jolt-tzo/uqi/lcn still applies (push logic into jolt-core/).
2. Tests are the contract. The spec/conformance corpus is host-neutral data ([name expected-clj actual-clj] triples compared via jolt's own =). It is the acceptance gate for "the port is correct" — Chez-jolt must pass the same corpus the Janet host passes, with no regression to the clojure-test-suite baseline.

The Chez host RT vs portable Clojure (target end-state)

What MUST be hand-written Chez Scheme (the irreducible primitive layer the self-hosted core rests on) vs what MOVES into portable Clojure. Nothing here is Janet — this is the split after Janet is gone:

Stays in Scheme (the Chez host RT)

• Value primitives that can't bottom out in Clojure without circularity: the nil sentinel (distinct from #f and '() — the classic Lisp-on-Lisp trap), keyword/symbol records (Clojure symbols carry ns + meta), char/string bridging. NOTE: Chez's numeric tower is a windfall — int/float/ratio/ bignum are native, so jolt gets exact ratios + bignums for free (Janet lacked them).
• Mutable cell / box + array primitives the self-hosted RT builds on (var roots, transients, HAMT node arrays).
• A type-tag / record primitive for deftype/protocol dispatch (Chez records).
• host/compile — eval a backend-emitted Scheme form to a procedure. On Chez this is literally eval/compile. Trivial. This is the whole backend host-dependency.
• FFI / interop bridge (foreign-procedure) for host interop calls.
• Persistent-collection hot nodes — ONLY IF the Clojure-on-Chez version (Phase 0c) doesn't hold perf. Open question, decided by measurement.

Moves into portable Clojure (jolt-core/)

• The reader (text -> forms). CLJS self-hosts its reader; ours can too. ~33KB of Janet leaves the host. Not hot.
• Analyzer + IR + passes — already portable Clojure source; the change is that they must EXECUTE on Chez-jolt, not on the Janet host (Phase 3).
• The backend emitter — today it is Janet (host/chez/emit.janet); its LOGIC becomes portable Clojure (jolt.backend-scheme) that emits Scheme forms as data, so it runs on Chez. Only host/compile (Chez eval) crosses the seam.
• macros + clojure.core — finish the jolt-uqi/tzo migration (most already Clojure).
• Protocol/multimethod dispatch logic — over the host tag primitive.
• Persistent collections — candidate (Phase 0c), perf-permitting.

Dropped entirely

• The tree-walking interpreter (eval_base/eval_runtime/eval_special/ eval_resolve, ~140KB Janet). On Chez, native compile is always present and cheap, so the compile-only path can cover every form — no jolt/uncompilable fallback needed. The interpreter's role as the correctness oracle transfers to the spec corpus + JVM Clojure (the real reference), which is strictly better. This is the single largest shim reduction and the biggest open risk; Phase 1 validates that compile-only is total before we commit to the drop.

Test contract strategy

• Corpus is the contract. The 44 test/spec/*.janet files, the conformance cases, clojure-test-suite, and clojure-stdlib-suite are host-neutral: pure Clojure source + expected values. Extract the triples into a runner that can target an arbitrary jolt binary (subprocess at the Clojure boundary).
• Parity gate. Chez-jolt must pass the same corpus as Janet-jolt; the clojure-test-suite baseline is the bar (raise it when it rises, never lower).
• Oracle shift. Today's 3-mode conformance (interpret/compile/self-host) loses the "interpret" leg when the interpreter is dropped; the golden expected values + JVM Clojure become the oracle. Keep the frozen expected values.
• Dual-run during migration. Run BOTH hosts against the corpus until Chez reaches parity, then retire the Janet host.

Phases (-> beads epic)

Phase 0 — Foundations & contract harness (de-risk; no jolt pipeline yet)

• 0a. Chez RT value model: nil sentinel, keyword/symbol records, numeric-tower mapping, =/hashing. Resolve the nil/'()/#f representation up front.
• 0b. Host-neutral test-contract runner: extract the spec/conformance corpus to drive an arbitrary jolt binary; stand up the parity-gate machinery.
• 0c. Persistent-collection perf experiment: HAMT/PV in Clojure-on-Chez vs Scheme-native — the data that decides what stays shim vs self-hosted.

Phase 1 — Minimal Chez kernel + real-pipeline bootstrap

• Scheme shim (value layer, var/ns cells, host/compile, cenv impl).
• jolt.backend Scheme-emit target for the IR the analyzer already produces.
• Bootstrap jolt-core (ir/analyzer) on Chez; compile + run (+ 1 2) -> fib -> mandelbrot through the REAL pipeline. Gate: compute benches run end-to-end and hit ~the spike ceiling; confirm compile-only is total (no fallback needed).

Phase 2 — clojure.core to spec parity

• Bring up persistent collections (per 0c) + seq/coll/print/refs/io tiers over the Chez RT. Gate: spec + conformance + clojure-test-suite parity with the Janet baseline.

Phase 3 — Self-host the compiler on Chez (the no-Janet spine)

• Rewrite the IR->Scheme emitter from Janet (host/chez/emit.janet + driver. janet) into portable Clojure in jolt-core (a jolt.backend-scheme target); folds jolt-lcn. Move the reader into jolt-core. Stand up Chez compile-from- source: Chez-jolt reads the .clj tiers, runs the analyzer executing on Chez (not Janet), emits Scheme, evals — replacing the Janet cross-compile of the prelude. Bootstrap fixpoint: Chez-jolt compiles clojure.core AND the analyzer from source with no Janet in the loop; verify stage2==stage3 emitted forms. Drop the tree-walking interpreter. Continue core-* leaf migration (jolt-uqi/ ded/tzo). Gate: Chez-jolt builds itself from source, full corpus parity holds, zero Janet invoked.

Phase 4 — Deployment & optimization modes (the "optimize specific cases" lever)

• Wire JOLT_WHOLE_PROGRAM/direct-link to emit specialized Scheme (fl*/fx*), feeding Chez compile-whole-program. jolt build: WPO + strip-fasl + AOT-under-petite + heap tuning -> small fast binary (jolt-0w9u reframed). Rebuild fibers/async on call/cc + threads. Gate: full bench suite incl. collections/binary-trees (the GC axes); size + memory measured vs spike baseline.

Phase 5 — Delete the Janet host (single substrate)

• Chez self-hosts + parity + perf confirmed -> delete both src/jolt/*.janet (the seed) and host/chez/*.janet (the Janet emitter/driver, now superseded by the Clojure jolt.backend-scheme). Chez is the only substrate; no alternate host retained. jolt-core unchanged. Oracle stays the spec corpus + JVM Clojure. Net: one host, no Janet, no cross-compile.

Host interop & the examples acceptance corpus

The ../examples repo holds real jolt apps with real git deps + C interop; they are the end-to-end acceptance gate complementing the unit-level spec corpus. The port must account for jolt's interop surface, which is layered:

• janet.* bridge — the general Clojure->Janet escape hatch (janet/get, janet/struct, janet.ev/sleep, janet.net/close, janet.spork.http/*), used mostly in demo glue. DECIDED (2026-06-17): rename the bridge to a neutral host.* namespace (not janet.*/chez.*), so app interop code is host-portable; each host implements host.* over its own FFI (Janet FFI today, Chez foreign-procedure tomorrow). The legacy janet.* aliases stay as deprecated shims during migration.
• FFI-backed shim libraries — the genuine C interop. jolt-lang/http-client implements java.net + TLS + gzip as host shims over Janet FFI, backing clj-http-lite's :clj branch. On Chez these are reimplemented over Chez FFI (libcurl/openssl/zlib or native sockets); the java.*-facing API is unchanged. Also in scope: jolt-lang/db, jolt-lang/logging, and jolt-lang/router (mirrors reitit.Trie; verify whether it carries native code).
• :jpm/module Janet-native deps — spork/http (server), ring-janet- adapter. No Chez equivalent: provide a Scheme/Chez-FFI HTTP server or treat these as test-only fixtures.
• Native-library dependencies in deps.edn (DECIDED 2026-06-17). C-interop shims need shared libraries (libcurl, openssl/libssl, zlib) that CANNOT be pulled from git like Clojure libs. Add a :native dep form so a project can declare what it needs, e.g. {:native/lib "curl" :native/min-version "7.0" :native/header "curl/curl.h"}. The resolver doesn't fetch them, but: (a) it surfaces the requirement to the user (and can suggest the install command per platform), and (b) the loader/foreign-procedure layer probes for the .so/.dylib at load and, if absent, raises a precise error — "missing native library: libcurl (declared by jolt-lang/http-client); install with brew install curl" — instead of a raw dlopen failure. Symmetric with :jpm/module's "verify importable, hint to install" pattern.
• Host-neutral pieces that port for free — Java-class mirror shims (pure Clojure) and JOLT_FEATURES reader conditionals.

Sequencing: the interop bridge mechanism is part of the Phase-1 shim, but the FFI-backed libraries land in Phase 4 (downstream of core parity), validated by running the examples end-to-end. Tracked as a dedicated epic child.

Beads reconciliation

• Closed (obsolete — Janet bytecode-VM / cgen / Janet-dispatch mechanisms Chez replaces wholesale): jolt-ffn (epic, already concluded flat), jolt-5vsp.1, jolt-qx70, jolt-l1l4 (cgen), jolt-cm7t, jolt-fw2 (Janet dispatch substrate), jolt-pria (Janet ctx cold-build startup).
• Recommend close (confirm) — Janet constant-factor passes Chez's JIT/GC/WPO subsume: jolt-826, jolt-27w, jolt-t6r, jolt-8flj, jolt-3ko, jolt-t34, jolt-u1f. (jolt-ffn's own STATUS says the gap is "generic-runtime overhead… the JVM erases via JIT + inline caching + unboxing" — exactly Chez's job.)
• Reframed under the Chez epic: jolt-0w9u + .1 -> Phase 4 deploy mode + closed-world audit; jolt-1r86 -> Phase 0b/4 bench validation; jolt-lcn / jolt-uqi / jolt-tzo / jolt-ded / jolt-brh / jolt-7dl -> Phase 2/3 self-hosting, now targeting Chez; jolt-5vsp (foundational-runtime epic) -> parent, the Chez port is its realization.
• Keep host-agnostic: deps beads (jolt-x4o, jolt-xkd, jolt-pnje, jolt-vley); correctness bug jolt-jk23.