jolt/docs/chez-port-plan.md

# Re-hosting Jolt on Chez Scheme — phased plan

Decision (2026-06-17): port jolt's runtime substrate from Janet to Chez Scheme
(cisco/ChezScheme). The spike (`spike/chez/RESULTS.md`) validated the thesis:
the compute-substrate ceiling is ~12-47x faster than Janet (mandelbrot
166->13.4ms matching jolt's own C-codegen result; fib 246->5.2ms), Chez's
native compiler reaches that at runtime with the REPL intact, size stays
single-digit MB, and the one regression (memory baseline ~2.5x) is opt-in
tunable via WPO + stripped boot + AOT-under-petite.

This plan is built around two north stars beyond raw speed:

1. **Zero Janet — Chez is the sole substrate** (revised 2026-06-18). The goal is
   not a minimal Janet shim that coexists with Chez; it is to rip Janet out
   entirely and rely on Chez going forward. Two things must move off Janet: the
   **runtime** (a hand-written Chez Scheme RT replaces the Janet value layer /
   vars / evaluator) and the **compiler**. The compiler is the subtle part: the
   analyzer/IR are already portable Clojure, but today they *execute on the Janet
   host*, and the IR->Scheme emitter + driver are *Janet code* (`host/chez/
   emit.janet`), so the current `clojure.core` prelude is a Janet cross-compile.
   The end state requires Chez-jolt to run the analyzer itself and the emitter to
   become portable Clojure — so Chez-jolt compiles its own `clojure.core` AND the
   analyzer from source, with no Janet in the loop (the bootstrap fixpoint). Then
   both `src/jolt/*.janet` and `host/chez/*.janet` are deleted. Every line that
   stays in Scheme is hand-written Chez RT, not Janet; the forcing function for
   jolt-tzo/uqi/lcn still applies (push logic into `jolt-core/`).
2. **Tests are the contract.** The spec/conformance corpus is host-neutral data
   (`[name expected-clj actual-clj]` triples compared via jolt's own `=`). It is
   the acceptance gate for "the port is correct" — Chez-jolt must pass the same
   corpus the Janet host passes, with no regression to the clojure-test-suite
   baseline.

## The Chez host RT vs portable Clojure (target end-state)

What MUST be hand-written Chez Scheme (the irreducible primitive layer the
self-hosted core rests on) vs what MOVES into portable Clojure. Nothing here is
Janet — this is the split *after* Janet is gone:

### Stays in Scheme (the Chez host RT)
- **Value primitives that can't bottom out in Clojure without circularity:**
  the `nil` sentinel (distinct from `#f` and `'()` — the classic Lisp-on-Lisp
  trap), keyword/symbol records (Clojure symbols carry ns + meta), char/string
  bridging. NOTE: Chez's **numeric tower is a windfall** — int/float/ratio/
  bignum are native, so jolt gets exact ratios + bignums for free (Janet lacked
  them).
- **Mutable cell / box + array primitives** the self-hosted RT builds on (var
  roots, transients, HAMT node arrays).
- **A type-tag / record primitive** for deftype/protocol dispatch (Chez records).
- **`host/compile`** — eval a backend-emitted Scheme form to a procedure. On Chez
  this is literally `eval`/`compile`. Trivial. This is the whole backend
  host-dependency.
- **FFI / interop bridge** (foreign-procedure) for host interop calls.
- **Persistent-collection hot nodes** — ONLY IF the Clojure-on-Chez version
  (Phase 0c) doesn't hold perf. Open question, decided by measurement.

### Moves into portable Clojure (`jolt-core/`)
- **The reader** (text -> forms). CLJS self-hosts its reader; ours can too. ~33KB
  of Janet leaves the host. Not hot.
- **Analyzer + IR + passes** — already portable Clojure source; the change is
  that they must EXECUTE on Chez-jolt, not on the Janet host (Phase 3).
- **The backend emitter** — today it is Janet (`host/chez/emit.janet`); its LOGIC
  becomes portable Clojure (`jolt.backend-scheme`) that emits Scheme forms as
  data, so it runs on Chez. Only `host/compile` (Chez `eval`) crosses the seam.
- **macros + clojure.core** — finish the jolt-uqi/tzo migration (most already
  Clojure).
- **Protocol/multimethod dispatch logic** — over the host tag primitive.
- **Persistent collections** — candidate (Phase 0c), perf-permitting.

### Dropped entirely
- **The tree-walking interpreter** (`eval_base/eval_runtime/eval_special/
  eval_resolve`, ~140KB Janet). On Chez, native `compile` is always present and
  cheap, so the compile-only path can cover every form — no `jolt/uncompilable`
  fallback needed. The interpreter's role as the correctness *oracle* transfers
  to the spec corpus + JVM Clojure (the real reference), which is strictly
  better. This is the single largest shim reduction and the biggest open risk;
  Phase 1 validates that compile-only is total before we commit to the drop.

## Test contract strategy

- **Corpus is the contract.** The 44 `test/spec/*.janet` files, the conformance
  cases, `clojure-test-suite`, and `clojure-stdlib-suite` are host-neutral: pure
  Clojure source + expected values. Extract the triples into a runner that can
  target an arbitrary `jolt` binary (subprocess at the Clojure boundary).
- **Parity gate.** Chez-jolt must pass the same corpus as Janet-jolt; the
  clojure-test-suite baseline is the bar (raise it when it rises, never lower).
- **Oracle shift.** Today's 3-mode conformance (interpret/compile/self-host)
  loses the "interpret" leg when the interpreter is dropped; the golden expected
  values + JVM Clojure become the oracle. Keep the frozen expected values.
- **Dual-run during migration.** Run BOTH hosts against the corpus until Chez
  reaches parity, then retire the Janet host.

## Phases (-> beads epic)

**Phase 0 — Foundations & contract harness** (de-risk; no jolt pipeline yet)
- 0a. Chez RT value model: nil sentinel, keyword/symbol records, numeric-tower
  mapping, `=`/hashing. Resolve the nil/`'()`/`#f` representation up front.
- 0b. Host-neutral test-contract runner: extract the spec/conformance corpus to
  drive an arbitrary jolt binary; stand up the parity-gate machinery.
- 0c. Persistent-collection perf experiment: HAMT/PV in Clojure-on-Chez vs
  Scheme-native — the data that decides what stays shim vs self-hosted.

**Phase 1 — Minimal Chez kernel + real-pipeline bootstrap**
- Scheme shim (value layer, var/ns cells, `host/compile`, cenv impl).
- `jolt.backend` Scheme-emit target for the IR the analyzer already produces.
- Bootstrap jolt-core (ir/analyzer) on Chez; compile + run `(+ 1 2)` -> fib ->
  mandelbrot through the REAL pipeline. Gate: compute benches run end-to-end and
  hit ~the spike ceiling; confirm compile-only is total (no fallback needed).

**Phase 2 — clojure.core to spec parity**
- Bring up persistent collections (per 0c) + seq/coll/print/refs/io tiers over
  the Chez RT. Gate: spec + conformance + clojure-test-suite parity with the
  Janet baseline.

**Phase 3 — Self-host the compiler on Chez** (the no-Janet spine)
- Rewrite the IR->Scheme emitter from Janet (`host/chez/emit.janet` + `driver.
  janet`) into portable Clojure in jolt-core (a `jolt.backend-scheme` target);
  folds jolt-lcn. Move the reader into jolt-core. Stand up Chez compile-from-
  source: Chez-jolt reads the `.clj` tiers, runs the analyzer *executing on Chez*
  (not Janet), emits Scheme, evals — replacing the Janet cross-compile of the
  prelude. Bootstrap fixpoint: Chez-jolt compiles `clojure.core` AND the analyzer
  from source with no Janet in the loop; verify stage2==stage3 emitted forms.
  Drop the tree-walking interpreter. Continue core-* leaf migration (jolt-uqi/
  ded/tzo). Gate: Chez-jolt builds itself from source, full corpus parity holds,
  zero Janet invoked.

**Phase 4 — Deployment & optimization modes** (the "optimize specific cases" lever)
- Wire `JOLT_WHOLE_PROGRAM`/direct-link to emit specialized Scheme (fl*/fx*),
  feeding Chez `compile-whole-program`. `jolt build`: WPO + strip-fasl +
  AOT-under-petite + heap tuning -> small fast binary (jolt-0w9u reframed).
  Rebuild fibers/async on call/cc + threads. Gate: full bench suite incl.
  collections/binary-trees (the GC axes); size + memory measured vs spike
  baseline.

**Phase 5 — Delete the Janet host** (single substrate)
- Chez self-hosts + parity + perf confirmed -> delete both `src/jolt/*.janet`
  (the seed) and `host/chez/*.janet` (the Janet emitter/driver, now superseded by
  the Clojure `jolt.backend-scheme`). Chez is the only substrate; no alternate
  host retained. jolt-core unchanged. Oracle stays the spec corpus + JVM Clojure.
  Net: one host, no Janet, no cross-compile.

## Host interop & the examples acceptance corpus

The `../examples` repo holds real jolt apps with real git deps + C interop; they
are the **end-to-end acceptance gate** complementing the unit-level spec corpus.
The port must account for jolt's interop surface, which is layered:

- **`janet.*` bridge** — the general Clojure->Janet escape hatch (`janet/get`,
  `janet/struct`, `janet.ev/sleep`, `janet.net/close`, `janet.spork.http/*`),
  used mostly in demo glue. DECIDED (2026-06-17): rename the bridge to a
  **neutral `host.*`** namespace (not `janet.*`/`chez.*`), so app interop code is
  host-portable; each host implements `host.*` over its own FFI (Janet FFI today,
  Chez `foreign-procedure` tomorrow). The legacy `janet.*` aliases stay as
  deprecated shims during migration.
- **FFI-backed shim libraries** — the genuine C interop. `jolt-lang/http-client`
  implements `java.net` + TLS + gzip as host shims **over Janet FFI**, backing
  clj-http-lite's `:clj` branch. On Chez these are reimplemented over Chez FFI
  (libcurl/openssl/zlib or native sockets); the `java.*`-facing API is unchanged.
  Also in scope: `jolt-lang/db`, `jolt-lang/logging`, and `jolt-lang/router`
  (mirrors `reitit.Trie`; verify whether it carries native code).
- **`:jpm/module` Janet-native deps** — `spork/http` (server), `ring-janet-
  adapter`. No Chez equivalent: provide a Scheme/Chez-FFI HTTP server or treat
  these as test-only fixtures.
- **Native-library dependencies in deps.edn** (DECIDED 2026-06-17). C-interop
  shims need shared libraries (libcurl, openssl/libssl, zlib) that CANNOT be
  pulled from git like Clojure libs. Add a `:native` dep form so a project can
  *declare* what it needs, e.g.
  `{:native/lib "curl" :native/min-version "7.0" :native/header "curl/curl.h"}`.
  The resolver doesn't fetch them, but: (a) it surfaces the requirement to the
  user (and can suggest the install command per platform), and (b) the
  loader/`foreign-procedure` layer probes for the `.so`/`.dylib` at load and, if
  absent, raises a precise error — "missing native library: libcurl (declared by
  jolt-lang/http-client); install with `brew install curl`" — instead of a raw
  dlopen failure. Symmetric with `:jpm/module`'s "verify importable, hint to
  install" pattern.
- **Host-neutral pieces that port for free** — Java-class mirror shims (pure
  Clojure) and `JOLT_FEATURES` reader conditionals.

Sequencing: the interop bridge mechanism is part of the Phase-1 shim, but the
FFI-backed libraries land in **Phase 4** (downstream of core parity), validated
by running the examples end-to-end. Tracked as a dedicated epic child.

## Beads reconciliation

- **Closed (obsolete — Janet bytecode-VM / cgen / Janet-dispatch mechanisms Chez
  replaces wholesale):** jolt-ffn (epic, already concluded flat), jolt-5vsp.1,
  jolt-qx70, jolt-l1l4 (cgen), jolt-cm7t, jolt-fw2 (Janet dispatch substrate),
  jolt-pria (Janet ctx cold-build startup).
- **Recommend close (confirm) — Janet constant-factor passes Chez's JIT/GC/WPO
  subsume:** jolt-826, jolt-27w, jolt-t6r, jolt-8flj, jolt-3ko, jolt-t34,
  jolt-u1f. (jolt-ffn's own STATUS says the gap is "generic-runtime overhead…
  the JVM erases via JIT + inline caching + unboxing" — exactly Chez's job.)
- **Reframed under the Chez epic:** jolt-0w9u + .1 -> Phase 4 deploy mode +
  closed-world audit; jolt-1r86 -> Phase 0b/4 bench validation; jolt-lcn /
  jolt-uqi / jolt-tzo / jolt-ded / jolt-brh / jolt-7dl -> Phase 2/3 self-hosting,
  now targeting Chez; jolt-5vsp (foundational-runtime epic) -> parent, the Chez
  port is its realization.
- **Keep host-agnostic:** deps beads (jolt-x4o, jolt-xkd, jolt-pnje, jolt-vley);
  correctness bug jolt-jk23.