# Re-hosting Jolt on Chez Scheme — phased plan

Decision (2026-06-17): port jolt's runtime substrate from Janet to Chez Scheme
(cisco/ChezScheme). The spike (`spike/chez/RESULTS.md`) validated the thesis:
the compute-substrate ceiling is ~12-47x faster than Janet (mandelbrot
166->13.4ms matching jolt's own C-codegen result; fib 246->5.2ms), Chez's
native compiler reaches that at runtime with the REPL intact, size stays
single-digit MB, and the one regression (memory baseline ~2.5x) is opt-in
tunable via WPO + stripped boot + AOT-under-petite.

This plan is built around two north stars beyond raw speed:

1. **Minimal host shim.** Every line that stays in Scheme is a line we failed to
   self-host. The port is the forcing function for jolt-tzo/uqi/lcn: shrink the
   Scheme seed to the irreducible primitive set, push everything else into
   portable Clojure (`jolt-core/`).
2. **Tests are the contract.** The spec/conformance corpus is host-neutral data
   (`[name expected-clj actual-clj]` triples compared via jolt's own `=`). It is
   the acceptance gate for "the port is correct" — Chez-jolt must pass the same
   corpus the Janet host passes, with no regression to the clojure-test-suite
   baseline.

## The minimal host shim (target end-state)

What MUST be Scheme (the irreducible primitive layer the self-hosted core rests
on) vs what MOVES into portable Clojure:

### Stays in Scheme (the shim)
- **Value primitives that can't bottom out in Clojure without circularity:**
  the `nil` sentinel (distinct from `#f` and `'()` — the classic Lisp-on-Lisp
  trap), keyword/symbol records (Clojure symbols carry ns + meta), char/string
  bridging. NOTE: Chez's **numeric tower is a windfall** — int/float/ratio/
  bignum are native, so jolt gets exact ratios + bignums for free (Janet lacked
  them).
- **Mutable cell / box + array primitives** the self-hosted RT builds on (var
  roots, transients, HAMT node arrays).
- **A type-tag / record primitive** for deftype/protocol dispatch (Chez records).
- **`host/compile`** — eval a backend-emitted Scheme form to a procedure. On Chez
  this is literally `eval`/`compile`. Trivial. This is the whole backend
  host-dependency.
- **FFI / interop bridge** (foreign-procedure) for host interop calls.
- **Persistent-collection hot nodes** — ONLY IF the Clojure-on-Chez version
  (Phase 0c) doesn't hold perf. Open question, decided by measurement.

### Moves into portable Clojure (`jolt-core/`)
- **The reader** (text -> forms). CLJS self-hosts its reader; ours can too. ~33KB
  of Janet leaves the host. Not hot.
- **Analyzer + IR + passes** — already portable Clojure. No change.
- **The backend emitter** — its LOGIC becomes Clojure that emits Scheme forms as
  data; only `host/compile` crosses the seam.
- **macros + clojure.core** — finish the jolt-uqi/tzo migration (most already
  Clojure).
- **Protocol/multimethod dispatch logic** — over the host tag primitive.
- **Persistent collections** — candidate (Phase 0c), perf-permitting.

### Dropped entirely
- **The tree-walking interpreter** (`eval_base/eval_runtime/eval_special/
  eval_resolve`, ~140KB Janet). On Chez, native `compile` is always present and
  cheap, so the compile-only path can cover every form — no `jolt/uncompilable`
  fallback needed. The interpreter's role as the correctness *oracle* transfers
  to the spec corpus + JVM Clojure (the real reference), which is strictly
  better. This is the single largest shim reduction and the biggest open risk;
  Phase 1 validates that compile-only is total before we commit to the drop.

## Test contract strategy

- **Corpus is the contract.** The 44 `test/spec/*.janet` files, the conformance
  cases, `clojure-test-suite`, and `clojure-stdlib-suite` are host-neutral: pure
  Clojure source + expected values. Extract the triples into a runner that can
  target an arbitrary `jolt` binary (subprocess at the Clojure boundary).
- **Parity gate.** Chez-jolt must pass the same corpus as Janet-jolt; the
  clojure-test-suite baseline is the bar (raise it when it rises, never lower).
- **Oracle shift.** Today's 3-mode conformance (interpret/compile/self-host)
  loses the "interpret" leg when the interpreter is dropped; the golden expected
  values + JVM Clojure become the oracle. Keep the frozen expected values.
- **Dual-run during migration.** Run BOTH hosts against the corpus until Chez
  reaches parity, then retire the Janet host.

## Phases (-> beads epic)

**Phase 0 — Foundations & contract harness** (de-risk; no jolt pipeline yet)
- 0a. Chez RT value model: nil sentinel, keyword/symbol records, numeric-tower
  mapping, `=`/hashing. Resolve the nil/`'()`/`#f` representation up front.
- 0b. Host-neutral test-contract runner: extract the spec/conformance corpus to
  drive an arbitrary jolt binary; stand up the parity-gate machinery.
- 0c. Persistent-collection perf experiment: HAMT/PV in Clojure-on-Chez vs
  Scheme-native — the data that decides what stays shim vs self-hosted.

**Phase 1 — Minimal Chez kernel + real-pipeline bootstrap**
- Scheme shim (value layer, var/ns cells, `host/compile`, cenv impl).
- `jolt.backend` Scheme-emit target for the IR the analyzer already produces.
- Bootstrap jolt-core (ir/analyzer) on Chez; compile + run `(+ 1 2)` -> fib ->
  mandelbrot through the REAL pipeline. Gate: compute benches run end-to-end and
  hit ~the spike ceiling; confirm compile-only is total (no fallback needed).

**Phase 2 — clojure.core to spec parity**
- Bring up persistent collections (per 0c) + seq/coll/print/refs/io tiers over
  the Chez RT. Gate: spec + conformance + clojure-test-suite parity with the
  Janet baseline.

**Phase 3 — Self-host expansion (shrink the shim)**
- Move the reader into jolt-core. Continue core-* leaf migration (jolt-uqi/ded/
  tzo, now targeting Chez). Drop the tree-walking interpreter. Gate: shim equals
  the documented minimal set; parity holds.

**Phase 4 — Deployment & optimization modes** (the "optimize specific cases" lever)
- Wire `JOLT_WHOLE_PROGRAM`/direct-link to emit specialized Scheme (fl*/fx*),
  feeding Chez `compile-whole-program`. `jolt build`: WPO + strip-fasl +
  AOT-under-petite + heap tuning -> small fast binary (jolt-0w9u reframed).
  Rebuild fibers/async on call/cc + threads. Gate: full bench suite incl.
  collections/binary-trees (the GC axes); size + memory measured vs spike
  baseline.

**Phase 5 — Retire the Janet host**
- Chez parity + perf confirmed -> remove `src/jolt/*.janet` seed. jolt-core
  unchanged. Janet becomes a historical/alternate host proving portability.

## Host interop & the examples acceptance corpus

The `../examples` repo holds real jolt apps with real git deps + C interop; they
are the **end-to-end acceptance gate** complementing the unit-level spec corpus.
The port must account for jolt's interop surface, which is layered:

- **`janet.*` bridge** — the general Clojure->Janet escape hatch (`janet/get`,
  `janet/struct`, `janet.ev/sleep`, `janet.net/close`, `janet.spork.http/*`),
  used mostly in demo glue. DECIDED (2026-06-17): rename the bridge to a
  **neutral `host.*`** namespace (not `janet.*`/`chez.*`), so app interop code is
  host-portable; each host implements `host.*` over its own FFI (Janet FFI today,
  Chez `foreign-procedure` tomorrow). The legacy `janet.*` aliases stay as
  deprecated shims during migration.
- **FFI-backed shim libraries** — the genuine C interop. `jolt-lang/http-client`
  implements `java.net` + TLS + gzip as host shims **over Janet FFI**, backing
  clj-http-lite's `:clj` branch. On Chez these are reimplemented over Chez FFI
  (libcurl/openssl/zlib or native sockets); the `java.*`-facing API is unchanged.
  Also in scope: `jolt-lang/db`, `jolt-lang/logging`, and `jolt-lang/router`
  (mirrors `reitit.Trie`; verify whether it carries native code).
- **`:jpm/module` Janet-native deps** — `spork/http` (server), `ring-janet-
  adapter`. No Chez equivalent: provide a Scheme/Chez-FFI HTTP server or treat
  these as test-only fixtures.
- **Native-library dependencies in deps.edn** (DECIDED 2026-06-17). C-interop
  shims need shared libraries (libcurl, openssl/libssl, zlib) that CANNOT be
  pulled from git like Clojure libs. Add a `:native` dep form so a project can
  *declare* what it needs, e.g.
  `{:native/lib "curl" :native/min-version "7.0" :native/header "curl/curl.h"}`.
  The resolver doesn't fetch them, but: (a) it surfaces the requirement to the
  user (and can suggest the install command per platform), and (b) the
  loader/`foreign-procedure` layer probes for the `.so`/`.dylib` at load and, if
  absent, raises a precise error — "missing native library: libcurl (declared by
  jolt-lang/http-client); install with `brew install curl`" — instead of a raw
  dlopen failure. Symmetric with `:jpm/module`'s "verify importable, hint to
  install" pattern.
- **Host-neutral pieces that port for free** — Java-class mirror shims (pure
  Clojure) and `JOLT_FEATURES` reader conditionals.

Sequencing: the interop bridge mechanism is part of the Phase-1 shim, but the
FFI-backed libraries land in **Phase 4** (downstream of core parity), validated
by running the examples end-to-end. Tracked as a dedicated epic child.

## Beads reconciliation

- **Closed (obsolete — Janet bytecode-VM / cgen / Janet-dispatch mechanisms Chez
  replaces wholesale):** jolt-ffn (epic, already concluded flat), jolt-5vsp.1,
  jolt-qx70, jolt-l1l4 (cgen), jolt-cm7t, jolt-fw2 (Janet dispatch substrate),
  jolt-pria (Janet ctx cold-build startup).
- **Recommend close (confirm) — Janet constant-factor passes Chez's JIT/GC/WPO
  subsume:** jolt-826, jolt-27w, jolt-t6r, jolt-8flj, jolt-3ko, jolt-t34,
  jolt-u1f. (jolt-ffn's own STATUS says the gap is "generic-runtime overhead…
  the JVM erases via JIT + inline caching + unboxing" — exactly Chez's job.)
- **Reframed under the Chez epic:** jolt-0w9u + .1 -> Phase 4 deploy mode +
  closed-world audit; jolt-1r86 -> Phase 0b/4 bench validation; jolt-lcn /
  jolt-uqi / jolt-tzo / jolt-ded / jolt-brh / jolt-7dl -> Phase 2/3 self-hosting,
  now targeting Chez; jolt-5vsp (foundational-runtime epic) -> parent, the Chez
  port is its realization.
- **Keep host-agnostic:** deps beads (jolt-x4o, jolt-xkd, jolt-pnje, jolt-vley);
  correctness bug jolt-jk23.