jolt/chez-janet-removal-handoff.md

Handoff — removing Janet (Chez re-host, Phase 4 → Phase 5)

Branch: spike/chez-bootstrap · LOCAL COMMITS ONLY — NEVER push. This overrides the session-close git-push protocol. Epic: jolt-cf1q (Re-host Jolt on Chez, zero Janet). Last session: finished self-hosting (inc8/inc9) + the portable conformance spec + fixed 4 certified bugs.

Where we are

Done and gated (all local commits, latest 6abbea3):

• Self-hosting proven. The on-Chez compiler reproduces itself (stage2==stage3, prelude pstage3==pstage4). bin/joltc + host/chez/bootstrap.ss build and run jolt with zero Janet in the loop, off the checked-in seed host/chez/seed/.
• Correctness proven. Both Chez corpus gates at 2534/2756, 0 divergences. The corpus is a JVM-certified host-neutral spec (test/conformance/, see SPEC.md): 2670 portable cases, 249 feature-gated.

We are NOT ready to delete Janet. Ripping it out now regresses real capability (below). The remaining work is Phase 4 (runtime features + deployment/perf) → then Phase 5 (jolt-cf1q.6) deletes src/jolt/*.janet + the Janet cross-compile.

Hard rules (every batch)

1. Never git push. Local commits only on this branch.
2. Gate discipline. After any change run, with a real exit code:
   • jpm build && janet run-tests.janet (full Janet gate; must end All tests passed.)
   • JOLT_CHEZ_ZEROJANET_CORPUS=1 janet test/chez/run-corpus-zero-janet.janet (fast, ~2s)
   • JOLT_CHEZ_PRELUDE_CORPUS=1 janet test/chez/run-corpus-prelude.janet (~18min; run ISOLATED — no concurrent chez work)
   • clojure -M test/conformance/certify.clj (JVM cert; fails on new/stale divergence)
3. If clojure.core or any seed source changes, re-mint the Chez seed, or test/chez/bootstrap-test.janet fails (rebuilt ≠ committed seed). Re-mint:

   (import host/chez/driver :as d) (import host/chez/jolt-chez :as jc)
   (def ctx (d/make-ctx))
   (d/mint-chez-seed* (jc/ensure-prelude ctx)
     (d/ensure-compiler-image ctx "/tmp/stage1.ss")
     "host/chez/seed/prelude.ss" "host/chez/seed/image.ss")
   

   (run from repo root via a throwaway test/chez/_x.janet). Raise the corpus floors when parity rises.
4. Use beads (bd ready, bd show <id>, bd update <id> --claim, bd close). bd remember for persistent notes — see memories chez-phase3-progress, conformance-spec. Run bd close as its own command then verify (close-race).

Verify current state (run these first next session)

git log --oneline -8
bin/joltc -e '(+ 1 2)'                 # => 3  (pure-Chez runtime works)
bin/joltc -e '(eval (quote (+ 1 2)))'  # => FAILS today — that's blocker #1

The blockers, in priority order

1. jolt-r8ku — runtime eval / load-string / defmacro on the Chez spine ⭐ DO FIRST

Why first: biggest functional gap, and mostly wiring — the compiler image is already resident at runtime on Chez (host/chez/compile-eval.ss loads it). Turns joltc from a one-shot -e into a real runtime/REPL. Unblocks ~the eval/load-string corpus crashes too.

Symptoms (tested):

joltc -e '(eval (quote (+ 1 2)))'           -> jolt/uncompilable: special form eval
joltc -e '(load-string "(+ 1 2)")'          -> not a fn (jolt-nil)
joltc -e '(defmacro m [x] ...) (m 5)'       -> jolt/uncompilable: special form defmacro

Why they punt: eval/defmacro are in the special-symbol list, so the analyzer treats them as special forms and the :else arm punts:

• host/chez/host-contract.ss:110 and :183 — hc-special-symbols / form-special? list (includes eval, defmacro).
• jolt-core/jolt/analyzer.clj:216 analyze-special (case op …), :279 (uncompilable (str "special form " op)).

Approach:

• eval / load-string are FUNCTIONS, not special forms. Remove eval from the special-symbol lists so it resolves as a normal var, then def-var! them in a Chez .ss (e.g. a new host/chez/eval.ss loaded by compile-eval.ss): eval = (lambda (form) (jolt-compile-eval-form form (current-ns))) — but jolt-compile-eval (compile-eval.ss) takes a SOURCE STRING and reads it; add a sibling that takes an already-read FORM and runs analyze→emit→eval on it. load-string = read-all + eval each form.
  • Mirror the Janet seed semantics: check src/jolt/eval_runtime.janet / eval_base.janet for what eval/load-string do there (the contract to match).
• defmacro must stay special (it defines a macro). The prelude already handles defmacro at build time (host/chez/emit-image.ss → ei-defmacro->fn emits a def of the expander fn + mark-macro!). For RUNTIME user defmacro, add a defmacro branch to analyze-special that lowers to the same shape: a def of the (fn …) expander
  • a runtime mark-macro! call, so the macro is usable in subsequent forms of the same program. Re-analysis of later forms must see the macro flag (hc-macro? reads var-macro-table).
• Watch jolt-lpvi (build-ctx vs eval-ctx isolation) — runtime eval defining vars must land in the user ns, not leak into the compiler image ns.

Verify: joltc -e of all three above; add cases to test/chez/spine-test.janet; the eval/load-string corpus crashes (660/661/2157/2158/… per the old chez-phase2 notes) should now run; re-run both corpus gates.

2. jolt-byjr — concurrency (future / promise / agent / pmap / pvalues / pcalls)

joltc -e '(deref (future (+ 1 2)))' → Unknown class clojure.core. The Janet oracle uses isolated-heap SNAPSHOT futures (the :concurrency/snapshot feature in test/conformance/profile.edn) which neither a sync shim nor raw Chez threads reproduce exactly. Decision needed: real Chez threads vs a sync shim vs snapshot emulation. The corpus snapshot cases are oracle-coupled (JVM gives shared-heap answers, jolt gives snapshot) — already classified as a deliberate delta, so don't chase JVM parity; match the jolt documented semantics. Revisit with a Chez thread-safety review of the RT (host/chez/*.ss mutable tables: var-table, ns-registry, atoms).

3. jolt-cf1q.7 — host/Java interop breadth

~167 runtime crashes in the corpus are host-coupled (profile.edn: :host/jvm-interop 174, :host/arrays 12, :host/janet 16). Java class shims, arrays, bean/proxy/definterface. Apply vendor-before-scratch: for non-trivial interop (date/time, crypto, etc.) vendor a mature Chez lib rather than hand-rolling. Many of these are intentionally non-portable — decide per case whether Chez should implement it or it stays a host-gated feature. janet.* cases are N/A on Chez (delete from corpus in Phase 5).

4. jolt-cf1q.5 — Phase 4: deployment & optimization modes

The optimizing compiler is Janet-only: inference, direct-linking, cgen-to-C, whole-program, and native binary builds (jpm build, the ring-app example). Chez emits Scheme and relies on Chez's own native compiler — perf is unmeasured. The Phase 5 bead literally gates on "perf confirmed." Tasks: benchmark the Chez path vs the Janet/JVM baselines; decide the Chez deployment story (Chez compile-program/ boot file vs script); decide which Janet optimization modes (if any) must be reproduced. This is the largest unknown — scope it before committing to deletion.

5. Move the test oracle off Janet, then Phase 5 (jolt-cf1q.6) delete

• Today build/jolt (Janet) is the oracle for test/chez/_*.janet, and run-corpus-prelude analyzes on Janet. The corpus is now JVM-certified, so the oracle can shift to spec-corpus + JVM. Do this swap before deleting Janet.
• Tooling still Janet: jolt-deps, nrepl-server, uberscript/DCE — port or drop.
• Then Phase 5: delete src/jolt/*.janet (reader, value layer, vars/ns, the tree-walking interpreter evaluator.janet, the Janet backend) AND host/chez/emit.janet + the Janet cross-compile in driver.janet. The tree-walking interpreter (the literal "drop the interpreter" goal) dies here.

Sequencing

jolt-r8ku (eval/macros)  ─┐
jolt-byjr (concurrency)   ├─► jolt-cf1q.7 (host interop) ─► jolt-cf1q.5 (Phase 4 perf/deploy)
                          ─┘                                          │
                                          oracle → JVM/corpus ◄───────┘
                                                   │
                                                   ▼
                                      jolt-cf1q.6 (Phase 5: DELETE Janet)

Key file map

Area	File
Pure-Chez runtime CLI	bin/joltc → host/chez/cli.ss
Zero-Janet compile spine	host/chez/compile-eval.ss (jolt-compile-eval)
Host contract (special syms, resolve, macro?)	host/chez/host-contract.ss
On-Chez image/prelude emitter	host/chez/emit-image.ss
Pure-Chez self-build + seed	host/chez/bootstrap.ss, host/chez/seed/
Portable analyzer / backend	jolt-core/jolt/analyzer.clj, …/backend_scheme.clj
clojure.core overlay	jolt-core/clojure/core/NN-*.clj
Janet seed (to delete in Phase 5)	src/jolt/*.janet (reader, evaluator, host_iface, …)
Chez RT shims	host/chez/*.ss
Conformance spec	test/conformance/ (SPEC.md, certify.clj, profile.edn, known-divergences.edn)
Corpus + gates	test/chez/corpus.edn, run-corpus-{zero-janet,prelude}.janet, extract-corpus.janet
Self-host gates	test/chez/{spine,fixpoint,bootstrap,cli}-test.janet