jolt/doc/self-hosting-compiler.md

Toward a self-hosting Jolt compiler

Research and design notes for evolving Jolt from "interpreter + opt-in ad-hoc compiler" toward a self-hosting Clojure-in-Clojure compiler that emits Janet bytecode, keeps full REPL live-redefinition, and rests on a minimal Janet bootstrap. This is a design doc, not a changelog — it describes where we are, the prior art, the constraints we verified, and a recommended path.

The goal

• Self-hosting, Clojure-in-Clojure. A small kernel in the host (Janet) is enough to start; the rest of Clojure — including the compiler — is written in Clojure and compiled by Jolt itself, growing the language as it compiles more of itself.
• Janet bytecode out. Compiled code runs as native Janet bytecode (fast), not tree-walking.
• Full runtime flexibility. def/defn redefinition, vars, protocols, multimethods, and everything else stay live and redefinable at the REPL even for compiled code.
• Minimal host requirement. Shrink what must exist in Janet to the irreducible base.

Where Jolt is today

• ~5,500 lines of Janet implement clojure.core (core.janet) and a tree-walking interpreter (evaluator.janet); ~1k lines of Clojure are the stdlib (clojure.string/set/walk/…, jolt.*). So the language is mostly in the host, inverted from the Clojure-in-Clojure ideal.
• The interpreter (eval-form) is the complete reference path.
• The compiler (compiler.janet) — analyze-form (reader form → :op AST) → emit (AST → Janet form) → Janet compile/eval — is now on by default in the shipped runtime (JOLT_INTERPRET=1 opts out). It is a hybrid: forms it can't compile correctly throw jolt/uncompilable and fall back to the interpreter (loader/eval-toplevel), so results always match the interpreter. Validated at parity — conformance 218/218 under both interpret and compile, and the clojure-test-suite under compile passes 3932 (vs the 3913 interpreter baseline) across ~4.6k assertions.
• Done so far: var-indirection (globals deref through var cells, so compiled code is REPL-redefinable); hybrid fallback; compilation of multi-arity / named / variadic fns and recur inside fn; map and vector literal compilation (mode-correct via make-vec / build-map-literal); resolution that mirrors the interpreter (current ns → clojure.core → Janet-env fallback); and AOT (aot.janet) that marshals a compiled namespace to a Janet bytecode image against the baked-in runtime dictionary and loads it back.
• Still open — the actual self-hosting: the compiler and most of clojure.core are still Janet. Rewriting them in Clojure (compiled by Jolt) is the remaining Clojure-in-Clojure work.

What the host gives us (verified)

Janet already is the backend and the AOT story — we don't need a custom bytecode emitter:

• (compile form env source) → a function (compiled bytecode). Jolt's job is Clojure form → correct Janet form → compile.
• marshal/unmarshal, make-image/load-image → serialize a compiled environment to a bytecode image and load it back: this is Phase 4 AOT.
• asm/disasm → bytecode assembler/disassembler if we ever want to bypass the form layer (we shouldn't need to).

The catch we verified: Janet early-binds top-level references. Compile (defn caller [] foo), then redefine foo — the compiled caller still returns the old value. So emitting Jolt globals as plain Janet symbols (what the current compiler largely does) is fundamentally incompatible with REPL redefinition. This is the single most important design constraint below.

Prior art

• Clojure (JVM). A Java runtime + compiler bootstraps clojure.core, which is written in Clojure; thereafter Clojure compiles Clojure to JVM bytecode. Only ~20 special forms are primitive; everything else is macros/functions. Crucially, compiled call sites go through Var objects (a deref), so redefining a var is visible to existing compiled callers — that's how speed and live redefinition coexist. Clojure 1.8 added opt-in direct linking (inline the call, drop the var indirection) for speed where you don't need redefinition (used for core in production). AOT compiles namespaces to .class files.
• ClojureScript self-hosting. Two stages: an analyzer (source → AST plus a "compiler state" map of namespaces/defs/macros) and a compiler (AST → JS). cljs.js exposes compile/eval at runtime; bootstrapped CLJS compiles CLJS at ~2× the JVM compiler. The host VM (JS engine) is the backend — the same shape we want with Janet as the backend.
• Nanopass (Chez Scheme). A compiler as many small passes over formally specified intermediate languages, with autogenerated boilerplate to recur through unchanged forms and checks that each pass's output matches its grammar. The lesson for "grow the language as it compiles itself": keep passes small and IRs explicit so adding a form is local and verifiable.
• Guile. A Lisp on a bytecode VM: source → Tree-IL (high-level IR) → CPS (optimization IR) → VM bytecode, with several front-end languages targeting Tree-IL. The closest analog to "Lisp → bytecode on a VM."

Assessment: is the current approach the right one?

The overall shape is right and matches ClojureScript: front-end (analyze → emit) with the host VM as the backend, emitting host forms that the host compiles to bytecode. Two things need to change to reach the goal:

1. Late binding for globals. Compile a reference to a Jolt var as a deref through the var cell, not as a Janet symbol. Jolt vars are already cells ({:jolt/type :jolt/var :root …}); a compiled global call becomes roughly ((var-root cell) args…) instead of (janet-symbol args…). Redefinition updates the cell's root, so compiled callers see it — exactly Clojure's model. One indirection per global call; locals and control flow stay direct and fast. Offer opt-in direct linking for hot/AOT code that doesn't need redefinition.
2. Move the compiler and core into Clojure. Today both are Janet. Self-hosting means the compiler is Clojure compiled by Jolt, and most of clojure.core is Clojure. That's the bulk of the work and where the "language builds itself" payoff lives.

So: keep the emit-to-Janet target (it's correct and gives us bytecode + AOT for free), fix global binding, and progressively self-host.

Recommended architecture

Pipeline (nanopass-lite). Keep the data-driven :op AST and grow it as small, named passes rather than one big walker:

1. read — reader → forms (already have it).
2. macroexpand — fully expand to special forms + calls (the interpreter already expands; share one expander).
3. analyze — forms → AST, resolving locals vs vars and tagging ops.
4. (optional) optimize — constant-fold, direct-link hot calls, etc.
5. emit — AST → Janet form, with globals as var-cell derefs.
6. compile — Janet compile → bytecode; make-image for AOT.

Make each pass total over the IR so an unhandled node is an explicit gap, not a silent miss.

The kernel (minimal Janet bootstrap). The irreducible base that must exist in the host before any Clojure can run: the reader; the value/representation layer (vars, namespaces, symbols, keywords, persistent collections, chars); host interop (the janet.* bridge); fn/if/do/let/quote/def/loop/recur evaluation; and compile/eval. Everything else — the rest of clojure.core, the macros, and the compiler — is Clojure loaded and (eventually) compiled by the kernel. Today the kernel is far larger than this; shrinking it is a long game.

Hybrid interpret/compile (Phase 3, and a bootstrap safety net). When a pass can't yet compile a sub-form, emit a call back into the interpreter (eval-form) for that sub-form instead of erroring. This lets the compiler be incomplete and still correct (hot paths compile, cold/unsupported paths interpret), lets us grow coverage incrementally, and de-risks the self-hosting bootstrap.

Live flexibility. Vars stay first-class cells; compiled code derefs them; def updates the root; protocol/multimethod dispatch stays dynamic. Direct linking is opt-in, never the default, so the REPL is always live.

A staged path

1. Var-indirection in the emitter — done. Global refs compile as var-cell derefs, so a compiled defn is redefinable at the REPL.
2. Hybrid fallback + coverage (jolt-1bj) — done. Forms the compiler can't compile throw jolt/uncompilable and fall back to the interpreter, so compile mode is always correct. Covered: multi-arity/named/variadic fns, recur in fn, map/vector literals, and resolution matching the interpreter. Destructuring compiles via the shared destructure expander: the fn/let/ loop/defn macros desugar to plain-symbol fn*/let*/loop*, so it no longer falls back — and the primitives reject patterns outright, matching Clojure (jolt-f79).
3. Compile-by-default + AOT (jolt-7j9) — done, done out of order. Once the hybrid path was validated at parity, compilation was flipped on by default and AOT images (aot.janet) landed. Done before 3–4 because it's the runtime payoff and only needed the hybrid path to be correct, not self-hosting.
4. Self-host the compiler (jolt-lcn) — open. Rewrite compiler.janet as Clojure (jolt.compiler) that Jolt compiles. Now the compiler is part of the language it compiles.
5. Shrink the kernel / core-in-Clojure (jolt-uqi) — open. Move clojure.core from Janet to Clojure incrementally, each piece compiled by the previous stage — the language building itself — leaving a minimal Janet kernel.

What remains (3 and 4) is the actual Clojure-in-Clojure rewrite: the largest part of the work and where the "language builds itself" payoff lives. The correctness and runtime foundations it needs — redefinable compiled code, an always-correct hybrid path, compile-by-default, and AOT — are now in place.