diff --git a/doc/self-hosting-compiler.md b/doc/self-hosting-compiler.md new file mode 100644 index 0000000..72db447 --- /dev/null +++ b/doc/self-hosting-compiler.md @@ -0,0 +1,152 @@ +# Toward a self-hosting Jolt compiler + +Research and design notes for evolving Jolt from "interpreter + opt-in ad-hoc +compiler" toward a self-hosting Clojure-in-Clojure compiler that emits Janet +bytecode, keeps full REPL live-redefinition, and rests on a minimal Janet +bootstrap. This is a design doc, not a changelog — it describes where we are, the +prior art, the constraints we verified, and a recommended path. + +## The goal + +- **Self-hosting, Clojure-in-Clojure.** A small kernel in the host (Janet) is + enough to start; the rest of Clojure — including the compiler — is written in + Clojure and compiled by Jolt itself, growing the language as it compiles more + of itself. +- **Janet bytecode out.** Compiled code runs as native Janet bytecode (fast), + not tree-walking. +- **Full runtime flexibility.** `def`/`defn` redefinition, vars, protocols, + multimethods, and everything else stay live and redefinable at the REPL even + for compiled code. +- **Minimal host requirement.** Shrink what must exist in Janet to the + irreducible base. + +## Where Jolt is today + +- ~5,500 lines of **Janet** implement `clojure.core` (`core.janet`) and a + tree-walking interpreter (`evaluator.janet`); ~1k lines of **Clojure** are the + stdlib (`clojure.string/set/walk/…`, `jolt.*`). So the language is mostly in + the host, inverted from the Clojure-in-Clojure ideal. +- The interpreter (`eval-form`) is the live, complete path. +- There's an opt-in compiler (`compiler.janet`): `analyze-form` (reader form → + AST tagged with `:op`) → `emit` (AST → Janet form) → Janet `compile`/`eval`. + Phases 1–2 are done (per-context env so defs persist and resolve; native + arithmetic ops + direct calls — recursive `fib(30)` ≈ 0.08 s). Phase 3 + (destructuring, multi-arity, hybrid fallback) is open. + +## What the host gives us (verified) + +Janet already is the backend and the AOT story — we don't need a custom bytecode +emitter: + +- `(compile form env source)` → a **function** (compiled bytecode). Jolt's job is + Clojure form → correct Janet form → `compile`. +- `marshal`/`unmarshal`, `make-image`/`load-image` → serialize a compiled + environment to a **bytecode image** and load it back: this is Phase 4 AOT. +- `asm`/`disasm` → bytecode assembler/disassembler if we ever want to bypass the + form layer (we shouldn't need to). + +**The catch we verified:** Janet *early-binds* top-level references. Compile +`(defn caller [] foo)`, then redefine `foo` — the compiled `caller` still returns +the old value. So emitting Jolt globals as plain Janet symbols (what the current +compiler largely does) is fundamentally incompatible with REPL redefinition. This +is the single most important design constraint below. + +## Prior art + +- **Clojure (JVM).** A Java runtime + compiler bootstraps `clojure.core`, which is + written in Clojure; thereafter Clojure compiles Clojure to JVM bytecode. Only + ~20 special forms are primitive; everything else is macros/functions. Crucially, + compiled call sites go **through Var objects** (a deref), so redefining a var is + visible to existing compiled callers — that's how speed and live redefinition + coexist. Clojure 1.8 added opt-in **direct linking** (inline the call, drop the + var indirection) for speed where you don't need redefinition (used for core in + production). AOT compiles namespaces to `.class` files. +- **ClojureScript self-hosting.** Two stages: an **analyzer** (source → AST plus + a "compiler state" map of namespaces/defs/macros) and a **compiler** (AST → JS). + `cljs.js` exposes compile/eval at runtime; bootstrapped CLJS compiles CLJS at + ~2× the JVM compiler. The host VM (JS engine) is the backend — the same shape we + want with Janet as the backend. +- **Nanopass (Chez Scheme).** A compiler as *many small passes* over *formally + specified* intermediate languages, with autogenerated boilerplate to recur + through unchanged forms and checks that each pass's output matches its grammar. + The lesson for "grow the language as it compiles itself": keep passes small and + IRs explicit so adding a form is local and verifiable. +- **Guile.** A Lisp on a bytecode VM: source → Tree-IL (high-level IR) → CPS + (optimization IR) → VM bytecode, with several front-end languages targeting + Tree-IL. The closest analog to "Lisp → bytecode on a VM." + +## Assessment: is the current approach the right one? + +The overall *shape* is right and matches ClojureScript: front-end (analyze → +emit) with the host VM as the backend, emitting host forms that the host compiles +to bytecode. Two things need to change to reach the goal: + +1. **Late binding for globals.** Compile a reference to a Jolt var as a **deref + through the var cell**, not as a Janet symbol. Jolt vars are already cells + (`{:jolt/type :jolt/var :root …}`); a compiled global call becomes roughly + `((var-root cell) args…)` instead of `(janet-symbol args…)`. Redefinition + updates the cell's root, so compiled callers see it — exactly Clojure's model. + One indirection per global call; locals and control flow stay direct and fast. + Offer opt-in **direct linking** for hot/AOT code that doesn't need redefinition. +2. **Move the compiler and core into Clojure.** Today both are Janet. Self-hosting + means the compiler is Clojure compiled by Jolt, and most of `clojure.core` is + Clojure. That's the bulk of the work and where the "language builds itself" + payoff lives. + +So: keep the emit-to-Janet target (it's correct and gives us bytecode + AOT for +free), fix global binding, and progressively self-host. + +## Recommended architecture + +**Pipeline (nanopass-lite).** Keep the data-driven `:op` AST and grow it as small, +named passes rather than one big walker: + +1. *read* — reader → forms (already have it). +2. *macroexpand* — fully expand to special forms + calls (the interpreter already + expands; share one expander). +3. *analyze* — forms → AST, resolving locals vs vars and tagging ops. +4. *(optional) optimize* — constant-fold, direct-link hot calls, etc. +5. *emit* — AST → Janet form, with globals as var-cell derefs. +6. *compile* — Janet `compile` → bytecode; `make-image` for AOT. + +Make each pass total over the IR so an unhandled node is an explicit gap, not a +silent miss. + +**The kernel (minimal Janet bootstrap).** The irreducible base that must exist in +the host before any Clojure can run: the reader; the value/representation layer +(vars, namespaces, symbols, keywords, persistent collections, chars); host +interop (the `janet.*` bridge); `fn`/`if`/`do`/`let`/`quote`/`def`/`loop`/`recur` +evaluation; and `compile`/`eval`. Everything else — the rest of `clojure.core`, +the macros, and the compiler — is Clojure loaded and (eventually) compiled by the +kernel. Today the kernel is far larger than this; shrinking it is a long game. + +**Hybrid interpret/compile (Phase 3, and a bootstrap safety net).** When a pass +can't yet compile a sub-form, emit a call back into the interpreter (`eval-form`) +for that sub-form instead of erroring. This lets the compiler be incomplete and +still correct (hot paths compile, cold/unsupported paths interpret), lets us grow +coverage incrementally, and de-risks the self-hosting bootstrap. + +**Live flexibility.** Vars stay first-class cells; compiled code derefs them; +`def` updates the root; protocol/multimethod dispatch stays dynamic. Direct +linking is opt-in, never the default, so the REPL is always live. + +## A staged path (maps onto the existing beads) + +1. **Var-indirection in the emitter** *(new, foundational — do before more + compiler work)*. Compile global refs as var-cell derefs; verify a compiled + `defn` is redefinable at the REPL. Without this, more compiler coverage just + bakes in more early-binding to undo later. +2. **Hybrid fallback + finish coverage** (`jolt-1bj`, Phase 3): per-form fallback + to `eval-form`; then compile destructuring, multi-arity/variadic, and the + remaining forms as optimizations on top of the always-correct fallback. +3. **Self-host the compiler.** Rewrite `compiler.janet` as Clojure (`jolt.compiler`) + that Jolt compiles. Now the compiler is part of the language it compiles. +4. **Shrink the kernel / core-in-Clojure.** Move `clojure.core` from Janet to + Clojure incrementally, leaving only the minimal kernel in Janet. Each moved + piece is compiled by the previous stage — the language building itself. +5. **Compile-by-default + AOT** (`jolt-7j9`, Phase 4): once the hybrid path is + robust, flip compilation on by default; ship AOT images via `make-image`. + +The ordering matters: var-indirection first (correctness for redefinition), then +the hybrid fallback (correctness for coverage), then self-hosting and kernel +shrinking (the Clojure-in-Clojure payoff), then default-on + AOT.