Research notes: self-hosting compiler design

Survey (Clojure/JVM var indirection + direct linking, ClojureScript self-host,
nanopass, Guile) and a recommended path to a self-hosting Clojure-in-Clojure
compiler emitting Janet bytecode while keeping REPL redefinition. Key finding:
Janet early-binds top-level refs, so compiled globals must deref through var
cells to stay redefinable. Recommends var-indirection first, then hybrid
fallback, then self-hosting the compiler and shrinking the Janet kernel.

doc/self-hosting-compiler.md

@@ -0,0 +1,152 @@
+# Toward a self-hosting Jolt compiler
+
+Research and design notes for evolving Jolt from "interpreter + opt-in ad-hoc
+compiler" toward a self-hosting Clojure-in-Clojure compiler that emits Janet
+bytecode, keeps full REPL live-redefinition, and rests on a minimal Janet
+bootstrap. This is a design doc, not a changelog — it describes where we are, the
+prior art, the constraints we verified, and a recommended path.
+
+## The goal
+
+- **Self-hosting, Clojure-in-Clojure.** A small kernel in the host (Janet) is
+  enough to start; the rest of Clojure — including the compiler — is written in
+  Clojure and compiled by Jolt itself, growing the language as it compiles more
+  of itself.
+- **Janet bytecode out.** Compiled code runs as native Janet bytecode (fast),
+  not tree-walking.
+- **Full runtime flexibility.** `def`/`defn` redefinition, vars, protocols,
+  multimethods, and everything else stay live and redefinable at the REPL even
+  for compiled code.
+- **Minimal host requirement.** Shrink what must exist in Janet to the
+  irreducible base.
+
+## Where Jolt is today
+
+- ~5,500 lines of **Janet** implement `clojure.core` (`core.janet`) and a
+  tree-walking interpreter (`evaluator.janet`); ~1k lines of **Clojure** are the
+  stdlib (`clojure.string/set/walk/…`, `jolt.*`). So the language is mostly in
+  the host, inverted from the Clojure-in-Clojure ideal.
+- The interpreter (`eval-form`) is the live, complete path.
+- There's an opt-in compiler (`compiler.janet`): `analyze-form` (reader form →
+  AST tagged with `:op`) → `emit` (AST → Janet form) → Janet `compile`/`eval`.
+  Phases 1–2 are done (per-context env so defs persist and resolve; native
+  arithmetic ops + direct calls — recursive `fib(30)` ≈ 0.08 s). Phase 3
+  (destructuring, multi-arity, hybrid fallback) is open.
+
+## What the host gives us (verified)
+
+Janet already is the backend and the AOT story — we don't need a custom bytecode
+emitter:
+
+- `(compile form env source)` → a **function** (compiled bytecode). Jolt's job is
+  Clojure form → correct Janet form → `compile`.
+- `marshal`/`unmarshal`, `make-image`/`load-image` → serialize a compiled
+  environment to a **bytecode image** and load it back: this is Phase 4 AOT.
+- `asm`/`disasm` → bytecode assembler/disassembler if we ever want to bypass the
+  form layer (we shouldn't need to).
+
+**The catch we verified:** Janet *early-binds* top-level references. Compile
+`(defn caller [] foo)`, then redefine `foo` — the compiled `caller` still returns
+the old value. So emitting Jolt globals as plain Janet symbols (what the current
+compiler largely does) is fundamentally incompatible with REPL redefinition. This
+is the single most important design constraint below.
+
+## Prior art
+
+- **Clojure (JVM).** A Java runtime + compiler bootstraps `clojure.core`, which is
+  written in Clojure; thereafter Clojure compiles Clojure to JVM bytecode. Only
+  ~20 special forms are primitive; everything else is macros/functions. Crucially,
+  compiled call sites go **through Var objects** (a deref), so redefining a var is
+  visible to existing compiled callers — that's how speed and live redefinition
+  coexist. Clojure 1.8 added opt-in **direct linking** (inline the call, drop the
+  var indirection) for speed where you don't need redefinition (used for core in
+  production). AOT compiles namespaces to `.class` files.
+- **ClojureScript self-hosting.** Two stages: an **analyzer** (source → AST plus
+  a "compiler state" map of namespaces/defs/macros) and a **compiler** (AST → JS).
+  `cljs.js` exposes compile/eval at runtime; bootstrapped CLJS compiles CLJS at
+  ~2× the JVM compiler. The host VM (JS engine) is the backend — the same shape we
+  want with Janet as the backend.
+- **Nanopass (Chez Scheme).** A compiler as *many small passes* over *formally
+  specified* intermediate languages, with autogenerated boilerplate to recur
+  through unchanged forms and checks that each pass's output matches its grammar.
+  The lesson for "grow the language as it compiles itself": keep passes small and
+  IRs explicit so adding a form is local and verifiable.
+- **Guile.** A Lisp on a bytecode VM: source → Tree-IL (high-level IR) → CPS
+  (optimization IR) → VM bytecode, with several front-end languages targeting
+  Tree-IL. The closest analog to "Lisp → bytecode on a VM."
+
+## Assessment: is the current approach the right one?
+
+The overall *shape* is right and matches ClojureScript: front-end (analyze →
+emit) with the host VM as the backend, emitting host forms that the host compiles
+to bytecode. Two things need to change to reach the goal:
+
+1. **Late binding for globals.** Compile a reference to a Jolt var as a **deref
+   through the var cell**, not as a Janet symbol. Jolt vars are already cells
+   (`{:jolt/type :jolt/var :root …}`); a compiled global call becomes roughly
+   `((var-root cell) args…)` instead of `(janet-symbol args…)`. Redefinition
+   updates the cell's root, so compiled callers see it — exactly Clojure's model.
+   One indirection per global call; locals and control flow stay direct and fast.
+   Offer opt-in **direct linking** for hot/AOT code that doesn't need redefinition.
+2. **Move the compiler and core into Clojure.** Today both are Janet. Self-hosting
+   means the compiler is Clojure compiled by Jolt, and most of `clojure.core` is
+   Clojure. That's the bulk of the work and where the "language builds itself"
+   payoff lives.
+
+So: keep the emit-to-Janet target (it's correct and gives us bytecode + AOT for
+free), fix global binding, and progressively self-host.
+
+## Recommended architecture
+
+**Pipeline (nanopass-lite).** Keep the data-driven `:op` AST and grow it as small,
+named passes rather than one big walker:
+
+1. *read* — reader → forms (already have it).
+2. *macroexpand* — fully expand to special forms + calls (the interpreter already
+   expands; share one expander).
+3. *analyze* — forms → AST, resolving locals vs vars and tagging ops.
+4. *(optional) optimize* — constant-fold, direct-link hot calls, etc.
+5. *emit* — AST → Janet form, with globals as var-cell derefs.
+6. *compile* — Janet `compile` → bytecode; `make-image` for AOT.
+
+Make each pass total over the IR so an unhandled node is an explicit gap, not a
+silent miss.
+
+**The kernel (minimal Janet bootstrap).** The irreducible base that must exist in
+the host before any Clojure can run: the reader; the value/representation layer
+(vars, namespaces, symbols, keywords, persistent collections, chars); host
+interop (the `janet.*` bridge); `fn`/`if`/`do`/`let`/`quote`/`def`/`loop`/`recur`
+evaluation; and `compile`/`eval`. Everything else — the rest of `clojure.core`,
+the macros, and the compiler — is Clojure loaded and (eventually) compiled by the
+kernel. Today the kernel is far larger than this; shrinking it is a long game.
+
+**Hybrid interpret/compile (Phase 3, and a bootstrap safety net).** When a pass
+can't yet compile a sub-form, emit a call back into the interpreter (`eval-form`)
+for that sub-form instead of erroring. This lets the compiler be incomplete and
+still correct (hot paths compile, cold/unsupported paths interpret), lets us grow
+coverage incrementally, and de-risks the self-hosting bootstrap.
+
+**Live flexibility.** Vars stay first-class cells; compiled code derefs them;
+`def` updates the root; protocol/multimethod dispatch stays dynamic. Direct
+linking is opt-in, never the default, so the REPL is always live.
+
+## A staged path (maps onto the existing beads)
+
+1. **Var-indirection in the emitter** *(new, foundational — do before more
+   compiler work)*. Compile global refs as var-cell derefs; verify a compiled
+   `defn` is redefinable at the REPL. Without this, more compiler coverage just
+   bakes in more early-binding to undo later.
+2. **Hybrid fallback + finish coverage** (`jolt-1bj`, Phase 3): per-form fallback
+   to `eval-form`; then compile destructuring, multi-arity/variadic, and the
+   remaining forms as optimizations on top of the always-correct fallback.
+3. **Self-host the compiler.** Rewrite `compiler.janet` as Clojure (`jolt.compiler`)
+   that Jolt compiles. Now the compiler is part of the language it compiles.
+4. **Shrink the kernel / core-in-Clojure.** Move `clojure.core` from Janet to
+   Clojure incrementally, leaving only the minimal kernel in Janet. Each moved
+   piece is compiled by the previous stage — the language building itself.
+5. **Compile-by-default + AOT** (`jolt-7j9`, Phase 4): once the hybrid path is
+   robust, flip compilation on by default; ship AOT images via `make-image`.
+
+The ordering matters: var-indirection first (correctness for redefinition), then
+the hybrid fallback (correctness for coverage), then self-hosting and kernel
+shrinking (the Clojure-in-Clojure payoff), then default-on + AOT.