jolt/doc/self-hosting-compiler.md

# Toward a self-hosting Jolt compiler

Research and design notes for evolving Jolt from "interpreter + opt-in ad-hoc
compiler" toward a self-hosting Clojure-in-Clojure compiler that emits Janet
bytecode, keeps full REPL live-redefinition, and rests on a minimal Janet
bootstrap. This is a design doc, not a changelog — it describes where we are, the
prior art, the constraints we verified, and a recommended path.

## The goal

- **Self-hosting, Clojure-in-Clojure.** A small kernel in the host (Janet) is
  enough to start; the rest of Clojure — including the compiler — is written in
  Clojure and compiled by Jolt itself, growing the language as it compiles more
  of itself.
- **Janet bytecode out.** Compiled code runs as native Janet bytecode (fast),
  not tree-walking.
- **Full runtime flexibility.** `def`/`defn` redefinition, vars, protocols,
  multimethods, and everything else stay live and redefinable at the REPL even
  for compiled code.
- **Minimal host requirement.** Shrink what must exist in Janet to the
  irreducible base.

## Where Jolt is today

- ~5,500 lines of **Janet** implement `clojure.core` (`core.janet`) and a
  tree-walking interpreter (`evaluator.janet`); ~1k lines of **Clojure** are the
  stdlib (`clojure.string/set/walk/…`, `jolt.*`). So the language is mostly in
  the host, inverted from the Clojure-in-Clojure ideal.
- The interpreter (`eval-form`) is the live, complete path.
- There's an opt-in compiler (`compiler.janet`): `analyze-form` (reader form →
  AST tagged with `:op`) → `emit` (AST → Janet form) → Janet `compile`/`eval`.
  Phases 1–2 are done (per-context env so defs persist and resolve; native
  arithmetic ops + direct calls — recursive `fib(30)` ≈ 0.08 s). Phase 3
  (destructuring, multi-arity, hybrid fallback) is open.

## What the host gives us (verified)

Janet already is the backend and the AOT story — we don't need a custom bytecode
emitter:

- `(compile form env source)` → a **function** (compiled bytecode). Jolt's job is
  Clojure form → correct Janet form → `compile`.
- `marshal`/`unmarshal`, `make-image`/`load-image` → serialize a compiled
  environment to a **bytecode image** and load it back: this is Phase 4 AOT.
- `asm`/`disasm` → bytecode assembler/disassembler if we ever want to bypass the
  form layer (we shouldn't need to).

**The catch we verified:** Janet *early-binds* top-level references. Compile
`(defn caller [] foo)`, then redefine `foo` — the compiled `caller` still returns
the old value. So emitting Jolt globals as plain Janet symbols (what the current
compiler largely does) is fundamentally incompatible with REPL redefinition. This
is the single most important design constraint below.

## Prior art

- **Clojure (JVM).** A Java runtime + compiler bootstraps `clojure.core`, which is
  written in Clojure; thereafter Clojure compiles Clojure to JVM bytecode. Only
  ~20 special forms are primitive; everything else is macros/functions. Crucially,
  compiled call sites go **through Var objects** (a deref), so redefining a var is
  visible to existing compiled callers — that's how speed and live redefinition
  coexist. Clojure 1.8 added opt-in **direct linking** (inline the call, drop the
  var indirection) for speed where you don't need redefinition (used for core in
  production). AOT compiles namespaces to `.class` files.
- **ClojureScript self-hosting.** Two stages: an **analyzer** (source → AST plus
  a "compiler state" map of namespaces/defs/macros) and a **compiler** (AST → JS).
  `cljs.js` exposes compile/eval at runtime; bootstrapped CLJS compiles CLJS at
  ~2× the JVM compiler. The host VM (JS engine) is the backend — the same shape we
  want with Janet as the backend.
- **Nanopass (Chez Scheme).** A compiler as *many small passes* over *formally
  specified* intermediate languages, with autogenerated boilerplate to recur
  through unchanged forms and checks that each pass's output matches its grammar.
  The lesson for "grow the language as it compiles itself": keep passes small and
  IRs explicit so adding a form is local and verifiable.
- **Guile.** A Lisp on a bytecode VM: source → Tree-IL (high-level IR) → CPS
  (optimization IR) → VM bytecode, with several front-end languages targeting
  Tree-IL. The closest analog to "Lisp → bytecode on a VM."

## Assessment: is the current approach the right one?

The overall *shape* is right and matches ClojureScript: front-end (analyze →
emit) with the host VM as the backend, emitting host forms that the host compiles
to bytecode. Two things need to change to reach the goal:

1. **Late binding for globals.** Compile a reference to a Jolt var as a **deref
   through the var cell**, not as a Janet symbol. Jolt vars are already cells
   (`{:jolt/type :jolt/var :root …}`); a compiled global call becomes roughly
   `((var-root cell) args…)` instead of `(janet-symbol args…)`. Redefinition
   updates the cell's root, so compiled callers see it — exactly Clojure's model.
   One indirection per global call; locals and control flow stay direct and fast.
   Offer opt-in **direct linking** for hot/AOT code that doesn't need redefinition.
2. **Move the compiler and core into Clojure.** Today both are Janet. Self-hosting
   means the compiler is Clojure compiled by Jolt, and most of `clojure.core` is
   Clojure. That's the bulk of the work and where the "language builds itself"
   payoff lives.

So: keep the emit-to-Janet target (it's correct and gives us bytecode + AOT for
free), fix global binding, and progressively self-host.

## Recommended architecture

**Pipeline (nanopass-lite).** Keep the data-driven `:op` AST and grow it as small,
named passes rather than one big walker:

1. *read* — reader → forms (already have it).
2. *macroexpand* — fully expand to special forms + calls (the interpreter already
   expands; share one expander).
3. *analyze* — forms → AST, resolving locals vs vars and tagging ops.
4. *(optional) optimize* — constant-fold, direct-link hot calls, etc.
5. *emit* — AST → Janet form, with globals as var-cell derefs.
6. *compile* — Janet `compile` → bytecode; `make-image` for AOT.

Make each pass total over the IR so an unhandled node is an explicit gap, not a
silent miss.

**The kernel (minimal Janet bootstrap).** The irreducible base that must exist in
the host before any Clojure can run: the reader; the value/representation layer
(vars, namespaces, symbols, keywords, persistent collections, chars); host
interop (the `janet.*` bridge); `fn`/`if`/`do`/`let`/`quote`/`def`/`loop`/`recur`
evaluation; and `compile`/`eval`. Everything else — the rest of `clojure.core`,
the macros, and the compiler — is Clojure loaded and (eventually) compiled by the
kernel. Today the kernel is far larger than this; shrinking it is a long game.

**Hybrid interpret/compile (Phase 3, and a bootstrap safety net).** When a pass
can't yet compile a sub-form, emit a call back into the interpreter (`eval-form`)
for that sub-form instead of erroring. This lets the compiler be incomplete and
still correct (hot paths compile, cold/unsupported paths interpret), lets us grow
coverage incrementally, and de-risks the self-hosting bootstrap.

**Live flexibility.** Vars stay first-class cells; compiled code derefs them;
`def` updates the root; protocol/multimethod dispatch stays dynamic. Direct
linking is opt-in, never the default, so the REPL is always live.

## A staged path (maps onto the existing beads)

1. **Var-indirection in the emitter** *(new, foundational — do before more
   compiler work)*. Compile global refs as var-cell derefs; verify a compiled
   `defn` is redefinable at the REPL. Without this, more compiler coverage just
   bakes in more early-binding to undo later.
2. **Hybrid fallback + finish coverage** (`jolt-1bj`, Phase 3): per-form fallback
   to `eval-form`; then compile destructuring, multi-arity/variadic, and the
   remaining forms as optimizations on top of the always-correct fallback.
3. **Self-host the compiler.** Rewrite `compiler.janet` as Clojure (`jolt.compiler`)
   that Jolt compiles. Now the compiler is part of the language it compiles.
4. **Shrink the kernel / core-in-Clojure.** Move `clojure.core` from Janet to
   Clojure incrementally, leaving only the minimal kernel in Janet. Each moved
   piece is compiled by the previous stage — the language building itself.
5. **Compile-by-default + AOT** (`jolt-7j9`, Phase 4): once the hybrid path is
   robust, flip compilation on by default; ship AOT images via `make-image`.

The ordering matters: var-indirection first (correctness for redefinition), then
the hybrid fallback (correctness for coverage), then self-hosting and kernel
shrinking (the Clojure-in-Clojure payoff), then default-on + AOT.