docs: consolidate doc/ into docs/

One documentation root: the prose docs (building-and-deps, self-hosting
architecture/compiler, tools-deps, grammar.ebnf) join the spec and RFCs under
docs/. References in README and deps-conformance-test updated.

docs/self-hosting-compiler.md

@@ -0,0 +1,172 @@
+# Toward a self-hosting Jolt compiler
+
+Research and design notes for evolving Jolt from "interpreter + opt-in ad-hoc
+compiler" toward a self-hosting Clojure-in-Clojure compiler that emits Janet
+bytecode, keeps full REPL live-redefinition, and rests on a minimal Janet
+bootstrap. This is a design doc, not a changelog — it describes where we are, the
+prior art, the constraints we verified, and a recommended path.
+
+## The goal
+
+- **Self-hosting, Clojure-in-Clojure.** A small kernel in the host (Janet) is
+  enough to start; the rest of Clojure — including the compiler — is written in
+  Clojure and compiled by Jolt itself, growing the language as it compiles more
+  of itself.
+- **Janet bytecode out.** Compiled code runs as native Janet bytecode (fast),
+  not tree-walking.
+- **Full runtime flexibility.** `def`/`defn` redefinition, vars, protocols,
+  multimethods, and everything else stay live and redefinable at the REPL even
+  for compiled code.
+- **Minimal host requirement.** Shrink what must exist in Janet to the
+  irreducible base.
+
+## Where Jolt is today
+
+- ~5,500 lines of **Janet** implement `clojure.core` (`core.janet`) and a
+  tree-walking interpreter (`evaluator.janet`); ~1k lines of **Clojure** are the
+  stdlib (`clojure.string/set/walk/…`, `jolt.*`). So the language is mostly in
+  the host, inverted from the Clojure-in-Clojure ideal.
+- The interpreter (`eval-form`) is the complete reference path.
+- The compiler (`compiler.janet`) — `analyze-form` (reader form → `:op` AST) →
+  `emit` (AST → Janet form) → Janet `compile`/`eval` — is now **on by default**
+  in the shipped runtime (`JOLT_INTERPRET=1` opts out). It is a *hybrid*: forms
+  it can't compile correctly throw `jolt/uncompilable` and fall back to the
+  interpreter (`loader/eval-toplevel`), so results always match the interpreter.
+  Validated at parity — conformance 218/218 under both interpret and compile, and
+  the clojure-test-suite under compile passes 3932 (vs the 3913 interpreter
+  baseline) across ~4.6k assertions.
+- Done so far: var-indirection (globals deref through var cells, so compiled code
+  is REPL-redefinable); hybrid fallback; compilation of multi-arity / named /
+  variadic fns and `recur` inside `fn`; map and vector literal compilation
+  (mode-correct via `make-vec` / `build-map-literal`); resolution that mirrors
+  the interpreter (current ns → `clojure.core` → Janet-env fallback); and AOT
+  (`aot.janet`) that marshals a compiled namespace to a Janet bytecode image
+  against the baked-in runtime dictionary and loads it back.
+- Still open — the actual self-hosting: the compiler and most of `clojure.core`
+  are still Janet. Rewriting them in Clojure (compiled by Jolt) is the remaining
+  Clojure-in-Clojure work.
+
+## What the host gives us (verified)
+
+Janet already is the backend and the AOT story — we don't need a custom bytecode
+emitter:
+
+- `(compile form env source)` → a **function** (compiled bytecode). Jolt's job is
+  Clojure form → correct Janet form → `compile`.
+- `marshal`/`unmarshal`, `make-image`/`load-image` → serialize a compiled
+  environment to a **bytecode image** and load it back: this is Phase 4 AOT.
+- `asm`/`disasm` → bytecode assembler/disassembler if we ever want to bypass the
+  form layer (we shouldn't need to).
+
+**The catch we verified:** Janet *early-binds* top-level references. Compile
+`(defn caller [] foo)`, then redefine `foo` — the compiled `caller` still returns
+the old value. So emitting Jolt globals as plain Janet symbols (what the current
+compiler largely does) is fundamentally incompatible with REPL redefinition. This
+is the single most important design constraint below.
+
+## Prior art
+
+- **Clojure (JVM).** A Java runtime + compiler bootstraps `clojure.core`, which is
+  written in Clojure; thereafter Clojure compiles Clojure to JVM bytecode. Only
+  ~20 special forms are primitive; everything else is macros/functions. Crucially,
+  compiled call sites go **through Var objects** (a deref), so redefining a var is
+  visible to existing compiled callers — that's how speed and live redefinition
+  coexist. Clojure 1.8 added opt-in **direct linking** (inline the call, drop the
+  var indirection) for speed where you don't need redefinition (used for core in
+  production). AOT compiles namespaces to `.class` files.
+- **ClojureScript self-hosting.** Two stages: an **analyzer** (source → AST plus
+  a "compiler state" map of namespaces/defs/macros) and a **compiler** (AST → JS).
+  `cljs.js` exposes compile/eval at runtime; bootstrapped CLJS compiles CLJS at
+  ~2× the JVM compiler. The host VM (JS engine) is the backend — the same shape we
+  want with Janet as the backend.
+- **Nanopass (Chez Scheme).** A compiler as *many small passes* over *formally
+  specified* intermediate languages, with autogenerated boilerplate to recur
+  through unchanged forms and checks that each pass's output matches its grammar.
+  The lesson for "grow the language as it compiles itself": keep passes small and
+  IRs explicit so adding a form is local and verifiable.
+- **Guile.** A Lisp on a bytecode VM: source → Tree-IL (high-level IR) → CPS
+  (optimization IR) → VM bytecode, with several front-end languages targeting
+  Tree-IL. The closest analog to "Lisp → bytecode on a VM."
+
+## Assessment: is the current approach the right one?
+
+The overall *shape* is right and matches ClojureScript: front-end (analyze →
+emit) with the host VM as the backend, emitting host forms that the host compiles
+to bytecode. Two things need to change to reach the goal:
+
+1. **Late binding for globals.** Compile a reference to a Jolt var as a **deref
+   through the var cell**, not as a Janet symbol. Jolt vars are already cells
+   (`{:jolt/type :jolt/var :root …}`); a compiled global call becomes roughly
+   `((var-root cell) args…)` instead of `(janet-symbol args…)`. Redefinition
+   updates the cell's root, so compiled callers see it — exactly Clojure's model.
+   One indirection per global call; locals and control flow stay direct and fast.
+   Offer opt-in **direct linking** for hot/AOT code that doesn't need redefinition.
+2. **Move the compiler and core into Clojure.** Today both are Janet. Self-hosting
+   means the compiler is Clojure compiled by Jolt, and most of `clojure.core` is
+   Clojure. That's the bulk of the work and where the "language builds itself"
+   payoff lives.
+
+So: keep the emit-to-Janet target (it's correct and gives us bytecode + AOT for
+free), fix global binding, and progressively self-host.
+
+## Recommended architecture
+
+**Pipeline (nanopass-lite).** Keep the data-driven `:op` AST and grow it as small,
+named passes rather than one big walker:
+
+1. *read* — reader → forms (already have it).
+2. *macroexpand* — fully expand to special forms + calls (the interpreter already
+   expands; share one expander).
+3. *analyze* — forms → AST, resolving locals vs vars and tagging ops.
+4. *(optional) optimize* — constant-fold, direct-link hot calls, etc.
+5. *emit* — AST → Janet form, with globals as var-cell derefs.
+6. *compile* — Janet `compile` → bytecode; `make-image` for AOT.
+
+Make each pass total over the IR so an unhandled node is an explicit gap, not a
+silent miss.
+
+**The kernel (minimal Janet bootstrap).** The irreducible base that must exist in
+the host before any Clojure can run: the reader; the value/representation layer
+(vars, namespaces, symbols, keywords, persistent collections, chars); host
+interop (the `janet.*` bridge); `fn`/`if`/`do`/`let`/`quote`/`def`/`loop`/`recur`
+evaluation; and `compile`/`eval`. Everything else — the rest of `clojure.core`,
+the macros, and the compiler — is Clojure loaded and (eventually) compiled by the
+kernel. Today the kernel is far larger than this; shrinking it is a long game.
+
+**Hybrid interpret/compile (Phase 3, and a bootstrap safety net).** When a pass
+can't yet compile a sub-form, emit a call back into the interpreter (`eval-form`)
+for that sub-form instead of erroring. This lets the compiler be incomplete and
+still correct (hot paths compile, cold/unsupported paths interpret), lets us grow
+coverage incrementally, and de-risks the self-hosting bootstrap.
+
+**Live flexibility.** Vars stay first-class cells; compiled code derefs them;
+`def` updates the root; protocol/multimethod dispatch stays dynamic. Direct
+linking is opt-in, never the default, so the REPL is always live.
+
+## A staged path
+
+1. **Var-indirection in the emitter** — *done*. Global refs compile as var-cell
+   derefs, so a compiled `defn` is redefinable at the REPL.
+2. **Hybrid fallback + coverage** (`jolt-1bj`) — *done*. Forms the compiler can't
+   compile throw `jolt/uncompilable` and fall back to the interpreter, so compile
+   mode is always correct. Covered: multi-arity/named/variadic fns, `recur` in
+   `fn`, map/vector literals, and resolution matching the interpreter.
+   Destructuring compiles via the shared `destructure` expander: the `fn`/`let`/
+   `loop`/`defn` macros desugar to plain-symbol `fn*`/`let*`/`loop*`, so it no
+   longer falls back — and the primitives reject patterns outright, matching
+   Clojure (`jolt-f79`).
+5. **Compile-by-default + AOT** (`jolt-7j9`) — *done, done out of order*. Once the
+   hybrid path was validated at parity, compilation was flipped on by default and
+   AOT images (`aot.janet`) landed. Done before 3–4 because it's the runtime
+   payoff and only needed the hybrid path to be correct, not self-hosting.
+3. **Self-host the compiler** (`jolt-lcn`) — *open*. Rewrite `compiler.janet` as
+   Clojure (`jolt.compiler`) that Jolt compiles. Now the compiler is part of the
+   language it compiles.
+4. **Shrink the kernel / core-in-Clojure** (`jolt-uqi`) — *open*. Move
+   `clojure.core` from Janet to Clojure incrementally, each piece compiled by the
+   previous stage — the language building itself — leaving a minimal Janet kernel.
+
+What remains (3 and 4) is the actual Clojure-in-Clojure rewrite: the largest part
+of the work and where the "language builds itself" payoff lives. The correctness
+and runtime foundations it needs — redefinable compiled code, an always-correct
+hybrid path, compile-by-default, and AOT — are now in place.