534007641e
* core: compile macro expanders via staged bootstrap (Stage 2 Task 1)
Macros are now compiled, not interpreted, by steady state — matching
Clojure (macros are ordinary compiled fns the Java seed compiles for
clojure.core) and ClojureScript (macros compiled, invoked at compile
time). Neither reference keeps an interpreted-closure fallback.
The early macros (00-syntax) are defined while the self-hosted analyzer
is still being bootstrapped (it builds lazily after the overlay loads),
so macro-compile-hook returns nil and they get an interpreted closure.
The bootstrap compiler.janet can't substitute (it punts on syntax-quote,
which nearly every expander uses).
Fix = staged bootstrap, the same pattern as the compiler fixpoint:
defmacro stashes the expander source on the var (:macro-src) plus a
:macro-uses-env flag; once the overlay + analyzer are fully built,
backend/recompile-macros! (via ensure-macros-compiled! at the end of
load-core-overlay!) compiles each stashed expander through the now-live
analyzer and rebinds the var, marking :macro-compiled. Idempotent;
&env/&form macros keep the interpreted closure (the compiled fn* has no
such params). The interpreter is now a build-time crutch, gone by
steady state.
Rewrote if-not/if-let/if-some/assert from '& [else]' rest-destructuring
(which the analyzer punts on) to a plain rest param + (first rest), so
all 47 overlay macros compile. Analyzer rest-destructuring gap: jolt-f79.
47/47 overlay macros compiled, 0 interpreted; user macros compile
immediately post-init. Gate green: conformance 267x3, fallback-zero
31/5, bootstrap-fixpoint stage1==2==3, self-host, staged-bootstrap,
lazy-infinite 44/44, clojure-test-suite >=4034/67, sci-bootstrap,
features 78/78, all unit+spec, core-bench neutral.
* core: wrap macro expanders in fn (not fn*) so destructured arglists compile
Follow-up to the staged macro-compile change. The macro-recompile pass
wrapped each expander in the raw fn* primitive, which punts on a
destructuring rest param — so if-not/if-let/if-some/assert (using the
canonical '& [else]') couldn't compile and had been rewritten to plain
rest params as a workaround.
Root cause: fn* is the primitive; the fn MACRO is what desugars
destructuring (rest, map, nested) into the body before lowering. Wrapping
expanders in fn instead of fn* compiles any destructured macro arglist
uniformly, so the workaround is unnecessary — reverted those 4 macros to
the canonical '& [else]' forms.
Net: rest-destructuring is fully compiled for all normal code (fn/defn/
let/macro params). Only the hand-written raw fn* primitive still punts
(jolt-f79, downgraded to P4 — falls back to interpreter, still correct).
47/47 overlay macros compiled. Gate green: conformance 267x3,
fallback-zero 31/5, bootstrap-fixpoint stage1==2==3, self-host,
staged-bootstrap, lazy-infinite 44/44, clojure-test-suite >=4034/67,
sci-bootstrap, features, all unit+spec.
* core: fn*/let*/loop* are plain-symbol primitives, matching Clojure (jolt-f79)
jolt-f79 asked to compile destructuring in fn* rest params. Checking
against Clojure inverts the premise: Clojure's fn* REJECTS destructuring
at compile time ('fn params must be Symbols'; let*/loop* 'Bad binding
form, expected symbol'). So the self-hosted analyzer was already correct
— fn*/let*/loop* are plain-symbol primitives; the fn/let/loop/defn
MACROS desugar destructuring. The real defect was the interpreter
leniently destructuring raw fn*, and defn emitting raw fn* to rely on it.
Changes:
- evaluator: fn*/let*/loop* now reject non-symbol binding forms with
Clojure's exact messages (require-symbol-params/plain-sym?), so the
interpreter agrees with the analyzer + Clojure.
- 00-syntax: defn emits the fn MACRO (not raw fn*) so destructuring
params desugar; unnamed, so self-recursion still resolves via the var.
- 00-syntax: completing that exposed a real gap — the overlay destructure
fn didn't handle kwargs (a map pattern bound against a fn's sequential
rest); it had only worked via the interpreter's destructure-bind. Added
the seq->map coercion to the map? branch (sequential: 1 map elt => that
map, else apply hash-map), matching destructure-bind so interpret ==
compile.
Net: fn*/let*/loop* are plain-symbol primitives across interpreter,
analyzer, and Clojure; all real destructuring (fn/defn/let/loop/macro
params, incl kwargs & {:keys}) compiles through the macros with no
interpreter fallback. Regression spec added.
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, lazy-infinite 44/44,
clojure-test-suite >=4034/67, sci-bootstrap, features 78/78, all
unit+spec (destructuring 50/50).
* docs: clarify fn*/let*/loop* take plain symbols only (jolt-f79)
grammar.ebnf: the destructuring note already attributed patterns to the
binding MACROS; make the boundary explicit — the fn*/let*/loop* PRIMITIVES
they desugar to take plain symbols only (a non-symbol binding errors, as
in Clojure).
self-hosting-compiler.md: the 'compile destructuring via a shared
destructure expander instead of falling back' item is done (and its
jolt-7dl ref was stale) — destructuring now compiles through the
fn/let/loop/defn macros' desugaring; the primitives reject patterns.
* core: Stage 2 Task 2 tier 1 — compile syntax-quote + definterface/extend/proxy
First slice of moving stateful forms onto the compile path (jolt-eaa).
- loader stateful-head?: drop syntax-quote (the analyzer's `handled` set
already compiles it; routing it to the interpreter was redundant).
- host_iface special-names: drop definterface/extend/proxy (stub macros
expanding to def/nil — their expansions compile once unpunted).
- letfn stays interpreted: its let* expansion needs letrec semantics
(mutual recursion between the fns), which sequential compiled let* lacks
— a later tier.
Gate green: conformance 267x3, fallback-zero 31/5, bootstrap-fixpoint
stage1==2==3, self-host, staged-bootstrap, clojure-test-suite >=4034/67,
features 78/78, all unit + protocol/multimethod/macro specs.
---------
Co-authored-by: Yogthos <yogthos@gmail.com>
172 lines
10 KiB
Markdown
172 lines
10 KiB
Markdown
# Toward a self-hosting Jolt compiler
|
||
|
||
Research and design notes for evolving Jolt from "interpreter + opt-in ad-hoc
|
||
compiler" toward a self-hosting Clojure-in-Clojure compiler that emits Janet
|
||
bytecode, keeps full REPL live-redefinition, and rests on a minimal Janet
|
||
bootstrap. This is a design doc, not a changelog — it describes where we are, the
|
||
prior art, the constraints we verified, and a recommended path.
|
||
|
||
## The goal
|
||
|
||
- **Self-hosting, Clojure-in-Clojure.** A small kernel in the host (Janet) is
|
||
enough to start; the rest of Clojure — including the compiler — is written in
|
||
Clojure and compiled by Jolt itself, growing the language as it compiles more
|
||
of itself.
|
||
- **Janet bytecode out.** Compiled code runs as native Janet bytecode (fast),
|
||
not tree-walking.
|
||
- **Full runtime flexibility.** `def`/`defn` redefinition, vars, protocols,
|
||
multimethods, and everything else stay live and redefinable at the REPL even
|
||
for compiled code.
|
||
- **Minimal host requirement.** Shrink what must exist in Janet to the
|
||
irreducible base.
|
||
|
||
## Where Jolt is today
|
||
|
||
- ~5,500 lines of **Janet** implement `clojure.core` (`core.janet`) and a
|
||
tree-walking interpreter (`evaluator.janet`); ~1k lines of **Clojure** are the
|
||
stdlib (`clojure.string/set/walk/…`, `jolt.*`). So the language is mostly in
|
||
the host, inverted from the Clojure-in-Clojure ideal.
|
||
- The interpreter (`eval-form`) is the complete reference path.
|
||
- The compiler (`compiler.janet`) — `analyze-form` (reader form → `:op` AST) →
|
||
`emit` (AST → Janet form) → Janet `compile`/`eval` — is now **on by default**
|
||
in the shipped runtime (`JOLT_INTERPRET=1` opts out). It is a *hybrid*: forms
|
||
it can't compile correctly throw `jolt/uncompilable` and fall back to the
|
||
interpreter (`loader/eval-toplevel`), so results always match the interpreter.
|
||
Validated at parity — conformance 218/218 under both interpret and compile, and
|
||
the clojure-test-suite under compile passes 3932 (vs the 3913 interpreter
|
||
baseline) across ~4.6k assertions.
|
||
- Done so far: var-indirection (globals deref through var cells, so compiled code
|
||
is REPL-redefinable); hybrid fallback; compilation of multi-arity / named /
|
||
variadic fns and `recur` inside `fn`; map and vector literal compilation
|
||
(mode-correct via `make-vec` / `build-map-literal`); resolution that mirrors
|
||
the interpreter (current ns → `clojure.core` → Janet-env fallback); and AOT
|
||
(`aot.janet`) that marshals a compiled namespace to a Janet bytecode image
|
||
against the baked-in runtime dictionary and loads it back.
|
||
- Still open — the actual self-hosting: the compiler and most of `clojure.core`
|
||
are still Janet. Rewriting them in Clojure (compiled by Jolt) is the remaining
|
||
Clojure-in-Clojure work.
|
||
|
||
## What the host gives us (verified)
|
||
|
||
Janet already is the backend and the AOT story — we don't need a custom bytecode
|
||
emitter:
|
||
|
||
- `(compile form env source)` → a **function** (compiled bytecode). Jolt's job is
|
||
Clojure form → correct Janet form → `compile`.
|
||
- `marshal`/`unmarshal`, `make-image`/`load-image` → serialize a compiled
|
||
environment to a **bytecode image** and load it back: this is Phase 4 AOT.
|
||
- `asm`/`disasm` → bytecode assembler/disassembler if we ever want to bypass the
|
||
form layer (we shouldn't need to).
|
||
|
||
**The catch we verified:** Janet *early-binds* top-level references. Compile
|
||
`(defn caller [] foo)`, then redefine `foo` — the compiled `caller` still returns
|
||
the old value. So emitting Jolt globals as plain Janet symbols (what the current
|
||
compiler largely does) is fundamentally incompatible with REPL redefinition. This
|
||
is the single most important design constraint below.
|
||
|
||
## Prior art
|
||
|
||
- **Clojure (JVM).** A Java runtime + compiler bootstraps `clojure.core`, which is
|
||
written in Clojure; thereafter Clojure compiles Clojure to JVM bytecode. Only
|
||
~20 special forms are primitive; everything else is macros/functions. Crucially,
|
||
compiled call sites go **through Var objects** (a deref), so redefining a var is
|
||
visible to existing compiled callers — that's how speed and live redefinition
|
||
coexist. Clojure 1.8 added opt-in **direct linking** (inline the call, drop the
|
||
var indirection) for speed where you don't need redefinition (used for core in
|
||
production). AOT compiles namespaces to `.class` files.
|
||
- **ClojureScript self-hosting.** Two stages: an **analyzer** (source → AST plus
|
||
a "compiler state" map of namespaces/defs/macros) and a **compiler** (AST → JS).
|
||
`cljs.js` exposes compile/eval at runtime; bootstrapped CLJS compiles CLJS at
|
||
~2× the JVM compiler. The host VM (JS engine) is the backend — the same shape we
|
||
want with Janet as the backend.
|
||
- **Nanopass (Chez Scheme).** A compiler as *many small passes* over *formally
|
||
specified* intermediate languages, with autogenerated boilerplate to recur
|
||
through unchanged forms and checks that each pass's output matches its grammar.
|
||
The lesson for "grow the language as it compiles itself": keep passes small and
|
||
IRs explicit so adding a form is local and verifiable.
|
||
- **Guile.** A Lisp on a bytecode VM: source → Tree-IL (high-level IR) → CPS
|
||
(optimization IR) → VM bytecode, with several front-end languages targeting
|
||
Tree-IL. The closest analog to "Lisp → bytecode on a VM."
|
||
|
||
## Assessment: is the current approach the right one?
|
||
|
||
The overall *shape* is right and matches ClojureScript: front-end (analyze →
|
||
emit) with the host VM as the backend, emitting host forms that the host compiles
|
||
to bytecode. Two things need to change to reach the goal:
|
||
|
||
1. **Late binding for globals.** Compile a reference to a Jolt var as a **deref
|
||
through the var cell**, not as a Janet symbol. Jolt vars are already cells
|
||
(`{:jolt/type :jolt/var :root …}`); a compiled global call becomes roughly
|
||
`((var-root cell) args…)` instead of `(janet-symbol args…)`. Redefinition
|
||
updates the cell's root, so compiled callers see it — exactly Clojure's model.
|
||
One indirection per global call; locals and control flow stay direct and fast.
|
||
Offer opt-in **direct linking** for hot/AOT code that doesn't need redefinition.
|
||
2. **Move the compiler and core into Clojure.** Today both are Janet. Self-hosting
|
||
means the compiler is Clojure compiled by Jolt, and most of `clojure.core` is
|
||
Clojure. That's the bulk of the work and where the "language builds itself"
|
||
payoff lives.
|
||
|
||
So: keep the emit-to-Janet target (it's correct and gives us bytecode + AOT for
|
||
free), fix global binding, and progressively self-host.
|
||
|
||
## Recommended architecture
|
||
|
||
**Pipeline (nanopass-lite).** Keep the data-driven `:op` AST and grow it as small,
|
||
named passes rather than one big walker:
|
||
|
||
1. *read* — reader → forms (already have it).
|
||
2. *macroexpand* — fully expand to special forms + calls (the interpreter already
|
||
expands; share one expander).
|
||
3. *analyze* — forms → AST, resolving locals vs vars and tagging ops.
|
||
4. *(optional) optimize* — constant-fold, direct-link hot calls, etc.
|
||
5. *emit* — AST → Janet form, with globals as var-cell derefs.
|
||
6. *compile* — Janet `compile` → bytecode; `make-image` for AOT.
|
||
|
||
Make each pass total over the IR so an unhandled node is an explicit gap, not a
|
||
silent miss.
|
||
|
||
**The kernel (minimal Janet bootstrap).** The irreducible base that must exist in
|
||
the host before any Clojure can run: the reader; the value/representation layer
|
||
(vars, namespaces, symbols, keywords, persistent collections, chars); host
|
||
interop (the `janet.*` bridge); `fn`/`if`/`do`/`let`/`quote`/`def`/`loop`/`recur`
|
||
evaluation; and `compile`/`eval`. Everything else — the rest of `clojure.core`,
|
||
the macros, and the compiler — is Clojure loaded and (eventually) compiled by the
|
||
kernel. Today the kernel is far larger than this; shrinking it is a long game.
|
||
|
||
**Hybrid interpret/compile (Phase 3, and a bootstrap safety net).** When a pass
|
||
can't yet compile a sub-form, emit a call back into the interpreter (`eval-form`)
|
||
for that sub-form instead of erroring. This lets the compiler be incomplete and
|
||
still correct (hot paths compile, cold/unsupported paths interpret), lets us grow
|
||
coverage incrementally, and de-risks the self-hosting bootstrap.
|
||
|
||
**Live flexibility.** Vars stay first-class cells; compiled code derefs them;
|
||
`def` updates the root; protocol/multimethod dispatch stays dynamic. Direct
|
||
linking is opt-in, never the default, so the REPL is always live.
|
||
|
||
## A staged path
|
||
|
||
1. **Var-indirection in the emitter** — *done*. Global refs compile as var-cell
|
||
derefs, so a compiled `defn` is redefinable at the REPL.
|
||
2. **Hybrid fallback + coverage** (`jolt-1bj`) — *done*. Forms the compiler can't
|
||
compile throw `jolt/uncompilable` and fall back to the interpreter, so compile
|
||
mode is always correct. Covered: multi-arity/named/variadic fns, `recur` in
|
||
`fn`, map/vector literals, and resolution matching the interpreter.
|
||
Destructuring compiles via the shared `destructure` expander: the `fn`/`let`/
|
||
`loop`/`defn` macros desugar to plain-symbol `fn*`/`let*`/`loop*`, so it no
|
||
longer falls back — and the primitives reject patterns outright, matching
|
||
Clojure (`jolt-f79`).
|
||
5. **Compile-by-default + AOT** (`jolt-7j9`) — *done, done out of order*. Once the
|
||
hybrid path was validated at parity, compilation was flipped on by default and
|
||
AOT images (`aot.janet`) landed. Done before 3–4 because it's the runtime
|
||
payoff and only needed the hybrid path to be correct, not self-hosting.
|
||
3. **Self-host the compiler** (`jolt-lcn`) — *open*. Rewrite `compiler.janet` as
|
||
Clojure (`jolt.compiler`) that Jolt compiles. Now the compiler is part of the
|
||
language it compiles.
|
||
4. **Shrink the kernel / core-in-Clojure** (`jolt-uqi`) — *open*. Move
|
||
`clojure.core` from Janet to Clojure incrementally, each piece compiled by the
|
||
previous stage — the language building itself — leaving a minimal Janet kernel.
|
||
|
||
What remains (3 and 4) is the actual Clojure-in-Clojure rewrite: the largest part
|
||
of the work and where the "language builds itself" payoff lives. The correctness
|
||
and runtime foundations it needs — redefinable compiled code, an always-correct
|
||
hybrid path, compile-by-default, and AOT — are now in place.
|