d3194aae59
adds self-hosted compiler is functionally: - The default compile path is the portable pipeline using jolt.analyzer (Clojure) → host-neutral IR → backend.janet. - The analyzer is itself Clojure, compiled by jolt for true self-hosting. - bootstrap-fixpoint passes (stage1 == stage2 == stage3): rebuilding the compiler on its own output. - clojure.core is now self-hosted in the overlay. - Stateful forms (defmacro/ns/deftype/defmulti/require/in-ns) are interpreted by design.
223 lines
12 KiB
Markdown
223 lines
12 KiB
Markdown
# Jolt — Self-Hosted Clojure on Janet · Handoff
|
||
|
||
Onboarding for a fresh agent picking up this work. Read this, then
|
||
`bd prime` and `bd memories` for the live issue/knowledge state.
|
||
|
||
---
|
||
|
||
## 1. What this project is
|
||
|
||
**Jolt** is a Clojure implementation written in [Janet](https://janet-lang.org).
|
||
It has two execution paths and a **self-hosting compiler**:
|
||
|
||
- **Interpreter** — `src/jolt/evaluator.janet`. A tree-walking evaluator over
|
||
reader forms. Always correct; the fallback for anything the compiler can't yet
|
||
handle. The *live path* for stateful/context-modifying forms.
|
||
- **Self-hosted compiler** — the portable front end lives in **Clojure** under
|
||
`jolt-core/jolt/` (`analyzer.clj` reader-form → host-neutral IR `ir.clj`), and a
|
||
**Janet back end** (`src/jolt/backend.janet`) emits Janet from that IR. This is
|
||
the default compile path. It is *self-hosted*: the compiler that compiles
|
||
clojure.core is itself (mostly) Clojure compiled by jolt.
|
||
- **Bootstrap compiler** — `src/jolt/compiler.janet`. A Janet-native compiler used
|
||
**only** to bootstrap-compile the kernel tier before the self-hosted analyzer
|
||
exists. Not the main path.
|
||
- **Hybrid fallback** — the analyzer throws `:jolt/uncompilable` on forms it can't
|
||
handle; the loader catches that and interprets instead. Three "uncompilable"
|
||
lists are kept in sync (see compile-pipeline notes in the code).
|
||
|
||
Entry point: `src/jolt/api.janet` — `(init opts)` builds a context, installs the
|
||
host contract, and loads clojure.core (seed + overlay). `:compile? true` enables
|
||
the self-hosted pipeline; off = interpret.
|
||
|
||
---
|
||
|
||
## 2. The architecture that matters: seed + overlay
|
||
|
||
clojure.core is split into a shrinking **Janet seed** and a growing **Clojure
|
||
overlay**. This split *is* the project's main arc.
|
||
|
||
### The Janet seed — `src/jolt/core.janet` (~3200 lines, ~365 `core-*` fns)
|
||
The irreducible base: the `core-renames` primitives the compiler emits directly
|
||
(`first`/`nth`/`conj`/`get`/…) plus genuinely host-coupled fns (atoms, vars,
|
||
transients, arrays, futures, meta, print, the persistent-collection kernel). Each
|
||
fn is `core-<name>` and interned into the `clojure.core` namespace via the
|
||
`core-bindings` table near the bottom of the file.
|
||
|
||
### The Clojure overlay — `jolt-core/clojure/core/NN-*.clj` (loaded in order)
|
||
Plain Clojure expressing the *rest* of clojure.core on top of the seed. Tiers:
|
||
|
||
| Tier | Role |
|
||
|---|---|
|
||
| `00-syntax.clj` | control macros (`when`/`cond`/`and`/`or`/`let`/`loop`/`fn`/`for`/…), `destructure`, `when-let`. Interpreted, loaded **first** so macros exist before any code compiles. |
|
||
| `00-kernel.clj` | structural fns the analyzer itself needs (`second`/`peek`/`subvec`/`mapv`/`update`). **Bootstrap-compiled** into clojure.core before the analyzer is built. |
|
||
| `10-seq.clj` | seq-tier fns |
|
||
| `20-coll.clj` | pure collection/misc fns + the Phase-4 host-primitive wrappers |
|
||
| `30-macros.clj` | the remaining user-facing macros |
|
||
| `40-lazy.clj` | lazy seq transformers (Phase 5) |
|
||
|
||
Loader: `api.janet` → `load-core-overlay!` / `core-tiers`. Sources are read **fresh
|
||
from disk** at startup when running from the repo (`stdlib_embed.janet` collects
|
||
`jolt-core/` and `src/jolt/clojure/`), so editing a `.clj` tier takes effect with
|
||
no rebuild. (A `jpm build` bakes them into the image; that can go stale — tests
|
||
run from source.)
|
||
|
||
### The host contract — `src/jolt/host_iface.janet` (ns `jolt.host`)
|
||
The portability seam. jolt-core (analyzer/IR/overlay) calls **only** `jolt.host`
|
||
fns, never Janet directly. Originally compiler-facing (`form-sym?`, `form-list?`,
|
||
`resolve-global`, …). Phase 4 added the first runtime primitive: **`ref-put!`**
|
||
(set/remove a key on a mutable reference cell) — the minimal mutation kernel the
|
||
overlay uses for atom watches/validators, volatiles, and `aset`. The overlay calls
|
||
these qualified, e.g. `(jolt.host/ref-put! ...)`.
|
||
|
||
---
|
||
|
||
## 3. The migration epic (`jolt-1j0`) — essentially COMPLETE
|
||
|
||
**Goal:** shrink the Janet seed to `core-renames` + genuinely host-coupled fns;
|
||
express everything else (pure fns, macros, lazy machinery) in the self-hosted
|
||
overlay. Started at core.janet = 4145 lines / 421 `core-*` fns.
|
||
|
||
**Phases (all done):**
|
||
- **Phase 1** — compiler-dependency kernel tier. (Was found already essentially
|
||
complete — the analyzer needs nothing beyond the kernel tier + atom/swap!/reset!.)
|
||
- **Phase 2** — ~193 movable pure-eager fns → overlay.
|
||
- **Phase 3** (`jolt-461`, closed) — ~46 core macros → `defmacro` in the overlay.
|
||
Last one was `when-let`.
|
||
- **Phase 4** (`jolt-ldf`, closed) — host-coupled fns. ~27 moved over the `ref-put!`
|
||
primitive + pure composition (vary-meta, reduce-kv, ex-info accessors,
|
||
tagged-value predicates, atom peripheral ops, volatiles, future predicates,
|
||
ns-name, array reads/aset). The rest stay native by design (atom/swap!/reset!/
|
||
deref, transients, var cells, meta tables, namespace, constructors, proxy,
|
||
print dispatch).
|
||
- **Phase 5** (`jolt-c09`, closed) — true laziness. Lazy seq generators +
|
||
transformers, the `40-lazy.clj` tier, realization-boundary discipline. See
|
||
`phase-5.md` for the full implementation + testing plan and what landed
|
||
(representation decision = **Option B / hybrid**: lazy over lazy input, eager
|
||
representation-preserving over concrete finite collections).
|
||
|
||
The epic issue (`jolt-1j0`) may still read IN_PROGRESS — verify with `bd show
|
||
jolt-1j0` and close it if all five phase issues are closed and gates are green.
|
||
|
||
**Where to confirm current state:** `phase-5.md` (detailed, step-annotated),
|
||
`jolt-core/clojure/core/MIGRATION.md` (the worklist + bucket classification), and
|
||
the bd memories `phase4-host-primitive-pattern` / `phase4-movable-classification`.
|
||
|
||
---
|
||
|
||
## 4. Representation facts you MUST know (the trap floor)
|
||
|
||
Jolt's value/form representations bite every time. The essentials:
|
||
|
||
- **Reader forms:** a *call/list* `(f x)` is a Janet **array**; a *vector literal*
|
||
`[a b]` is a Janet **tuple**; a *map literal* `{..}` is a Janet **struct** (or a
|
||
**phm** when a key/val is nil or a key is a collection). A **symbol** is a struct
|
||
`{:jolt/type :symbol :ns _ :name _}`; a **keyword** is a Janet keyword.
|
||
- **Runtime values:** vectors are persistent-vectors (`pvec`, tagged tables) or
|
||
tuples; lists/seq-results are Janet arrays or `plist`; sets are `phs`; maps are
|
||
struct-or-phm. `vector?` is true for tuple **and** pvec. `seq?` is arrays/plists/
|
||
lazy-seqs (not vectors). In `JOLT_MUTABLE` builds vectors are plain arrays — so
|
||
`vector?`/`array?` collapse (this is why `ifn?` couldn't move — see `jolt-1vx`).
|
||
- **Tagged values** carry their kind in `:jolt/type` (atoms, volatiles, delays,
|
||
futures, ex-info, reader-conditional, lazy-seq) or `:jolt/deftype` (records). The
|
||
overlay can **read** these via `(get x :jolt/type)` / `(get x :field)` — `get`
|
||
returns nil on non-tables, no error. It **cannot construct** them without a host
|
||
primitive. This is the Phase-4 movability rule: accessors/predicates move,
|
||
constructors stay.
|
||
- **`canon-key`** (core.janet ~line 51) is the canonical-hashing kernel of the
|
||
whole persistent-collection system — woven into `get`/`count`/`contains?`. This
|
||
is why transients are irreducibly host.
|
||
- **LazySeq** (`phm.janet`): `@{:jolt/type :jolt/lazy-seq :fn thunk ...}`; thunk →
|
||
`nil` or `[first rest-thunk]`; `realize-ls` memoizes with a `:jolt/pending` guard
|
||
that makes self-referential seqs (`lazy-cat` fib) work.
|
||
|
||
### Macro/overlay-authoring gotchas (learned the hard way)
|
||
- Build binding/forms via syntax-quote templates `` `[~@xs] `` (a tuple form), not
|
||
`conj`/`list` (those make pvecs/plists the analyzer/compiler rejects).
|
||
- A fresh symbol inside a macro body: `(symbol (str (gensym)))` — a bare `(gensym)`
|
||
returns a *Janet* symbol the destructurer rejects.
|
||
- A `.clj` tier is **Clojure** (`;;` comments). A `.janet` test/spec is **Janet**
|
||
(`#` comments — `;` is splice!). Mixing them is a frequent self-inflicted error.
|
||
- In a tier, a fn must be defined *after* the macro it uses is defined; use `def` +
|
||
`fn*` if you need it before `defn` exists (as `destructure` does in 00-syntax).
|
||
|
||
---
|
||
|
||
## 5. Build, run, and the test gate
|
||
|
||
No special build needed to run from source — Janet reads the tiers off disk.
|
||
|
||
```bash
|
||
# Smoke
|
||
janet -e '(use ./src/jolt/api) (pp (eval-string (init) "(+ 1 2)"))'
|
||
|
||
# THE GATE — run all of these green before committing any core change:
|
||
janet test/integration/conformance-test.janet # 229 cases × 3 modes (interpret/compile/self-host)
|
||
janet test/integration/bootstrap-fixpoint-test.janet # stage1 == stage2 == stage3
|
||
janet test/integration/self-host-test.janet
|
||
janet test/integration/sci-bootstrap-test.janet # loads vendored SCI through jolt
|
||
janet test/integration/clojure-test-suite-test.janet # battery; baseline-pass=3971, clean-files=45
|
||
for f in test/spec/*.janet test/unit/*.janet; do janet "$f"; done # all must exit 0
|
||
```
|
||
|
||
- **Specs** (`test/spec/*-spec.janet`) — data-driven `defspec` tables, behavioral.
|
||
- **Conformance** — real-Clojure-semantics assertions, run in all 3 execution modes.
|
||
- **clojure-test-suite** — runs `lread/clojure-test-suite` (from `~/src/clojure-
|
||
test-suite`) via a per-file **subprocess under a 6 s deadline** (infinite seqs are
|
||
CPU-bound and uninterruptible in-process — never probe them inline). Skips if the
|
||
suite dir is absent. Raise `baseline-pass` as jolt improves; never lower it.
|
||
- **Laziness** must be tested via the deadlined subprocess harness, not in-process.
|
||
|
||
### Per-change workflow (mirror this)
|
||
1. Make a small, single-purpose change.
|
||
2. Add/extend spec + (for subtle behavior) 3-mode conformance cases.
|
||
3. Run the full gate. Commit only if green.
|
||
4. `git push` (the project's session-close protocol requires pushed work).
|
||
|
||
---
|
||
|
||
## 6. Conventions
|
||
|
||
- **Issue tracker: beads (`bd`)**, not TodoWrite/markdown. `bd ready`, `bd show
|
||
<id>`, `bd create`, `bd update <id> --status=…`, `bd close`. `bd remember
|
||
--key … "…"` for durable knowledge; `bd memories` to recall. The `.beads/`
|
||
dir is git-ignored and auto-synced — don't `git add` it.
|
||
- **Commits/PRs**: terse, factual, human-dev tone. No marketing words, no emoji, no
|
||
"This commit…". Say what changed and why it matters.
|
||
- **Branch**: work happens on `compiler-research` (main is `main`).
|
||
- Don't lower `baseline-pass`. If a moved fn surfaces a latent bug, fix it to match
|
||
Clojure and add a regression test rather than preserving the bug (this happened
|
||
with `reduce-kv` on vectors and `ifn?` on lists).
|
||
|
||
---
|
||
|
||
## 7. Where to pick up
|
||
|
||
The migration epic is functionally complete; the seed is at its intended floor
|
||
(core-renames + genuinely host-coupled). Candidate next work:
|
||
|
||
- **Close out `jolt-1j0`** if not already closed (verify all phase issues closed,
|
||
gates green).
|
||
- **`jolt-1vx`** (filed) — `ifn?` is wrongly true for lists; move to overlay but
|
||
it's representation-mode-sensitive (`JOLT_MUTABLE`). Needs both-mode verification.
|
||
- **Phase-5 loose ends** (see `phase-5.md`): a few transformers were kept eager or
|
||
reverted due to compile-mode `~@`/defrecord splice issues (`partition-by`,
|
||
`dedupe`, `tree-seq`, lazy `mapcat`). Re-verify the ~9 previously-timing-out suite
|
||
files actually stopped timing out. The Step 4 "apply/`~@` over lazy" fix would
|
||
unblock the reverted lazy `mapcat`.
|
||
- **Bigger lifts not attempted** (deliberately): the `print-method`/`pr-str`
|
||
dispatch machinery and the `deftype`/`defrecord`/`defprotocol`/multimethod surface
|
||
— both substantial and host-entangled.
|
||
- **Open issues**: `bd ready` for the current actionable list (CI, edn/walk/zip
|
||
stdlib, `into #{}` bug, recur-into-variadic hang, real futures via ev/thread,
|
||
etc. — these predate the migration).
|
||
|
||
### Map of the territory
|
||
- `src/jolt/core.janet` — the Janet seed (`core-*` fns, `core-bindings`, `core-renames`).
|
||
- `src/jolt/evaluator.janet` — interpreter. `src/jolt/compiler.janet` — bootstrap compiler.
|
||
- `src/jolt/backend.janet` — IR → Janet emitter. `src/jolt/host_iface.janet` — `jolt.host`.
|
||
- `src/jolt/phm.janet` — persistent maps/sets/vectors + LazySeq.
|
||
- `src/jolt/api.janet` — context init + tier loading. `src/jolt/reader.janet` — reader.
|
||
- `jolt-core/jolt/{analyzer,ir}.clj` — portable self-hosted front end.
|
||
- `jolt-core/clojure/core/*.clj` — the overlay tiers + `MIGRATION.md`.
|
||
- `phase-5.md` — the laziness plan, annotated with what landed.
|
||
- `CLAUDE.md` / `AGENTS.md` — project agent instructions (beads, session-close).
|