From 5e7aea70da2c6ef4f0445797974e219c328a4d07 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Mon, 8 Jun 2026 13:12:55 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20HANDOFF.md=20=E2=80=94=20onboarding=20s?= =?UTF-8?q?ummary=20for=20the=20self-hosted=20compiler=20work?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit General orientation for a fresh agent: the two execution paths (interpreter + self-hosted analyzer/IR/backend), the seed/overlay split and how clojure.core loads in tiers, the jolt.host contract, the jolt-1j0 migration arc (phases 1-5, now complete), the representation/macro-authoring traps, the test gate, and where to pick up next. --- HANDOFF.md | 223 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 223 insertions(+) create mode 100644 HANDOFF.md diff --git a/HANDOFF.md b/HANDOFF.md new file mode 100644 index 0000000..1b45044 --- /dev/null +++ b/HANDOFF.md @@ -0,0 +1,223 @@ +# Jolt — Self-Hosted Clojure on Janet · Handoff + +Onboarding for a fresh agent picking up this work. Read this, then +`bd prime` and `bd memories` for the live issue/knowledge state. + +--- + +## 1. What this project is + +**Jolt** is a Clojure implementation written in [Janet](https://janet-lang.org). +It has two execution paths and a **self-hosting compiler**: + +- **Interpreter** — `src/jolt/evaluator.janet`. A tree-walking evaluator over + reader forms. Always correct; the fallback for anything the compiler can't yet + handle. The *live path* for stateful/context-modifying forms. +- **Self-hosted compiler** — the portable front end lives in **Clojure** under + `jolt-core/jolt/` (`analyzer.clj` reader-form → host-neutral IR `ir.clj`), and a + **Janet back end** (`src/jolt/backend.janet`) emits Janet from that IR. This is + the default compile path. It is *self-hosted*: the compiler that compiles + clojure.core is itself (mostly) Clojure compiled by jolt. +- **Bootstrap compiler** — `src/jolt/compiler.janet`. A Janet-native compiler used + **only** to bootstrap-compile the kernel tier before the self-hosted analyzer + exists. Not the main path. +- **Hybrid fallback** — the analyzer throws `:jolt/uncompilable` on forms it can't + handle; the loader catches that and interprets instead. Three "uncompilable" + lists are kept in sync (see compile-pipeline notes in the code). + +Entry point: `src/jolt/api.janet` — `(init opts)` builds a context, installs the +host contract, and loads clojure.core (seed + overlay). `:compile? true` enables +the self-hosted pipeline; off = interpret. + +--- + +## 2. The architecture that matters: seed + overlay + +clojure.core is split into a shrinking **Janet seed** and a growing **Clojure +overlay**. This split *is* the project's main arc. + +### The Janet seed — `src/jolt/core.janet` (~3200 lines, ~365 `core-*` fns) +The irreducible base: the `core-renames` primitives the compiler emits directly +(`first`/`nth`/`conj`/`get`/…) plus genuinely host-coupled fns (atoms, vars, +transients, arrays, futures, meta, print, the persistent-collection kernel). Each +fn is `core-` and interned into the `clojure.core` namespace via the +`core-bindings` table near the bottom of the file. + +### The Clojure overlay — `jolt-core/clojure/core/NN-*.clj` (loaded in order) +Plain Clojure expressing the *rest* of clojure.core on top of the seed. Tiers: + +| Tier | Role | +|---|---| +| `00-syntax.clj` | control macros (`when`/`cond`/`and`/`or`/`let`/`loop`/`fn`/`for`/…), `destructure`, `when-let`. Interpreted, loaded **first** so macros exist before any code compiles. | +| `00-kernel.clj` | structural fns the analyzer itself needs (`second`/`peek`/`subvec`/`mapv`/`update`). **Bootstrap-compiled** into clojure.core before the analyzer is built. | +| `10-seq.clj` | seq-tier fns | +| `20-coll.clj` | pure collection/misc fns + the Phase-4 host-primitive wrappers | +| `30-macros.clj` | the remaining user-facing macros | +| `40-lazy.clj` | lazy seq transformers (Phase 5) | + +Loader: `api.janet` → `load-core-overlay!` / `core-tiers`. Sources are read **fresh +from disk** at startup when running from the repo (`stdlib_embed.janet` collects +`jolt-core/` and `src/jolt/clojure/`), so editing a `.clj` tier takes effect with +no rebuild. (A `jpm build` bakes them into the image; that can go stale — tests +run from source.) + +### The host contract — `src/jolt/host_iface.janet` (ns `jolt.host`) +The portability seam. jolt-core (analyzer/IR/overlay) calls **only** `jolt.host` +fns, never Janet directly. Originally compiler-facing (`form-sym?`, `form-list?`, +`resolve-global`, …). Phase 4 added the first runtime primitive: **`ref-put!`** +(set/remove a key on a mutable reference cell) — the minimal mutation kernel the +overlay uses for atom watches/validators, volatiles, and `aset`. The overlay calls +these qualified, e.g. `(jolt.host/ref-put! ...)`. + +--- + +## 3. The migration epic (`jolt-1j0`) — essentially COMPLETE + +**Goal:** shrink the Janet seed to `core-renames` + genuinely host-coupled fns; +express everything else (pure fns, macros, lazy machinery) in the self-hosted +overlay. Started at core.janet = 4145 lines / 421 `core-*` fns. + +**Phases (all done):** +- **Phase 1** — compiler-dependency kernel tier. (Was found already essentially + complete — the analyzer needs nothing beyond the kernel tier + atom/swap!/reset!.) +- **Phase 2** — ~193 movable pure-eager fns → overlay. +- **Phase 3** (`jolt-461`, closed) — ~46 core macros → `defmacro` in the overlay. + Last one was `when-let`. +- **Phase 4** (`jolt-ldf`, closed) — host-coupled fns. ~27 moved over the `ref-put!` + primitive + pure composition (vary-meta, reduce-kv, ex-info accessors, + tagged-value predicates, atom peripheral ops, volatiles, future predicates, + ns-name, array reads/aset). The rest stay native by design (atom/swap!/reset!/ + deref, transients, var cells, meta tables, namespace, constructors, proxy, + print dispatch). +- **Phase 5** (`jolt-c09`, closed) — true laziness. Lazy seq generators + + transformers, the `40-lazy.clj` tier, realization-boundary discipline. See + `phase-5.md` for the full implementation + testing plan and what landed + (representation decision = **Option B / hybrid**: lazy over lazy input, eager + representation-preserving over concrete finite collections). + +The epic issue (`jolt-1j0`) may still read IN_PROGRESS — verify with `bd show +jolt-1j0` and close it if all five phase issues are closed and gates are green. + +**Where to confirm current state:** `phase-5.md` (detailed, step-annotated), +`jolt-core/clojure/core/MIGRATION.md` (the worklist + bucket classification), and +the bd memories `phase4-host-primitive-pattern` / `phase4-movable-classification`. + +--- + +## 4. Representation facts you MUST know (the trap floor) + +Jolt's value/form representations bite every time. The essentials: + +- **Reader forms:** a *call/list* `(f x)` is a Janet **array**; a *vector literal* + `[a b]` is a Janet **tuple**; a *map literal* `{..}` is a Janet **struct** (or a + **phm** when a key/val is nil or a key is a collection). A **symbol** is a struct + `{:jolt/type :symbol :ns _ :name _}`; a **keyword** is a Janet keyword. +- **Runtime values:** vectors are persistent-vectors (`pvec`, tagged tables) or + tuples; lists/seq-results are Janet arrays or `plist`; sets are `phs`; maps are + struct-or-phm. `vector?` is true for tuple **and** pvec. `seq?` is arrays/plists/ + lazy-seqs (not vectors). In `JOLT_MUTABLE` builds vectors are plain arrays — so + `vector?`/`array?` collapse (this is why `ifn?` couldn't move — see `jolt-1vx`). +- **Tagged values** carry their kind in `:jolt/type` (atoms, volatiles, delays, + futures, ex-info, reader-conditional, lazy-seq) or `:jolt/deftype` (records). The + overlay can **read** these via `(get x :jolt/type)` / `(get x :field)` — `get` + returns nil on non-tables, no error. It **cannot construct** them without a host + primitive. This is the Phase-4 movability rule: accessors/predicates move, + constructors stay. +- **`canon-key`** (core.janet ~line 51) is the canonical-hashing kernel of the + whole persistent-collection system — woven into `get`/`count`/`contains?`. This + is why transients are irreducibly host. +- **LazySeq** (`phm.janet`): `@{:jolt/type :jolt/lazy-seq :fn thunk ...}`; thunk → + `nil` or `[first rest-thunk]`; `realize-ls` memoizes with a `:jolt/pending` guard + that makes self-referential seqs (`lazy-cat` fib) work. + +### Macro/overlay-authoring gotchas (learned the hard way) +- Build binding/forms via syntax-quote templates `` `[~@xs] `` (a tuple form), not + `conj`/`list` (those make pvecs/plists the analyzer/compiler rejects). +- A fresh symbol inside a macro body: `(symbol (str (gensym)))` — a bare `(gensym)` + returns a *Janet* symbol the destructurer rejects. +- A `.clj` tier is **Clojure** (`;;` comments). A `.janet` test/spec is **Janet** + (`#` comments — `;` is splice!). Mixing them is a frequent self-inflicted error. +- In a tier, a fn must be defined *after* the macro it uses is defined; use `def` + + `fn*` if you need it before `defn` exists (as `destructure` does in 00-syntax). + +--- + +## 5. Build, run, and the test gate + +No special build needed to run from source — Janet reads the tiers off disk. + +```bash +# Smoke +janet -e '(use ./src/jolt/api) (pp (eval-string (init) "(+ 1 2)"))' + +# THE GATE — run all of these green before committing any core change: +janet test/integration/conformance-test.janet # 229 cases × 3 modes (interpret/compile/self-host) +janet test/integration/bootstrap-fixpoint-test.janet # stage1 == stage2 == stage3 +janet test/integration/self-host-test.janet +janet test/integration/sci-bootstrap-test.janet # loads vendored SCI through jolt +janet test/integration/clojure-test-suite-test.janet # battery; baseline-pass=3971, clean-files=45 +for f in test/spec/*.janet test/unit/*.janet; do janet "$f"; done # all must exit 0 +``` + +- **Specs** (`test/spec/*-spec.janet`) — data-driven `defspec` tables, behavioral. +- **Conformance** — real-Clojure-semantics assertions, run in all 3 execution modes. +- **clojure-test-suite** — runs `lread/clojure-test-suite` (from `~/src/clojure- + test-suite`) via a per-file **subprocess under a 6 s deadline** (infinite seqs are + CPU-bound and uninterruptible in-process — never probe them inline). Skips if the + suite dir is absent. Raise `baseline-pass` as jolt improves; never lower it. +- **Laziness** must be tested via the deadlined subprocess harness, not in-process. + +### Per-change workflow (mirror this) +1. Make a small, single-purpose change. +2. Add/extend spec + (for subtle behavior) 3-mode conformance cases. +3. Run the full gate. Commit only if green. +4. `git push` (the project's session-close protocol requires pushed work). + +--- + +## 6. Conventions + +- **Issue tracker: beads (`bd`)**, not TodoWrite/markdown. `bd ready`, `bd show + `, `bd create`, `bd update --status=…`, `bd close`. `bd remember + --key … "…"` for durable knowledge; `bd memories` to recall. The `.beads/` + dir is git-ignored and auto-synced — don't `git add` it. +- **Commits/PRs**: terse, factual, human-dev tone. No marketing words, no emoji, no + "This commit…". Say what changed and why it matters. +- **Branch**: work happens on `compiler-research` (main is `main`). +- Don't lower `baseline-pass`. If a moved fn surfaces a latent bug, fix it to match + Clojure and add a regression test rather than preserving the bug (this happened + with `reduce-kv` on vectors and `ifn?` on lists). + +--- + +## 7. Where to pick up + +The migration epic is functionally complete; the seed is at its intended floor +(core-renames + genuinely host-coupled). Candidate next work: + +- **Close out `jolt-1j0`** if not already closed (verify all phase issues closed, + gates green). +- **`jolt-1vx`** (filed) — `ifn?` is wrongly true for lists; move to overlay but + it's representation-mode-sensitive (`JOLT_MUTABLE`). Needs both-mode verification. +- **Phase-5 loose ends** (see `phase-5.md`): a few transformers were kept eager or + reverted due to compile-mode `~@`/defrecord splice issues (`partition-by`, + `dedupe`, `tree-seq`, lazy `mapcat`). Re-verify the ~9 previously-timing-out suite + files actually stopped timing out. The Step 4 "apply/`~@` over lazy" fix would + unblock the reverted lazy `mapcat`. +- **Bigger lifts not attempted** (deliberately): the `print-method`/`pr-str` + dispatch machinery and the `deftype`/`defrecord`/`defprotocol`/multimethod surface + — both substantial and host-entangled. +- **Open issues**: `bd ready` for the current actionable list (CI, edn/walk/zip + stdlib, `into #{}` bug, recur-into-variadic hang, real futures via ev/thread, + etc. — these predate the migration). + +### Map of the territory +- `src/jolt/core.janet` — the Janet seed (`core-*` fns, `core-bindings`, `core-renames`). +- `src/jolt/evaluator.janet` — interpreter. `src/jolt/compiler.janet` — bootstrap compiler. +- `src/jolt/backend.janet` — IR → Janet emitter. `src/jolt/host_iface.janet` — `jolt.host`. +- `src/jolt/phm.janet` — persistent maps/sets/vectors + LazySeq. +- `src/jolt/api.janet` — context init + tier loading. `src/jolt/reader.janet` — reader. +- `jolt-core/jolt/{analyzer,ir}.clj` — portable self-hosted front end. +- `jolt-core/clojure/core/*.clj` — the overlay tiers + `MIGRATION.md`. +- `phase-5.md` — the laziness plan, annotated with what landed. +- `CLAUDE.md` / `AGENTS.md` — project agent instructions (beads, session-close).