From fdf2c742c46fdbbd42fdaa9dc748461f96c95509 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Mon, 8 Jun 2026 19:44:32 -0400 Subject: [PATCH] remove handoff file --- HANDOFF.md | 223 ----------------------------------------------------- 1 file changed, 223 deletions(-) delete mode 100644 HANDOFF.md diff --git a/HANDOFF.md b/HANDOFF.md deleted file mode 100644 index 1b45044..0000000 --- a/HANDOFF.md +++ /dev/null @@ -1,223 +0,0 @@ -# Jolt — Self-Hosted Clojure on Janet · Handoff - -Onboarding for a fresh agent picking up this work. Read this, then -`bd prime` and `bd memories` for the live issue/knowledge state. - ---- - -## 1. What this project is - -**Jolt** is a Clojure implementation written in [Janet](https://janet-lang.org). -It has two execution paths and a **self-hosting compiler**: - -- **Interpreter** — `src/jolt/evaluator.janet`. A tree-walking evaluator over - reader forms. Always correct; the fallback for anything the compiler can't yet - handle. The *live path* for stateful/context-modifying forms. -- **Self-hosted compiler** — the portable front end lives in **Clojure** under - `jolt-core/jolt/` (`analyzer.clj` reader-form → host-neutral IR `ir.clj`), and a - **Janet back end** (`src/jolt/backend.janet`) emits Janet from that IR. This is - the default compile path. It is *self-hosted*: the compiler that compiles - clojure.core is itself (mostly) Clojure compiled by jolt. -- **Bootstrap compiler** — `src/jolt/compiler.janet`. A Janet-native compiler used - **only** to bootstrap-compile the kernel tier before the self-hosted analyzer - exists. Not the main path. -- **Hybrid fallback** — the analyzer throws `:jolt/uncompilable` on forms it can't - handle; the loader catches that and interprets instead. Three "uncompilable" - lists are kept in sync (see compile-pipeline notes in the code). - -Entry point: `src/jolt/api.janet` — `(init opts)` builds a context, installs the -host contract, and loads clojure.core (seed + overlay). `:compile? true` enables -the self-hosted pipeline; off = interpret. - ---- - -## 2. The architecture that matters: seed + overlay - -clojure.core is split into a shrinking **Janet seed** and a growing **Clojure -overlay**. This split *is* the project's main arc. - -### The Janet seed — `src/jolt/core.janet` (~3200 lines, ~365 `core-*` fns) -The irreducible base: the `core-renames` primitives the compiler emits directly -(`first`/`nth`/`conj`/`get`/…) plus genuinely host-coupled fns (atoms, vars, -transients, arrays, futures, meta, print, the persistent-collection kernel). Each -fn is `core-` and interned into the `clojure.core` namespace via the -`core-bindings` table near the bottom of the file. - -### The Clojure overlay — `jolt-core/clojure/core/NN-*.clj` (loaded in order) -Plain Clojure expressing the *rest* of clojure.core on top of the seed. Tiers: - -| Tier | Role | -|---|---| -| `00-syntax.clj` | control macros (`when`/`cond`/`and`/`or`/`let`/`loop`/`fn`/`for`/…), `destructure`, `when-let`. Interpreted, loaded **first** so macros exist before any code compiles. | -| `00-kernel.clj` | structural fns the analyzer itself needs (`second`/`peek`/`subvec`/`mapv`/`update`). **Bootstrap-compiled** into clojure.core before the analyzer is built. | -| `10-seq.clj` | seq-tier fns | -| `20-coll.clj` | pure collection/misc fns + the Phase-4 host-primitive wrappers | -| `30-macros.clj` | the remaining user-facing macros | -| `40-lazy.clj` | lazy seq transformers (Phase 5) | - -Loader: `api.janet` → `load-core-overlay!` / `core-tiers`. Sources are read **fresh -from disk** at startup when running from the repo (`stdlib_embed.janet` collects -`jolt-core/` and `src/jolt/clojure/`), so editing a `.clj` tier takes effect with -no rebuild. (A `jpm build` bakes them into the image; that can go stale — tests -run from source.) - -### The host contract — `src/jolt/host_iface.janet` (ns `jolt.host`) -The portability seam. jolt-core (analyzer/IR/overlay) calls **only** `jolt.host` -fns, never Janet directly. Originally compiler-facing (`form-sym?`, `form-list?`, -`resolve-global`, …). Phase 4 added the first runtime primitive: **`ref-put!`** -(set/remove a key on a mutable reference cell) — the minimal mutation kernel the -overlay uses for atom watches/validators, volatiles, and `aset`. The overlay calls -these qualified, e.g. `(jolt.host/ref-put! ...)`. - ---- - -## 3. The migration epic (`jolt-1j0`) — essentially COMPLETE - -**Goal:** shrink the Janet seed to `core-renames` + genuinely host-coupled fns; -express everything else (pure fns, macros, lazy machinery) in the self-hosted -overlay. Started at core.janet = 4145 lines / 421 `core-*` fns. - -**Phases (all done):** -- **Phase 1** — compiler-dependency kernel tier. (Was found already essentially - complete — the analyzer needs nothing beyond the kernel tier + atom/swap!/reset!.) -- **Phase 2** — ~193 movable pure-eager fns → overlay. -- **Phase 3** (`jolt-461`, closed) — ~46 core macros → `defmacro` in the overlay. - Last one was `when-let`. -- **Phase 4** (`jolt-ldf`, closed) — host-coupled fns. ~27 moved over the `ref-put!` - primitive + pure composition (vary-meta, reduce-kv, ex-info accessors, - tagged-value predicates, atom peripheral ops, volatiles, future predicates, - ns-name, array reads/aset). The rest stay native by design (atom/swap!/reset!/ - deref, transients, var cells, meta tables, namespace, constructors, proxy, - print dispatch). -- **Phase 5** (`jolt-c09`, closed) — true laziness. Lazy seq generators + - transformers, the `40-lazy.clj` tier, realization-boundary discipline. See - `phase-5.md` for the full implementation + testing plan and what landed - (representation decision = **Option B / hybrid**: lazy over lazy input, eager - representation-preserving over concrete finite collections). - -The epic issue (`jolt-1j0`) may still read IN_PROGRESS — verify with `bd show -jolt-1j0` and close it if all five phase issues are closed and gates are green. - -**Where to confirm current state:** `phase-5.md` (detailed, step-annotated), -`jolt-core/clojure/core/MIGRATION.md` (the worklist + bucket classification), and -the bd memories `phase4-host-primitive-pattern` / `phase4-movable-classification`. - ---- - -## 4. Representation facts you MUST know (the trap floor) - -Jolt's value/form representations bite every time. The essentials: - -- **Reader forms:** a *call/list* `(f x)` is a Janet **array**; a *vector literal* - `[a b]` is a Janet **tuple**; a *map literal* `{..}` is a Janet **struct** (or a - **phm** when a key/val is nil or a key is a collection). A **symbol** is a struct - `{:jolt/type :symbol :ns _ :name _}`; a **keyword** is a Janet keyword. -- **Runtime values:** vectors are persistent-vectors (`pvec`, tagged tables) or - tuples; lists/seq-results are Janet arrays or `plist`; sets are `phs`; maps are - struct-or-phm. `vector?` is true for tuple **and** pvec. `seq?` is arrays/plists/ - lazy-seqs (not vectors). In `JOLT_MUTABLE` builds vectors are plain arrays — so - `vector?`/`array?` collapse (this is why `ifn?` couldn't move — see `jolt-1vx`). -- **Tagged values** carry their kind in `:jolt/type` (atoms, volatiles, delays, - futures, ex-info, reader-conditional, lazy-seq) or `:jolt/deftype` (records). The - overlay can **read** these via `(get x :jolt/type)` / `(get x :field)` — `get` - returns nil on non-tables, no error. It **cannot construct** them without a host - primitive. This is the Phase-4 movability rule: accessors/predicates move, - constructors stay. -- **`canon-key`** (core.janet ~line 51) is the canonical-hashing kernel of the - whole persistent-collection system — woven into `get`/`count`/`contains?`. This - is why transients are irreducibly host. -- **LazySeq** (`phm.janet`): `@{:jolt/type :jolt/lazy-seq :fn thunk ...}`; thunk → - `nil` or `[first rest-thunk]`; `realize-ls` memoizes with a `:jolt/pending` guard - that makes self-referential seqs (`lazy-cat` fib) work. - -### Macro/overlay-authoring gotchas (learned the hard way) -- Build binding/forms via syntax-quote templates `` `[~@xs] `` (a tuple form), not - `conj`/`list` (those make pvecs/plists the analyzer/compiler rejects). -- A fresh symbol inside a macro body: `(symbol (str (gensym)))` — a bare `(gensym)` - returns a *Janet* symbol the destructurer rejects. -- A `.clj` tier is **Clojure** (`;;` comments). A `.janet` test/spec is **Janet** - (`#` comments — `;` is splice!). Mixing them is a frequent self-inflicted error. -- In a tier, a fn must be defined *after* the macro it uses is defined; use `def` + - `fn*` if you need it before `defn` exists (as `destructure` does in 00-syntax). - ---- - -## 5. Build, run, and the test gate - -No special build needed to run from source — Janet reads the tiers off disk. - -```bash -# Smoke -janet -e '(use ./src/jolt/api) (pp (eval-string (init) "(+ 1 2)"))' - -# THE GATE — run all of these green before committing any core change: -janet test/integration/conformance-test.janet # 229 cases × 3 modes (interpret/compile/self-host) -janet test/integration/bootstrap-fixpoint-test.janet # stage1 == stage2 == stage3 -janet test/integration/self-host-test.janet -janet test/integration/sci-bootstrap-test.janet # loads vendored SCI through jolt -janet test/integration/clojure-test-suite-test.janet # battery; baseline-pass=3971, clean-files=45 -for f in test/spec/*.janet test/unit/*.janet; do janet "$f"; done # all must exit 0 -``` - -- **Specs** (`test/spec/*-spec.janet`) — data-driven `defspec` tables, behavioral. -- **Conformance** — real-Clojure-semantics assertions, run in all 3 execution modes. -- **clojure-test-suite** — runs `lread/clojure-test-suite` (from `~/src/clojure- - test-suite`) via a per-file **subprocess under a 6 s deadline** (infinite seqs are - CPU-bound and uninterruptible in-process — never probe them inline). Skips if the - suite dir is absent. Raise `baseline-pass` as jolt improves; never lower it. -- **Laziness** must be tested via the deadlined subprocess harness, not in-process. - -### Per-change workflow (mirror this) -1. Make a small, single-purpose change. -2. Add/extend spec + (for subtle behavior) 3-mode conformance cases. -3. Run the full gate. Commit only if green. -4. `git push` (the project's session-close protocol requires pushed work). - ---- - -## 6. Conventions - -- **Issue tracker: beads (`bd`)**, not TodoWrite/markdown. `bd ready`, `bd show - `, `bd create`, `bd update --status=…`, `bd close`. `bd remember - --key … "…"` for durable knowledge; `bd memories` to recall. The `.beads/` - dir is git-ignored and auto-synced — don't `git add` it. -- **Commits/PRs**: terse, factual, human-dev tone. No marketing words, no emoji, no - "This commit…". Say what changed and why it matters. -- **Branch**: work happens on `compiler-research` (main is `main`). -- Don't lower `baseline-pass`. If a moved fn surfaces a latent bug, fix it to match - Clojure and add a regression test rather than preserving the bug (this happened - with `reduce-kv` on vectors and `ifn?` on lists). - ---- - -## 7. Where to pick up - -The migration epic is functionally complete; the seed is at its intended floor -(core-renames + genuinely host-coupled). Candidate next work: - -- **Close out `jolt-1j0`** if not already closed (verify all phase issues closed, - gates green). -- **`jolt-1vx`** (filed) — `ifn?` is wrongly true for lists; move to overlay but - it's representation-mode-sensitive (`JOLT_MUTABLE`). Needs both-mode verification. -- **Phase-5 loose ends** (see `phase-5.md`): a few transformers were kept eager or - reverted due to compile-mode `~@`/defrecord splice issues (`partition-by`, - `dedupe`, `tree-seq`, lazy `mapcat`). Re-verify the ~9 previously-timing-out suite - files actually stopped timing out. The Step 4 "apply/`~@` over lazy" fix would - unblock the reverted lazy `mapcat`. -- **Bigger lifts not attempted** (deliberately): the `print-method`/`pr-str` - dispatch machinery and the `deftype`/`defrecord`/`defprotocol`/multimethod surface - — both substantial and host-entangled. -- **Open issues**: `bd ready` for the current actionable list (CI, edn/walk/zip - stdlib, `into #{}` bug, recur-into-variadic hang, real futures via ev/thread, - etc. — these predate the migration). - -### Map of the territory -- `src/jolt/core.janet` — the Janet seed (`core-*` fns, `core-bindings`, `core-renames`). -- `src/jolt/evaluator.janet` — interpreter. `src/jolt/compiler.janet` — bootstrap compiler. -- `src/jolt/backend.janet` — IR → Janet emitter. `src/jolt/host_iface.janet` — `jolt.host`. -- `src/jolt/phm.janet` — persistent maps/sets/vectors + LazySeq. -- `src/jolt/api.janet` — context init + tier loading. `src/jolt/reader.janet` — reader. -- `jolt-core/jolt/{analyzer,ir}.clj` — portable self-hosted front end. -- `jolt-core/clojure/core/*.clj` — the overlay tiers + `MIGRATION.md`. -- `phase-5.md` — the laziness plan, annotated with what landed. -- `CLAUDE.md` / `AGENTS.md` — project agent instructions (beads, session-close).